Volume Number: 25
Issue Number: 04
Column Tag: System Administration

Collaborative Methods: Exchange Web Services

Exchange Web Services "Finally, we can all get along," or, "What administrators need to know about Microsoft's newest E-mail protocol"

by William Smith

"Collaboration" is the current buzzword in the Enterprise and three big players - Apple, Google and Microsoft - are all targeting the Cloud as their arena for releasing new services. Up in the corporate cloud lies Microsoft Exchange, a mature and robust set of technologies for bridging the sync gap between different applications, different devices and more importantly different platforms.

Exchange Server is Microsoft's product for serving E-mail, calendars, contacts and more. It is nearly ubiquitous in the Windows-dominated enterprise and has a strong foothold in the higher education market as well. Exchange acts as a hub with a variety of local, mobile and external clients using both standard and proprietary protocols. However, with the recent European Commission's decision requiring Microsoft "to disclose complete and accurate specifications" for its server communications protocols, third party companies now have the opportunity to expand and develop products that will work with Exchange.

Exchange Web Services (EWS) is the first of a few new Microsoft server technologies that will play a key role in both Apple's plans to penetrate the enterprise and Microsoft's plans to expand compatibility with Macs. While Google hasn't announced any plans for integrating its mail system with Exchange, it too may very well decide to capitalize on EWS in the future. It has already taken steps toward using one Exchange technology, ActiveSync.

In addition to mail, Exchange allows users to maintain and share calendar events, contacts, tasks and notes. Each of these objects can be sent and received via e-mail allowing groups within companies to set up their own workflows. Combined with a feature-rich client that can process these incoming items, such as Outlook for Windows, Exchange becomes the hub for relaying thousands of pieces of information daily for both small businesses and large institutions.

Isn't Entourage an Exchange client and can't it do all this too? The fairest answer to that question is "It's getting there." Entourage is akin to Outlook for Windows and is sold as part of the Office for Mac suite of applications that includes Excel, PowerPoint and Word. It is the closest to a feature-rich client available for the Mac but still lacks all the capabilities of Outlook when used with Exchange.

Outlook for Mac, never to be confused with Outlook Express, was actually developed by the Exchange Server Team at Microsoft a little more than 10 years ago but was discontinued after Outlook 2001. Outlook for Mac was a client that used the Messaging Application Programming Interface (MAPI) connection just like Outlook for Windows. MAPI is just mail protocol like the more common POP or IMAP protocols used in businesses and at home today, but what's important about MAPI is that Microsoft never licensed it. MAPI is an old and loosely designed protocol that Microsoft kept proprietary. While Outlook for Mac was still never equal in features with Outlook for Windows, it did communicate with Exchange Server the same way. Entourage had never done this until version 13.0.0, also known as Entourage for Exchange Web Services (EWS). More on that in a bit.

A tale of two protocols

The MacBU has probably some of the toughest decisions to make about software development. Their mission is to provide interoperability with their Office for Windows counterpart while maintaining the look and feel of a Mac OS application. That's not easy to do. When the Exchange Server Team decided for various reasons to discontinue Outlook for Mac, the MacBU decided to pick up and add Exchange support to Entourage, their existing E-mail product. They did this instead of continuing Outlook for Mac. After all, why develop and support two E-mail applications that do mostly the same thing?

One of the more controversial decisions was to shy away from MAPI and adopt WebDAV, which is an HTTP-based protocol for sharing information. At the beginning of this decade WebDAV was thought to be the future. It was an open standard, which also meant not proprietary to Microsoft, and was growing in popularity. Apple was also using WebDAV in a new product that it had released: iTools and .Mac, later to become mac.com and then MobileMe. They offered access to a personal iDisk that could be reached from anywhere on the Internet via WebDAV.

WebDAV in Entourage, however, was not robust enough to deliver everything that Outlook using MAPI could deliver. Tasks didn't sync, public folder support was very limited and connectivity was slow. Outlook Web Access (OWA), which is a web-based version of Exchange Server, came in two flavors: a full version that could be accessed only via Internet Explorer for Windows using ActiveX controls and a light version for everyone else. Since Entourage was not Internet Explorer and not ActiveX-enabled it fell into the "everyone else" category. It faced the same issues as the light version of OWA and to this day many of the limitations of OWA are the same as those for Entourage.

The original business decision by the MacBU to use WebDAV was thought to be a step forward. However, it may have been two steps back. The Exchange Server Team at Microsoft never adopted WebDAV as a major protocol of the future and MAPI continued to be the ongoing protocol of choice for Exchange Server and Outlook for Windows.

The first major release of Entourage with Exchange Support was Entourage 2004. (Entourage X, which was the first Mac OS X mail client from the MacBU, had very limited Exchange support through a later-released 10.1.4 update but relied on IMAP for connectivity rather than WebDAV.) Entourage 2004 was the first Mac OS X E-mail client that could truly be called an Exchange client. Through WebDAV it could access mail, calendars, contacts and public folders. However, the first release of Entourage 2004 lacked a lot of parity with its Outlook for Windows counterpart. This was mainly due to constraints in development time.

Outlook was not built in a day and likewise Entourage had to grow in its Exchange support. Entourage 2004 saw major improvements in Service Pack 2 (SP2), which introduced lots of complex features such as multiple calendars and address books, browsing the Global Address List (GAL), delegation support and support for sharing mail folders, calendars and address books. Essentially, everything that the MacBU wanted to put into Entourage 2004 from the beginning but didn't have time to include, got released as a major update a little more than one year later.

Unfortunately, Entourage still couldn't do everything that Outlook for Windows could do, even with these major enhancements. This was due partly to the amount of time needed to develop Exchange Server support within Entourage and the limitations of WebDAV. Entourage was still a second-class citizen and its users were demanding parity with Outlook for Windows.

Fast-forward a few years to 2008. The MacBU released Office 2008 with an upgraded version of Entourage at the Macworld Expo in January. Three years had passed after another major version of Office: v2004. However, Entourage 2008 was underwhelming for Exchange users. MacBU had released four major versions of Entourage, including SP2, with nearly 100 Exchange-specific features but parity with Outlook for Windows still seemed far away.

What was happening behind the scenes at MacBU, however, had been methodically planned and plotted to try to take that step forward again and maybe even regain the two steps lost using WebDAV. Entourage 2008 introduced a few new features such as the Out of Office Assistant and AutoDiscover. These features themselves didn't greatly advance feature parity with Outlook for Windows but the technology behind them, EWS, did.

Exchange Web Services is a feature found only in Exchange Server 2007; it is not present in Exchange Server 2003 and earlier. To illustrate this, the Out of Office Assistant in Entourage will still work with Exchange Server 2003 much like the Out of Office Assistant found in the light version of OWA. But when Entourage 2008 connects to Exchange Server 2007 it begins to take advantage of EWS. Remember, Entourage can do mostly what OWA can do because of WebDAV. With EWS, though, it can do more. Users connecting to Exchange Server 2003 will see a basic Out of Office Assistant in Entourage,

which is similar to the Out of Office Assistant found in OWA.

However, Entourage users connecting to Exchange Server 2007 will see an extended version of the Out of Office Assistant. It includes options for sending replies internal to the company as well as external to the company and the ability to specify start and end dates for the responses:

What happens now to those two protocols: MAPI for Outlook and WebDAV for Entourage? Effective with Exchange Server 2007 both MAPI and WebDAV were "de-emphasized". That doesn't mean Exchange client features are going away but rather newer technologies are assuming the responsibilities of those features. It does mean, though, that at some point in the future, maybe not necessarily the next version, MAPI and WebDAV will be gone from Exchange Server. It also means that further development to expand these technologies has ceased.

The Microsoft TechNet document "Discontinued Features and De-Emphasized Functionality in Exchange 2007 RTM" (aa998911) explains that both MAPI and WebDAV along with other technologies are now considered "legacy Exchange features". In another TechNet document "Exchange 2007 Development" (bb124516), Microsoft specifically states for WebDAV, "We recommend that you develop new messaging applications using Exchange Web Services, and migrate existing applications when feasible." Both articles are available at http://technet.microsoft.com.

On a side note, public folders have also been de-emphasized in favor of SharePoint and the Exchange Availability service. Exchange Server 2007 threw several administrators for a loop when it was released because although the public folders service was still available it wasn't enabled by default. MacBU announced at Macworld 2009 the Document Collaboration Companion (DCC), which went into private beta in February and will be released later this year.

The DCC is a stand-alone application for Mac OS X that provides better uploading, downloading and check-in/out from SharePoint servers as well as Office Live Workspaces. More information is available at Microsoft's Mactopia IT Pros website http://www.microsoft.com/mac/itpros/dcc.mspx.

Office 2008 took the MacBU nearly fours years to ship, which is a very long development cycle for any software. Why did it take this long? To say this was all because of Entourage would be an exaggeration. MacBU was also dealing with making Office 2008 compatible with Office 2007 for Windows. This meant porting over features such as the new XML file format, new core animation code and adapting the Ribbon interface from Office 2007 to the Elements Gallery in Office 2008 to maintain the Mac "feel". None of this affected Entourage but the Entourage developers themselves were busy building bridges. [Ed. Note: In a conversation with Microsoft, I was reminded that this transition was also hampered by the conversion to Intel and a new version of Xcode.]

Before 2005, the relationship between the MacBU and the Exchange Server Team at Microsoft was practically non-existent. The business units didn't meet and the Exchange Server Team didn't pay much attention to bug reports from the MacBU. Again, fast-forward to 2008. By that time the Exchange Server Team was not only meeting with the Entourage developers twice a week but they themselves were also involved with testing Entourage. This new relationship led to close collaboration on items such as the Office 2008 for Mac Administrator's Guide (http://www.microsoft.com/mac/itpros/default.mspx) and even EWS itself.

That's important to restate, highlight and underscore: MacBU was involved with the development and testing of Exchange Web Services in Exchange Server 2007.

MacBU announced a new product this year at Macworld 2009 or rather a major update to Entourage 2008: Entourage for Exchange Web Services. They even consider this an upgrade rather than an update. The version number, which can be found by selecting About Entourage from the Entourage menu, says 13.0.0. The prior version was 12.1.5. According to their IT Pros website, "Entourage for Exchange Web Services has been completely redesigned for Microsoft Exchange Server 2007 Service Pack 1 with Update Rollup 4 or later."

Only a handful of new features were introduced with Entourage EWS, which is currently in public beta, but some of them were long-standing requests. Specifically, Categories, Notes and Tasks now sync to Exchange Server. Additionally, the MacBU was able to add support for attachments in Calendar events and GAL searches when connected to Exchange using an OWA address. The interface itself is exactly the same as Entourage 12.1.5. They have effectively replaced the engine in the car leaving the body alone.

The Entourage EWS beta is open to the public but does require that participants affirm they indeed meet the minimum server requirements. The beta cannot be installed alongside Entourage 2008 (12.1.5), which means the user's computer must be dedicated to testing Entourage EWS. The beta is not intended for anyone who cannot support himself and restore from backups should something go wrong. For more information visit http://www.microsoft.com/mac/itpros/entourage-ews.mspx.

Will Outlook for Windows follow suit? More than likely. It too is already using Exchange Web Services for AutoDiscover and the extended Out of Office Assistant when connecting to Exchange Server 2007.

If Outlook switches to EWS then the ramifications are apparent. Both Entourage and Outlook will once again be connecting to Exchange using the same protocol, just like they did 10 years ago, and that means Exchange client feature parity between the products is possible.

That's another important point to restate, highlight and underscore: If Outlook for Windows and Entourage for Mac OS X both connect to Exchange Server using the same protocol then complete Exchange client feature parity is once again possible.

Exchange Web Services offers "anywhere access". That means any client sitting on a company intranet, connecting from the Internet or connecting via a wireless device can take advantage of EWS.

Many of today's applications use Remote Procedure Calls (RPCs) to work with other applications. Basically, that means one application tells another application running on another machine to do something. For example, Outlook for Windows may send an RPC call to an Exchange Server to set the Out of Office message for a user's account. The downside to using RPCs is that they are often blocked by firewalls. They're certainly not used for communications by computers over the Internet to access servers on a private company network. HTTP, on the other hand, is accessible most anywhere.

EWS sits on top of Microsoft's Internet Information Services (IIS), which uses HTTP. While this may sound like WebDAV, EWS is actually a cousin protocol sitting alongside ActiveSync for mobile devices, OWA for webmail and Unified Messaging for telephony systems. EWS replaces not only WebDAV but also other programming interfaces such as CDOEX. CDOEX, was necessary for complex calendaring, which Entourage connected to Exchange Server 2003 could never do.

EWS uses the Extensible Markup Language (XML) to provide the standard message format for the exchange of information between Exchange Server and the mail client. XML is an established framework for creating a custom language. In this case the custom language is the Exchange Server language. The beauty of XML is that it can be used to share information between different computer platforms and applications without having to pass through interpreters.

Unlike the proprietary MAPI protocol, XML is also an open standard and is becoming more and more prevalent in the Microsoft world. Office 2007 for Windows and Office 2008 for Mac both now ship with the new XML file formats that are flat and more easily exchanged cross-platform. XML can be used in other ways too such as E-mail storage, preference lists and general data storage for access by multiple applications.

Simple Object Access Protocol (SOAP) is a protocol that's also constructed using XML but intended for sending communications over HTTP. During an EWS session the client will take its XML instructions and wrap them in a SOAP envelope. This entire bundle is then transmitted over HTTP to the server.

This approach to communicating with Exchange Server translates to increased speed as well as increased accessibility. Whereas Entourage 2008 (12.1.5) takes six instructions to execute a command over WebDAV, Entourage EWS takes just one. See below.

Restate, highlight and underscore: Entourage EWS is faster than Entourage with WebDAV.

Apple announced last June that Mac OS X 10.6 (Snow Leopard) would include Exchange support out-of-the-box for Microsoft Exchange 2007. Will it connect using Exchange Web Services? Signs point to "Yes", but Apple has made no public comment about this.

Because they specify "Microsoft Exchange 2007" and not just "Microsoft Exchange", that almost certainly means they're excluding "Microsoft Exchange 2003", which is still prevalent in many organizations. And, remember, that MAPI and WebDAV are de-emphasized protocols with direction from the Microsoft Exchange Team to develop new applications that work with EWS. While POP and IMAP are still valid and available mail protocols in Exchange they cannot handle the extended set of features such as calendaring and contacts. If Apple were developing using WebDAV or MAPI then they would most certainly be making a huge mistake at this point in time.

Organizations planning to adopt Snow Leopard's Exchange support, assuming Apple is indeed going to use EWS, need to first ensure that they are onboard with Exchange Server 2007. They should also define their requirements for Exchange usage too. While Apple has a history of making great products that work very well, they don't necessarily have a history of offering a wide array of features in first releases.

Snow Leopard will most likely support syncing E-mail, Calendar, Contacts, Tasks and Notes the day it ships, but Apple may decide that some things like public folders are not worth supporting since that feature has been de-emphasized. Complex features such as delegation and shared folders may be slated for a later release or not at all and features currently specific to Outlook for Windows such as setting server-side rules and polling may get sent to the back burner.

What can be certain is that all developers will have access to the same documentation for developing for EWS. Of course, the folks at the MacBU may have the Exchange Server Team's ear a little more than anyone else. Interestingly, an EWS bug will be a double-edged sword. One bug will affect Entourage, Outlook, Snow Leopard and any other EWS application, but so will one bug fix.

Exchange parity has been a sore spot with corporate Mac users for nearly a decade but in the next 12-18 months that could all change. At least one more EWS client, Snow Leopard, and possibly a newer version of Entourage will be available. How about others?

Proprietary protocols, difficult-to-support protocols and workarounds will all have been replaced with open protocols and all Exchange clients will finally be communicating to Exchange Server the same way. The playing field for collaborative services will be level.

What happens next depends on the ingenuity of the client developers. Will most decide to offer full Exchange support to their customers by including every available feature or will some opt for "light" EWS clients? How will they distinguish their products from the competition's products?

New companies and maybe even some existing players may find the EWS arena appealing and offer more options than have ever been available. Online companies such as Google and Yahoo, for example, may decide to offer integrated Exchange support with their systems so that work Exchange calendars can finally be overlaid with online personal calendars without mixing information. Companies like Now Software may integrate their systems with Exchange to offer another interface choice for users. Or Omni Group may offer OmniFocus as a single robust task manager application that does nothing else except manage tasks very well.

Often seen as a barrier to rival products, Microsoft's server dominance may actually become a catalyst for new development and new methods for collaboration on more than just the Windows platform. So long as protocols such as EWS are made available to everyone then competition will flourish and consumers will benefit.

Let the competition begin!

William Smith is a technical analyst supporting Macs in a Windows world in the Twin Cities, a six-year Microsoft MVP and is co-founder of the Entourage Help Blog (http://blog.entourage.mvps.org). He can be reached at bill@talkingmoose.net.