Volume Number: 24 (2008)
Issue Number: 02
Column Tag: MacTech Labs

Virtualization Benchmarking

How do Boot Camp, Parallels Desktop, and VMware Fusion stack up?

by Neil Ticktin, Editor-in-Chief

Part of the reason many people own an Intel-based Mac is because of the possibility of running Windows. If you are like most, you are looking to understand the differences between Apple's Boot Camp, VMware Fusion, and Parallels Desktop.

Boot Camp, as you probably know, allows you to run Windows natively on your Intel Mac. Here, Mac OS X is nowhere to be seen, and if you want to switch back and forth, you have to reboot the machine. As we've seen from some of the recent reports, a Mac can run Windows faster than a native PC machine, and it's a nice solution. That said, you probably bought a Mac to run Mac OS X a good chunk of time, and that's where virtualization comes in.

Virtualization technology has been around since the 1960s. In general, it refers to the abstraction of computer resources. In our case, we're talking about the ability to run Windows on a Mac at the same time that you are running Mac OS X.

So, which solution do you go with? Does virtualization work well? Which virtualization product is faster? Should I run XP or Vista? In short, there are different answers for different people. It all depends on what your needs are.

To tackle this problem, MacTech undertook a huge benchmarking project starting in September. The goal was to see how Boot Camp, VMware Fusion, and Parallels performed on different levels of Mac hardware, covering both Windows XP and Vista, and comparing that to a baseline PC running Windows.

Sounds simple enough, but when you start to realize that there were 19 configurations of systems and a whole suite of tests that had to be run multiple times to make sure that we had good test values, there were over 2500 tests to be completed. In fact, during the course of implementing these tests, both VMware and Parallels came out with newer versions. We tested VMware Fusion 1.0 (51348) and Parallels 3.0 (5160), and opted not to upgrade mid-way through the test suite. Both of these versions should give a good overall baseline of comparison even with both having released new versions.

Let's take a look at the results.

When we were choosing computer models, we set out to choose not the fastest, latest models, but ones that would be a good representation of what most people may have. Certainly, the faster models of these computers will perform even better.

Similarly, we had a greater focus on XP simply because it's more prevalent at this point, but we did want to get an understanding of how Vista performed as well.

The baseline PC we used was a brand new Fujitsu Lifebook A6025, with an Intel Core Duo running at 1.86 GHz, 1GB RAM, running Windows XP SP2.

We chose three Mac models to compare alongside a name brand PC: a MacBook, a MacBook Pro, and a Mac Pro.

The MacBook was a 2GB RAM machine, running a 1.83 GHz Core Duo processor. The MacBook Pro was a 4GB RAM machine, running a 2.16 GHz Core 2 Duo processor. And, the Mac Pro was a 4GB RAM machine, running a Quad Core configuration with two 2.66 GHz Dual-Core Intel Xeon processors.

There are three kinds of tests that we ran: one step tasks, multi-step task tests, and quantitative benchmarks.

The first set of tests was focused on time to complete one step tasks in Microsoft Office 2007, Internet Explorer, and file/network i/o. These one step tasks mean that from the human point of view, once you clicked a mouse, or key, the test would run to completion without further human interaction. For example, launching applications, scrolling large documents, printing, etc.

The second set of tests were "task tests". These were primarily cross platform, and required multiple steps from beginning to end. These were focused on the interaction between Mac OS X and the virtualization environment. For example, if you were to receive a PDF enclosure in Outlook, and you wanted to open it up in Mac OS X Preview; or you wanted to click on an email link in Safari, and you wanted to use it to create a new email message in Outlook.

The third set of tests are quantitative benchmarks using a utility to test CPU, graphics, disk, etc... In our case, we experimented with Sandra 2007 benchmarks, but found the results to be erroneous and not usable for the virtualization environments. We threw out these results and focused on the first two sets.

For those interested in the benchmarking methodologies, see the more detailed testing information in Appendix A. For the detailed results of the tests used for the analysis, see Appendix B. Both appendices are available on the MacTech web site.

We won't keep you in suspense. In general, here are the conclusions.

When testing the one step tests described above, on average, Parallels is 17% faster than VMware Fusion when running Windows XP, and 1% faster than Boot Camp. For the same tests under Vista, VMware Fusion runs 46% slower than Boot Camp, and Parallels runs 44% slower than VMware Fusion (110% slower than Boot Camp). For the task tests, Parallels is the clear winner over VMware Fusion - averaging over 6x faster than its competitor on XP, and 5.2x faster on Vista. To be clear, however, this is not because of poor virtualization performance per se, but the way each product is designed. Parallels is designed to have the virtual machine interact transparently with the host OS (Mac OS X). VMware Fusion, on the other hand, is designed on purpose to keep the environments separate. As a result, there are many extra steps you have to do sometimes if you are going back and forth between the host OS (Mac OS X) and the virtual machine (Windows).

VMware Fusion has the capability of creating a virtual machine with two virtual processors. Parallels does not have that capability. For most tasks, two virtual processors had no significant impact, and sometimes ran even slower than the single processor configuration. And, even though we didn't test it here, we would expect those applications that can take advantage of multiple processors to run more quickly in a two virtual processor environment.

In virtually all cases, we found that the faster the machine, the less the difference between Parallels and VMware Fusion. Even more interesting is that both products could be faster than Boot Camp in certain activities. For example, Parallels was very fast at scrolling beating not only the PC, but also Boot Camp, time after time. Similarly, VMware Fusion appears to have some type of disk caching or acceleration in its virtual hard disk which makes it faster than both the PC and Boot Camp.

The tests used were selected specifically to give a real-world view of what Boot Camp, VMware Fusion and Parallels are like to run. We eliminated those tests that we ran that were so short a time frame (e.g., fast) that we could not create statistically significant results, or that had imperceivable differences.

We did one test suite for each of the four major applications in Microsoft Office 2007 for Windows, as well as a series of tests focused on complex tasks crossing between the host OS and the virtual machine. The goal is to give users a good idea of what it's like to use each of these apps. As a result, we came up with a list of tests that represented what we felt were the most relevant to regular use.

The one thing that we did see across the board is that Windows carries a great deal of variability with it, regardless of the environment.

Outlook

Of all the applications tested, Outlook is the only one that truly has no equivalent on the Mac. Entourage is of course available, but it is not a "Mac version of Outlook". As a result, many people are tied to Outlook use for corporate environments or because it has some specific feature set they need. Virtualization provides the perfect solution for this type of environment.

For Outlook, the one step tasks tests included launching (both initial and successive launches), IMAP account sync, opening messages and printing messages. In addition, we tested sending e-mail, sorting messages, and searching, but found them to function too fast to test on even the slowest machine.

Running XP, Parallels was up to 18% faster than VMware Fusion, and a bit faster than Boot Camp. Running Vista, Outlook tests ran about identical for both Parallels and VMware Fusion.

Figure 1 shows the results with normalized percentages. 100% is the baseline PC score, and the other times are relative to the PC.

Figure 1. Microsoft Outlook 2007 under XP
(shorter columns means faster)

Word

Of all the applications, Word is probably the most widely used. With Office 2008 for Mac no longer supporting VBA, some users will turn to Office 2007 for Windows under virtualization as a solution.

The tests relevant to most Word users are those that are repeated throughout the use of Word. For productivity tests, we selected a variety of launching, scrolling, saving, find & replace, opening files, and printing tests.

The end result is that Word under XP performed nearly identical on average between Parallels and VMware Fusion and Boot Camp. Under Vista, Parallels and VMware Fusion were very close to either other, but lagged some behind Boot Camp.

The figure shows the results with normalized percentages. 100% is the baseline PC score, and the other times are relative to the PC.

Figure 2. Microsoft Word 2007 under XP
(shorter columns means faster)

Excel

For Excel, we wanted to focus again on the most repetitive tasks. For productivity tests, we selected fill range, scrolling vertically, subtotals, and auto-formatting.

Under XP, Parallels was up to 21% faster than VMware Fusion, and 15% faster than Boot Camp. Under Vista, it was a very different story: VMware Fusion was over 40% faster than Parallels, and Boot Camp was 23% faster than VMware Fusion.

The figure shows the results with normalized percentages. 100% is the baseline PC score, and the other times are relative to the PC.

Figure 3. Microsoft Excel 2007 under XP
(shorter columns means faster)

PowerPoint

For PowerPoint, we focused on transitions: both viewing them, and changing them en masse. Here, we saw nearly identical results of the baseline PC running XP, and Boot Camp, VMware Fusion, and Parallels under both XP and Vista. (The chart shows the differences, but you'll notice that the scale is all within a couple of percent.)

The figure shows the results with normalized percentages. 100% is the baseline PC score, and the other times are relative to the PC.

Figure 4. Microsoft PowerPoint 2007 under XP
(shorter columns means faster)

Network and File I/O

Overall, the file I/O between Boot Camp, VMware Fusion and Parallels were all very similar. At times, we saw lag anomalies on Parallels. We also saw VMware Fusion's virtual drive consistently performing very close to a physical drive.

Networking for both virtualization environments lagged behind Boot Camp, but was considerably faster than our base PC. One thing to note is that VMware Fusion had problems several times scoring very poorly using "sharing the host's connection" (aka NAT). Out of each test batch, there was always at least one that scored more like we were expecting, and those best cases were used for the chart. VMware Fusion did perform more reliably with a "bridged" configuration.

Parallels, performed much more consistently across timings for a specific networking test, regardless of the type of virtual network adapter used.

The figure shows the results with normalized percentages. 100% is the baseline PC score, and the other times are relative to the PC.

Figure 5. Network and File I/O under XP
(shorter columns means faster)

Internet Explorer

For Internet Explorer, we ran tests focusing on loading complex web pages from a local web server, and scrolling long web pages. Presumably because of the same networking variability noted above in VMware Fusion, Parallels performed 55-60% faster than VMware Fusion on average for both XP and Vista. The large spike for VMware Fusion on a MacBook is a good indication for the network performance anomalies that we saw with VMware Fusion as well. Parallels was also slightly faster than Boot Camp, and about the same as the base PC.

The figure shows the results with normalized percentages. 100% is the baseline PC score, and the other times are relative to the PC.

Figure 6. Internet Explorer under XP
(shorter columns means faster)

Cross Platform Task Tests

VMware Fusion and Parallels have very different designs.

VMware Fusion is designed from the ground up to be separated from the host OS. Some have called this "sand boxed". It has the benefit of keeping things very clean, and intuitive ... but most importantly secure. You have to make real efforts for one OS to affect the other.

Parallels is designed from the ground up to have transparent boundaries. This has the benefit of making cross platform tasks quite easy, as the barriers are knocked down. To test the utility of this design, we chose a series of multi-step tests. For optimum results, many of these had to be set up in Parallels' preferences, or they would have performed similarly to VMware Fusion. For example, Parallels preferences allow you to you set which OS is the primary for handling web or mail. And, in version 3 of Parallels, there are now "lock down" features that allow a virtual machine to increase "isolation", and therefore allow less integration between Mac OS X and the guest OS (e.g., Windows).

The tests that we timed were as follows.

Within an Outlook message, there was a PDF enclosure. Double click on this enclosure and open it within the Windows version of Adobe Reader. This was our baseline test.

Similarly, within an Outlook message, open a PDF enclosure in Mac OS X's Preview. For VMware Fusion, this required moving the enclosure to the Mac desktop, and then opening. For Parallels, it could be configured as a double click

The same test for to open a Word document enclosure in Outlook in Microsoft Word for Mac. Parallels allows one to specify which application to open regardless of OS, where VMware Fusion required that you move the enclosure to a Mac accessible directory before opening.

If a user were to use Safari as a main browser, but Outlook as primary email, what would it look like if they had an email link on the page? In Parallels, you could click on the link and it would automatically open in Outlook with a new, addressed message. In VMware Fusion, you had to copy the link, open Outlook, create the new message and then paste the address in.

If your primary mail application was Mail.app on the Mac and you got a Visio (.vsd) enclosure, you would need to open that on Windows within Internet Explorer (with the Visio plug-in installed). Parallels could do this transparently, where VMware Fusion you had to move the file first before opening it.

Lastly, a Mail.app message has a web link, and you want to open it in Internet Explorer 7. Parallels can have a Windows browser specified as the primary browser, but in VMware Fusion, you would have to copy the link and past it into the Windows browser.

Because of the manual steps required for VMware Fusion to accomplish these tasks, Parallels was substantially faster in all cases. The chart in Figure 7 shows the results as measured in seconds (e.g., unlike the other charts, these are actual times, not normalized measurements).

Figure 7. Cross Platform Task Tests under XP
(shorter columns means faster)

Vista vs. XP

Clearly, most people are still using XP, and not Vista. And clearly, Vista doesn't perform anywhere near as fast as XP on the same hardware. But, Vista is ultimately the future for Windows, and over time will gain greater acceptance, and therefore is something that we want to keep an eye on.

Using our Boot Camp configuration as a means for comparison, we found Vista ran 17-30% slower than the equivalent XP setup.

By comparison, when we compared XP configurations against the baseline PC, we got these results:

• XP under Boot Camp averaged 12% faster than the baseline PC running XP

• XP under VMware Fusion averaged 1% faster than the baseline PC running XP

• XP under Parallels averaged 19% faster than the baseline PC running XP

There are three ways to look at how Vista performed in comparison to XP. The first way is to compare VMware Fusion and Parallels to Boot Camp. And the second way, is to compare Vista for each virtual machine compared to the XP performance for that same virtual machine. The third way is to look at the complex task tests which have to do mostly with the user interface of the virtualization environments.

When we compare VMware Fusion and Parallels running Vista to the same model of computer running Vista under Boot Camp, here are the averages that we got:

• Vista under VMware Fusion averaged 46% slower than Vista under Boot Camp

• Vista under VMware Fusion is 44% faster than Vista under Parallels

• Vista under Parallels averaged 110% slower than Vista under Boot Camp

When we compare each of the environments running Vista to the same environment running XP, here are the averages that we got:

• Vista under Boot Camp averaged 24% slower than XP

• Vista under VMware Fusion averaged 32% slower than XP

• Vista under Parallels averaged 85% slower than XP

It's a completely different story when we look at VMware Fusion and Parallels running Vista for the multi-step task tests. Since these tests focus on user interface focusing on the integration of Windows and Mac OS X applications, it's primarily focused on the steps the human has to take, not raw processing. Here, Parallels is consistently 5x faster than VMware Fusion when looking at cross platform task tests in Vista.

Boot Camp, VMware Fusion and Parallels are all very good, each in their own way. You should make your decisions based on what your needs are as a result.

If you don't want Mac integration, and just want to run Windows, go with Boot Camp. It's faster than a PC anyway.

If you want a virtualization product (that allows you to run Windows alongside Mac OS X), and you want the best performance for the types of things that we tested, then clearly you need to run XP and not Vista. Furthermore, in our tests, both VMware Fusion and Parallels performed well, and were a good user experience. That said, Parallels was somewhat faster in general than VMware Fusion for XP.

If you want the best virtualization performance for Vista, then VMware Fusion is your choice. And, if you want to keep your Mac OS X and Windows environments completely separate, VMware Fusion's design may be your better choice. (And, although we didn't test it, we would expect VMware Fusion to have better multi-processor support if you really have an application that is designed to take advantage of it.) If your goal is tight integration between one or more Windows applications and Mac OS X, Parallels is the clear winner when running either XP or Vista. And, as we said before, if you want the best XP performance with the types of applications tested here, Parallels is not only faster than VMware Fusion, but it's faster than Boot Camp on average for the applications that we tested.

While many users are interested in just running a version of Windows, there are many applications for specialized "canned" solutions. These are called "virtual appliances". Clearly, VMware's heritage shows here, and they have a very broad array of solutions. Parallels also has a variety of virtual appliances. For the user, it just comes down to what you are looking for, and which vendor has the solutions you need. For more information on the virtual appliances that VMware offers, see <http://www.vmware.com/appliances/>. And, for more information on the virtual appliances that Parallels offers, see <http://ptn.parallels.com/en/ptn/dir/>

There are additional reasons to choose one over another, including: for example, which user interface you prefer, and whether you frequently run different OS setups in your virtual machine. And, beyond the everyday Windows XP or Vista use, VMware Fusion also excels in the number of OSes supported.

And, given the track record, expect all Parallels and VMware Fusion to both keep getting better and better. Certainly, for XP, both run quite well right now. And, even with differences in benchmark performance, the Vista experience in both is quite usable as well.

Parallels Desktop is available from Parallels (formerly SWsoft). See <http://www.parallels.com> for more information. Parallels Desktop 3.0 for Mac retails for $79.99. Parallels Desktop 3.0 for Mac Premium Edition includes the Desktop version plus 3 additional packages retails for $99.99.

VMware Fusion 1.1 is available from VMware. See http:// <http://www.vmware.com/mac> for more information. VMware Fusion 1.1 retails for $79.99.

Introduction

The purpose of this appendix is to outline the basic parameters for how MacTech Magazine performed benchmarking tests on a native PC, Boot Camp, VMware Fusion, and Parallels Desktop for the purpose of evaluating the performance of virtual machines running Windows XP and Vista.

Consistency

Since the tests involve both multiple machines and multiple pieces of software, the focus was on creating as much consistency across the tests as possible. MacTech accomplished this in several ways.

First, all timing of tests were performed by a single MacTech staff member so as to eliminate any of the natural inconsistencies that often occur across individuals.

All of the tests were performed on the same version of the Mac operating system across the different hardware. At the time MacTech began the tests, Leopard (Mac OS X 10.5) was not yet shipping. At the time of the tests, this was Mac OS X 10.4.10 and included the most up-to-date versions of Apple patches as prescribed through "Software Update" in Mac OS X.

All of the tests were done on "virgin" systems, i.e., freshly wiped hard disks, with fresh Mac OS X and Office installations, with no third party software installed beyond the standard Mac OS X, virtualization software, Windows install, Adobe Reader, and Microsoft Office installations.

All of the tests were performed with the most up to date set of patches for Microsoft Office as prescribed by Microsoft's automatic updates.

Avoiding Interactions

While the tests covered the bulk of the Microsoft Office suite, all tests (where appropriate) were performed with only that single application open. In other words, to the extent possible, no other applications will be running. (Obviously excluding background and OS tasks that are part of a standard install of either OSes or Microsoft Office.)

To avoid issues with a noisy network, the test machines were installed on what was considered a "quiet" network with minimal traffic. MacTech monitored the use of the network to make sure that the machine does have network access, but is not impacted by the network.

Measurements, Testing and Outliers

For timed tests with results under 15 seconds, tests were measured to within 1/100th of a second. For those over 15 seconds, tests were measured to within a second.

Most tests were performed at least three times per test and per machine, and up to 5 times depending on the test. On average, tests were conducted three to four times per machine per test case. Outliers indicating a testing anomaly were retested as appropriate.

In most cases, tester used successive tests, not "Adam" or "first" tests to better emulate the daily use of Office products.

Those tests that could be impacted by the size of the window were tested with the same window size, and screen resolution under all scenarios.

Some tests were eliminated because the machines simply performed too fast to get an accurate measurement. For example, sending or sorting emails always performed faster on the machine than the tester could measure.

In keeping the results fully open, MacTech is making available the test data in the form of an Excel spreadsheet. In these results, all data is normalized to compare with the baseline PC running Windows XP. The results are therefore best thought of in terms of percentages.

The sole exception is the "Cross Platform Task Tests" which have no PC baseline, and are therefore left in raw time (seconds) for each test performed.

To download the spreadsheet of these results, click here for the MacTech ftp site.

The staff of MacTech Magazine is a jolly crew who spend their work time playing with their Macs and their spare time working with their Macs. You can reach them at editorial@mactech.com.