Test Types
There are a variety of often referred to, and utilized benchmarking suites in the computer industry including SPEC tests, PCMark, WorldBench, Performance Test, Unixbench and others. Each of these tests uses a series of tests that measure in a consistent way to assess specific types of performances. The key to each of these tests is appropriateness, repeatability, and accuracy.
When testing virtual machines, however, using outside utilities within a virtualized environment can provide strange and incorrect results. For example, we saw one testing tool that reported graphics running twice as fast when you could clearly see on the screen that it was running at half the speed. Other tests use older versions of applications. Clearly, if one is going to use testing tools, they need to be carefully picked: the results need to reflect what the user sees in real world conditions.
Instead, as we have in the past, MacTech chose to create a suite of tests that would mimic what many users would do in normal use, but stick to those actions that were repeatable, and measurable with a stopwatch. After all, if you couldn’t perceive a difference with a stopwatch, the user is not likely to perceive it at all.
There are two exceptions to this: graphics/gaming and CPU utilization. In these two cases, we found that testing utilities not only work well, but also are necessary to give the most repeatable and concrete results. This time, we had the opportunity to use a tool that measures frames per second to verify the numbers reported by 3DMark06.
To that end, there are several kinds of tests that we ran: Virtual Machine Launch/Suspend, CPU tests, File and Network IO, CPU Usage Footprint, Application Performance, 3D and verification of real world game play.
Overview
We won’t keep you in suspense. When we look at the "big picture" of all the testing, Parallels is the clear winner. If you count up the general tests (including one 3D graphics score), Parallels won 61% of the tests by 10% or more, and was also a bit faster on an additional 23% more of the tests. In other words, Parallels Desktop 6 beat VMware Fusion 3.1 in 84% of the general tests we ran.
Test Tally: General Virtualization Tests
If you focus exclusively on 3D graphics, as measured by 3DMark06 version 1.2, Parallels won by an even larger margin. Specifically, Parallels won 73% of the tests by 10% or more, and was also a bit faster on an additional 19% more of the tests. In other words, Parallels Desktop 6 beat VMware Fusion 3.1 in 92% of the 3D graphics tests we ran.
When VMware Fusion was faster in 3D graphics, it was typically on the HDR/SM3.0 Score and the Batch Size tests (e.g., triangles). But, as you can see from both the test tally, and the colored cell worksheet overview, Parallels Desktop was overwhelmingly faster in graphics, and in working with real games, it was easy to see confirmation of the 3DMark06 scores.
Test Tally: 3D Graphics Tests
There are a handful of places that VMware Fusion consistently was faster than Parallels Desktop. In Microsoft Word 2010, VMware launched (initial launch after boot, aka "Adam" launch) faster, and was faster on large, global search and replaces. It was also consistently faster under Windows 7 doing file compression, and loading SSL pages.
Overall, VMware Fusion won 6% of the tests by at least 10%, and was also a bit faster on an additional 4% more of the general tests. For the 3D tests, VMware Fusion won 3% of the tests by at least 10%, and was also a bit faster on an additional 3% more for a total of 6%.
One of the best ways to visualize the huge amount of data points is through MacTech‘s well-known "Colored Cell Worksheet Overview." In these worksheets, a cell represents the best result for each test for each version of Windows for each virtualization product. These are then colored according to which product was faster.
| Packed schedule. Learn from the best. Meet new people. Network with peers. One day semimar for techs and consultants who support home users, small office, and small to medium size business. Invest in yourself. Click here for more info. |
 |