01-The Test Bench and Test Types
TweetFollow Us on Twitter

01-The Test Bench and Test Types

The Test Bench

When we were choosing computer models, we chose the current model of Macs that give a good representation of what most people may have. Certainly, the faster models of these computers will perform even better.

We chose five Mac models to compare alongside each other: MacBook Air, MacBook Pro, a tricked out MacBook Pro Retina, iMac (late 2011), and Mac Pro. Given the RAM requirements of Windows 7, the minimum configuration tested (including on the MacBook) was 4 GB.

8 GB MacBook Air 13-inch, 1.8 GHz dual-core Intel Core i5 processor
Specifically: 8 GB/256 GB, Intel HD Graphics 4000

4 GB MacBook Pro 15.4-inch, 2.3GHz quad-core Intel Core i7 processor
Specifically: 4GB 1600MHz memory/500 GB 5400-rpm
Intel HD Graphics 4000 and NVIDIA GeForce GT 650M with 512MB of GDDR5 memory

16 GB MacBook Pro with Retina 15-inch, 2.7GHz quad-core Intel Core i7 processor
Specifically: 16GB 1600MHz DDR3L SDRAM/768GB Flash Storage
Intel HD Graphics 4000 and NVIDIA GeForce GT 650M with 1GB of GDDR5 memory

4 GB iMac 27", 2.7GHz Quad-Core Intel Core i5 processor
Specifically: 4 GB/1 TB 7200-rpm
AMD Radeon HD 6770M with 512MB

6 GB Mac Pro, One 3.2GHz Quad-Core Intel Xeon processor processors
Specifically: 6 GB/1 TB 7200-rpm / ATI Radeon HD 5770 with 1GB GDDR5

Memory for virtual machines can be configured with a wide array of settings. As a rule, both VMware Fusion's and Parallels Desktop's default memory for each configuration (of physical RAM and "guest" OS) were the same, and we made sure that was the case. Windows 7 and 8 virtual machines ran with 1 GB of virtual machine RAM (except for gaming). Lion and Mountain Lion (OS X) ran with 2 GB of virtual machine RAM. For gaming, we used 1.5 GB for 4 GB hardware, and 2 GB on hardware with 6 GB or more.

Similarly, for disk allocation we set up all of the virtual machines similarly. We used default sized (64 GB on Parallels, and 60 GB on VMware) expanding disks in a single file (e.g., not 2 GB chunks), but the disks were pre-expanded so that expanding actions wouldn't affect results. The disk location of the virtual hard drive was in a similar physical location between computers, as that can make a significant difference in disk performance.

The tests compared VMware Fusion 5.0.2 with Parallels Desktop for Mac 8.0.18314.813278, running on Mac OS X 10.8.2 with all updates. All "important," but not "optional," Windows updates also installed for Windows 7 and 8. By the time we got to testing games, we saw new versions released that could have impact, and therefore (games only) were tested on VMware Fusion 5.0.2 with Parallels Desktop for Mac 8.0.18354.823166, including additional updates to OS X 10.8.2.

Test Types

There are a variety of often referred to, and utilized benchmarking suites in the computer industry including SPEC tests, PCMark, WorldBench, Performance Test, Unixbench and others. Each of these tests uses a series of tests that measure in a consistent way to assess specific types of performances. The key to each of these tests is appropriateness, repeatability, and accuracy.

We are sometimes asked how we select the tests we have in benchmarks. The goal is to represent the types of actions that virtualization users are doing on a regular basis. In particular, we focus more on user tasks rather than installation or configuration (because they are typically only done once or infrequently).

In the PC market, PC World (magazine) uses WorldBench as their benchmarking tool of choice. If you don't know about WorldBench, you can see more at http://www.worldbench.com. WorldBench 7 uses a combination of applications bundled together to determine benchmarks. This may be a very good way for PC World to approach things, but obviously it's not reflecting the virtualization part of the experience or weighing towards the typical types of things people do in virtualization on the Mac.

There are a variety of other benchmarks available, and we've looked at many of them as possible tools to include in our mix. Often, we find their measurements simply don't reflect a true user experience. Other times, they don't reflect how a virtualization user may be using Windows, or even they simply report erroneous results. For example, in previous attempts to use PassMark, we found their graphics scores were not at all representative of real life graphics performance in games or other graphic uses. Those tests showed items as faster when, in fact, they were much slower. Benchmarks are only useful if they reflect the real world.

Rather than use WorldBench or others, we focus on the types of measurements we believe best represent the experience we see (in terms of speed). And while it takes practice and some skill, we test virtualization operations with a stopwatch—as the closest representation of how a real user would see it. CPU crunching ability, for example, is measured through tests like zip'ing files directly in Windows. For Office users, we use the most up to date version of Microsoft Office for Windows (2010), again with updates.

There are two exceptions to this: graphics/gaming and CPU utilization. In these two cases, we found that testing utilities not only work well, but also are necessary to give the most repeatable and concrete results. In the case of graphics benchmarking, we used 3DMark06 (the industry standard for tuning gaming systems). For game playing performance, we used FRAPS to measure frames per second. Finally, for CPU utilization we used "top" combined with Apple Remote Desktop.

Remember, benchmarks are not a review: they are benchmarks. They are simply meant to just tell you which runs faster. If you are interested in a specific feature, support, the user interface, or any of the other criteria for deciding on a product, that's a different article.




Virtualization is just one of the many critical topics at MacTech BootCamp II seminars
Packed schedule. Learn from the best.
Meet new people. Network with peers.


One day semimar for techs and consultants who support both small to medium size business, and larger organizations. Invest in yourself. Click here for more info.
Space is limited. Register today as early bird and save $200.
 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Herald 5.0.2 - Notification plugin for M...
Note: Versions 2.1.3 (for OS X 10.7), 3.0.6 (for OS X 10.8), and 4.0.8 (for OS X 10.9) are no longer supported by the developer. Herald is a notification plugin for Mail.app, Apple's Mac OS X email... Read more
DEVONthink Pro 2.8.3 - Knowledge base, i...
Save 10% with our exclusive coupon code: MACUPDATE10 DEVONthink Pro is your essential assistant for today's world, where almost everything is digital. From shopping receipts to important research... Read more
Boom 2 1.0.1 - System-wide pro audio app...
Boom 2 is a system-wide volume booster and equalizer app that is designed especially for OS X 10.10 Yosemite. It comes with a smart interface, self-calibrates itself according to your Mac, offers... Read more
Apple Security Update 2015-001 - For OS...
Apple Security Update 2015-001 is recommended for all users and improves the security of OS X. For detailed information about the security content of this update, please visit: http://support.apple.... Read more
Drive Genius 4.0.1 - Powerful system uti...
Drive Genius 4 gives you faster performance from your Mac while also protecting it. The award-winning and improved DrivePulse feature alerts you to hard drive issues before they become major problems... Read more
Yosemite Cache Cleaner 9.0.3 - Clear cac...
Yosemite Cache Cleaner is an award-winning general purpose tool for OS X. YCC makes system maintenance simple with an easy point-and-click interface to many OS X functions. Novice and expert users... Read more
xScope 4.1.1 - Onscreen graphic measurem...
xScope is powerful set of tools that are ideal for measuring, inspecting, and testing on-screen graphics and layouts. Its tools float above your desktop windows and can be accessed via a toolbar,... Read more
OS X Yosemite 10.10.2 - Apple's lat...
OS X Yosemite is Apple's newest operating system for Mac. An elegant design that feels entirely fresh, yet inherently familiar. The apps you use every day, enhanced with new features. And a... Read more
Apple iOS 8.1.3 - The latest version of...
The latest version of iOS can be downloaded through iTunes. Apple iOS 8 comes with big updates to apps you use every day, like Messages and Photos. A whole new way to share content with your family.... Read more
SpamSieve 2.9.19 - Robust spam filter fo...
SpamSieve is a robust spam filter for major email clients that uses powerful Bayesian spam filtering. SpamSieve understands what your spam looks like in order to block it all, but also learns what... Read more

Storm & Skye – Magical Adventure Sto...
Storm & Skye – Magical Adventure Story For Kids Review By Amy Solomon on January 28th, 2015 Our Rating: iPhone App - Designed for the iPhone, compatible with the iPad The first book in the Storm & Skye series includes... | Read more »
New WOTA: U-Boat Compass ‘Wets’ Your App...
New WOTA: U-Boat Compass ‘Wets’ Your Appetite for WOTA: Wolves of the Atlantic. Posted by Jessica Fisher on January 27th, 2015 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Warship Solitaire Review
Warship Solitaire Review By Nadia Oxford on January 27th, 2015 Our Rating: :: A WAR OF NUMBERSUniversal App - Designed for iPhone and iPad Warship Solitaire might lack frills and looks, but it offers some thought provoking... | Read more »
Keep Track of Your Hectic Work Schedule...
Keep Track of Your Hectic Work Schedule With Shifts Posted by Jessica Fisher on January 27th, 2015 [ permalink ] iPhone App - Designed for the iPhone, compatible with the iPad | Read more »
All Star Quarterback Review
All Star Quarterback Review By Campbell Bird on January 27th, 2015 Our Rating: :: PLAY YOUR PARTUniversal App - Designed for iPhone and iPad Train your way to the Super Bowl in this free-to-play sports management game.   | Read more »
Sentinel 4: Dark Star Adds New Campaign...
Sentinel 4: Dark Star Adds New Campaign and More Posted by Ellis Spice on January 27th, 2015 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Wheel & Deal Review
Wheel & Deal Review By Tre Lawrence on January 27th, 2015 Our Rating: :: SIMPLE FUNUniversal App - Designed for iPhone and iPad A simple arcade shooter that has a retro feel.   | Read more »
Join the NAVY in a Big New Update for WW...
Join the NAVY in a Big New Update for WW2: Sandbox. Strategy & Tactics Posted by Jessica Fisher on January 27th, 2015 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Earthworm Alchemy Review
Earthworm Alchemy Review By Campbell Bird on January 27th, 2015 Our Rating: :: FEED THE WORMUniversal App - Designed for iPhone and iPad Feed this funny looking worm and dodge bombs in this charming, free-to-play arcade game.   | Read more »
Graphic Novel App netwars – The Butterfl...
Graphic Novel App netwars – The Butterfly Attack has Received Some New Episodes Posted by Jessica Fisher on January 27th, 2015 [ permalink ] | Read more »

Price Scanner via MacPrices.net

New Good Management Suite Simplifies Enterpri...
Good Technology has announced the availability of the Good Management Suite, a comprehensive cross-platform solution for organizations getting started with mobile business initiatives. Built on the... Read more
15-inch 2.0GHz Retina MacBook Pro (refurbishe...
The Apple Store has Apple Certified Refurbished previous-generation 15″ 2.0GHz Retina MacBook Pros available for $1489 including free shipping plus Apple’s standard one-year warranty. Their price is... Read more
15-inch 2.5GHz Retina MacBook Pro on sale for...
Amazon has the 15″ 2.5GHz Retina MacBook Pro on sale for $2319.99 including free shipping. Their price is $180 off MSRP, and it’s the lowest price available for this model from any reseller. Read more
iPad Turns Five – The ”Book Mystique
Five years ago this week, the late Steve Jobs took the stage at an Apple special press event to unveil the first Apple iPad — a slab-shaped, hand-holdable device that was to shake the personal... Read more
Sale! 15-inch Retina MacBook Pros for up to $...
B&H Photo has the new 2014 15″ Retina MacBook Pros on sale for $150 off MSRP for a limited time. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.2GHz Retina MacBook Pro: $1889.99... Read more
Apple refurbished Mac minis for up to $150 of...
The Apple Store has restocked Apple Certified Refurbished 2014 Mac minis, with models available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: - 1.4GHz... Read more
Apple Releases New Tranche Of OS Updates For...
Apple has released incremental bugfix and security updates for its most recent Mac and iOS versions. OS X Yosemite 10.10.2 Update Apple says this update includes the following improvements: Resolves... Read more
Samsung To Be Main Supplier Of A9 Chips For i...
Korea’s Maeil Business Newspaper reports that Samsung has beat out its chipmaking rivals, Taiwan’s Qualcomm and TSMC, as supplier of chipsets for the next generations of both its own Galaxy S6 and... Read more
Save up to $340 on iMacs with Apple Certified...
The Apple Store has Apple Certified Refurbished iMacs available for up to $340 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free. These are the best prices on... Read more
AppleCare Protection Plans on sale for up to...
B&H Photo has 3-Year AppleCare Warranties on sale for up to $105 off MSRP including free shipping plus NY sales tax only: - Mac Laptops 15″ and Above: $244 $105 off MSRP - Mac Laptops 13″ and... Read more

Jobs Board

*Apple* Systems Engineer - Pre Sales, Educat...
…is responsible for proactively providing technical expertise to drive sales of Apple solutions into assigned accounts. The SE architects, validates, and assists in Read more
*Apple* Solutions Consultant - Retail Sales...
As an ambassador of the Apple brand, the ASC is accountable for driving sales performance by: Connecting with customers. Discovering customers' needs and values. Showing Read more
*Apple* Solutions Consultant - Retail Sales...
As an ambassador of the Apple brand, the ASC is accountable for driving sales performance by: Connecting with customers. Discovering customers' needs and values. Showing Read more
*Apple* Tree Children's Center is hirin...
Apple Tree Learning Center is Hiring Preschool Teachers! Phoenix Childrens Academy is a national leader in the early childhood education industry and we are currently Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.