01-The Test Bench and Test Types
TweetFollow Us on Twitter

01-The Test Bench and Test Types

The Test Bench

When we were choosing computer models, we chose the current model of Macs that give a good representation of what most people may have. Certainly, the faster models of these computers will perform even better.

We chose five Mac models to compare alongside each other: MacBook Air, MacBook Pro, a tricked out MacBook Pro Retina, iMac (late 2011), and Mac Pro. Given the RAM requirements of Windows 7, the minimum configuration tested (including on the MacBook) was 4 GB.

8 GB MacBook Air 13-inch, 1.8 GHz dual-core Intel Core i5 processor
Specifically: 8 GB/256 GB, Intel HD Graphics 4000

4 GB MacBook Pro 15.4-inch, 2.3GHz quad-core Intel Core i7 processor
Specifically: 4GB 1600MHz memory/500 GB 5400-rpm
Intel HD Graphics 4000 and NVIDIA GeForce GT 650M with 512MB of GDDR5 memory

16 GB MacBook Pro with Retina 15-inch, 2.7GHz quad-core Intel Core i7 processor
Specifically: 16GB 1600MHz DDR3L SDRAM/768GB Flash Storage
Intel HD Graphics 4000 and NVIDIA GeForce GT 650M with 1GB of GDDR5 memory

4 GB iMac 27", 2.7GHz Quad-Core Intel Core i5 processor
Specifically: 4 GB/1 TB 7200-rpm
AMD Radeon HD 6770M with 512MB

6 GB Mac Pro, One 3.2GHz Quad-Core Intel Xeon processor processors
Specifically: 6 GB/1 TB 7200-rpm / ATI Radeon HD 5770 with 1GB GDDR5

Memory for virtual machines can be configured with a wide array of settings. As a rule, both VMware Fusion's and Parallels Desktop's default memory for each configuration (of physical RAM and "guest" OS) were the same, and we made sure that was the case. Windows 7 and 8 virtual machines ran with 1 GB of virtual machine RAM (except for gaming). Lion and Mountain Lion (OS X) ran with 2 GB of virtual machine RAM. For gaming, we used 1.5 GB for 4 GB hardware, and 2 GB on hardware with 6 GB or more.

Similarly, for disk allocation we set up all of the virtual machines similarly. We used default sized (64 GB on Parallels, and 60 GB on VMware) expanding disks in a single file (e.g., not 2 GB chunks), but the disks were pre-expanded so that expanding actions wouldn't affect results. The disk location of the virtual hard drive was in a similar physical location between computers, as that can make a significant difference in disk performance.

The tests compared VMware Fusion 5.0.2 with Parallels Desktop for Mac 8.0.18314.813278, running on Mac OS X 10.8.2 with all updates. All "important," but not "optional," Windows updates also installed for Windows 7 and 8. By the time we got to testing games, we saw new versions released that could have impact, and therefore (games only) were tested on VMware Fusion 5.0.2 with Parallels Desktop for Mac 8.0.18354.823166, including additional updates to OS X 10.8.2.

Test Types

There are a variety of often referred to, and utilized benchmarking suites in the computer industry including SPEC tests, PCMark, WorldBench, Performance Test, Unixbench and others. Each of these tests uses a series of tests that measure in a consistent way to assess specific types of performances. The key to each of these tests is appropriateness, repeatability, and accuracy.

We are sometimes asked how we select the tests we have in benchmarks. The goal is to represent the types of actions that virtualization users are doing on a regular basis. In particular, we focus more on user tasks rather than installation or configuration (because they are typically only done once or infrequently).

In the PC market, PC World (magazine) uses WorldBench as their benchmarking tool of choice. If you don't know about WorldBench, you can see more at http://www.worldbench.com. WorldBench 7 uses a combination of applications bundled together to determine benchmarks. This may be a very good way for PC World to approach things, but obviously it's not reflecting the virtualization part of the experience or weighing towards the typical types of things people do in virtualization on the Mac.

There are a variety of other benchmarks available, and we've looked at many of them as possible tools to include in our mix. Often, we find their measurements simply don't reflect a true user experience. Other times, they don't reflect how a virtualization user may be using Windows, or even they simply report erroneous results. For example, in previous attempts to use PassMark, we found their graphics scores were not at all representative of real life graphics performance in games or other graphic uses. Those tests showed items as faster when, in fact, they were much slower. Benchmarks are only useful if they reflect the real world.

Rather than use WorldBench or others, we focus on the types of measurements we believe best represent the experience we see (in terms of speed). And while it takes practice and some skill, we test virtualization operations with a stopwatch—as the closest representation of how a real user would see it. CPU crunching ability, for example, is measured through tests like zip'ing files directly in Windows. For Office users, we use the most up to date version of Microsoft Office for Windows (2010), again with updates.

There are two exceptions to this: graphics/gaming and CPU utilization. In these two cases, we found that testing utilities not only work well, but also are necessary to give the most repeatable and concrete results. In the case of graphics benchmarking, we used 3DMark06 (the industry standard for tuning gaming systems). For game playing performance, we used FRAPS to measure frames per second. Finally, for CPU utilization we used "top" combined with Apple Remote Desktop.

Remember, benchmarks are not a review: they are benchmarks. They are simply meant to just tell you which runs faster. If you are interested in a specific feature, support, the user interface, or any of the other criteria for deciding on a product, that's a different article.




Virtualization is just one of the many critical topics at MacTech BootCamp II seminars
Packed schedule. Learn from the best.
Meet new people. Network with peers.


One day semimar for techs and consultants who support both small to medium size business, and larger organizations. Invest in yourself. Click here for more info.
Space is limited. Register today as early bird and save $200.
 
AAPL
$111.78
Apple Inc.
-0.87
MSFT
$47.66
Microsoft Corpora
+0.14
GOOG
$516.35
Google Inc.
+5.25

MacTech Search:
Community Search:

Software Updates via MacUpdate

NeoOffice 2014.6 - Mac-tailored, OpenOff...
NeoOffice is a complete office suite for OS X. With NeoOffice, users can view, edit, and save OpenOffice documents, PDF files, and most Microsoft Word, Excel, and PowerPoint documents. NeoOffice 3.x... Read more
LibreOffice 4.3.5.2 - Free Open Source o...
LibreOffice is an office suite (word processor, spreadsheet, presentations, drawing tool) compatible with other major office suites. The Document Foundation is coordinating development and... Read more
CleanApp 5.0.0 Beta 5 - Application dein...
CleanApp is an application deinstaller and archiver.... Your hard drive gets fuller day by day, but do you know why? CleanApp 5 provides you with insights how to reclaim disk space. There are... Read more
Monolingual 1.6.2 - Remove unwanted OS X...
Monolingual is a program for removing unnecesary language resources from OS X, in order to reclaim several hundred megabytes of disk space. It requires a 64-bit capable Intel-based Mac and at least... Read more
NetShade 6.1 - Browse privately using an...
NetShade is an Internet security tool that conceals your IP address on the web. NetShade routes your Web connection through either a public anonymous proxy server, or one of NetShade's own dedicated... Read more
calibre 2.13 - Complete e-library manage...
Calibre is a complete e-book library manager. Organize your collection, convert your books to multiple formats, and sync with all of your devices. Let Calibre be your multi-tasking digital librarian... Read more
Mellel 3.3.7 - Powerful word processor w...
Mellel is the leading word processor for OS X and has been widely considered the industry standard since its inception. Mellel focuses on writers and scholars for technical writing and multilingual... Read more
ScreenFlow 5.0.1 - Create screen recordi...
Save 10% with the exclusive MacUpdate coupon code: AFMacUpdate10 Buy now! ScreenFlow is powerful, easy-to-use screencasting software for the Mac. With ScreenFlow you can record the contents of your... Read more
Simon 4.0 - Monitor changes and crashes...
Simon monitors websites and alerts you of crashes and changes. Select pages to monitor, choose your alert options, and customize your settings. Simon does the rest. Keep a watchful eye on your... Read more
BBEdit 11.0.2 - Powerful text and HTML e...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more

Latest Forum Discussions

See All

Make your own Tribez Figures (and More)...
Make your own Tribez Figures (and More) with Toyze Posted by Jessica Fisher on December 19th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
So Many Holiday iOS Sales Oh My Goodness...
The holiday season is in full-swing, which means a whole lot of iOS apps and games are going on sale. A bunch already have, in fact. Naturally this means we’re putting together a hand-picked list of the best discounts and sales we can find in order... | Read more »
It’s Bird vs. Bird in the New PvP Mode f...
It’s Bird vs. Bird in the New PvP Mode for Angry Birds Epic Posted by Jessica Fisher on December 19th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Telltale Games and Mojang Announce Minec...
Telltale Games and Mojang Announce Minecraft: Story Mode – A Telltale Games Series Posted by Jessica Fisher on December 19th, 2014 [ permalink ] | Read more »
WarChest and Splash Damage Annouce Their...
WarChest and Splash Damage Annouce Their New Game: Tempo Posted by Jessica Fisher on December 19th, 2014 [ permalink ] WarChest Ltd and Splash Damage Ltd are teaming up again to work | Read more »
BulkyPix Celebrates its 6th Anniversary...
BulkyPix Celebrates its 6th Anniversary with a Bunch of Free Games Posted by Jessica Fisher on December 19th, 2014 [ permalink ] BulkyPix has | Read more »
Indulge in Japanese cuisine in Cooking F...
Indulge in Japanese cuisine in Cooking Fever’s new sushi-themed update Posted by Simon Reed on December 19th, 2014 [ permalink ] Lithuanian developer Nordcurrent has yet again updated its restaurant simulat | Read more »
Badland Daydream Level Pack Arrives to C...
Badland Daydream Level Pack Arrives to Celebrate 20 Million Downloads Posted by Ellis Spice on December 19th, 2014 [ permalink ] | Read more »
Far Cry 4, Assassin’s Creed Unity, Desti...
Far Cry 4, Assassin’s Creed Unity, Destiny, and Beyond – AppSpy Takes a Look at AAA Companion Apps Posted by Rob Rich on December 19th, 2014 [ permalink ] These day | Read more »
A Bunch of Halfbrick Games Are Going Fre...
A Bunch of Halfbrick Games Are Going Free for the Holidays Posted by Ellis Spice on December 19th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »

Price Scanner via MacPrices.net

13-inch 2.6GHz Retina MacBook Pro on sale for...
Best Buy has lowered their price on the 2014 13″ 2.6GHz/128GB Retina MacBook Pro to $1149.99 on their online store for a limited time. That’s $150 off MSRP and the lowest price available for this... Read more
Kodak Returns to CES With New Consumer Produ...
Former photography colossus Kodak is returning to CES for the first time in three years where the Kodak booth (#21818 South Hall 1) will showcase a wide range of innovative, imaging-related products... Read more
Invaluable Launches New Eponymously -Named A...
Invaluable, the world’s largest online live auction marketplace, hhas announced the official launch of the Invaluable app for iPad, now available for download in the iTunes App Store. Invaluable... Read more
IDC Reveals Worldwide Mobile Enterprise Appli...
International Data Corporation (IDC) last week hosted the IDC FutureScape: Worldwide Mobile Enterprise Applications and Solutions 2015 Predictions Web conference. The session provided organizations... Read more
Hello Vino Wine App Launches “Safe Ride Home”...
Hello Vino has announced addition of a new “Get a Safe Ride Home” feature in its Food & Drink app with a direct connection to Uber, the technology platform that connects users with rides. The... Read more
DEVON-technologies Releases DEVONthink To Go...
Coeur d’Alene, Idaho based DEVON-technologies, LLC has updated DEVONthink To Go, its mobile companion to DEVONthink, to version 1.5. The update includes an iOS 8 extension, compatibility with the... Read more
The Apple Store offering free next-day shippi...
The Apple Store is now offering free next-day shipping on all in stock items if ordered before 12/23/14 at 10:00am PT. Local store pickup is also available within an hour of ordering for any in stock... Read more
It’s 1992 Again At Sony Pictures, Except For...
Techcrunch’s John Biggs interviewed a Sony Pictures Entertainment (SPE) employee, who quite understandably wished to remain anonymous, regarding post-hack conditions in SPE’s L.A office, explaining “... Read more
OtterBox Defender Series Case For iPad mini 3...
With their innovative Touch ID technology and ultrathin profile, the latest tranche of Apple iPads are more desirable than ever, and OtterBox has just announced the Defender Series custom-engineered... Read more
Holiday sales this weekend: MacBook Pros for...
 B&H Photo has new MacBook Pros on sale for up to $300 off MSRP as part of their Holiday pricing. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.2GHz Retina MacBook Pro: $1699... Read more

Jobs Board

*Apple* Store Leader Program (US) - Apple, I...
…Summary Learn and grow as you explore the art of leadership at the Apple Store. You'll master our retail business inside and out through training, hands-on experience, Read more
Project Manager, *Apple* Financial Services...
**Job Summary** Apple Financial Services (AFS) offers consumers, businesses and educational institutions ways to finance Apple purchases. We work with national and Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.