01-The Test Bench and Test Types
TweetFollow Us on Twitter

01-The Test Bench and Test Types

The Test Bench

When we were choosing computer models, we chose the current model of Macs that give a good representation of what most people may have. Certainly, the faster models of these computers will perform even better.

We chose five Mac models to compare alongside each other: MacBook Air, MacBook Pro, a tricked out MacBook Pro Retina, iMac (late 2011), and Mac Pro. Given the RAM requirements of Windows 7, the minimum configuration tested (including on the MacBook) was 4 GB.

8 GB MacBook Air 13-inch, 1.8 GHz dual-core Intel Core i5 processor
Specifically: 8 GB/256 GB, Intel HD Graphics 4000

4 GB MacBook Pro 15.4-inch, 2.3GHz quad-core Intel Core i7 processor
Specifically: 4GB 1600MHz memory/500 GB 5400-rpm
Intel HD Graphics 4000 and NVIDIA GeForce GT 650M with 512MB of GDDR5 memory

16 GB MacBook Pro with Retina 15-inch, 2.7GHz quad-core Intel Core i7 processor
Specifically: 16GB 1600MHz DDR3L SDRAM/768GB Flash Storage
Intel HD Graphics 4000 and NVIDIA GeForce GT 650M with 1GB of GDDR5 memory

4 GB iMac 27", 2.7GHz Quad-Core Intel Core i5 processor
Specifically: 4 GB/1 TB 7200-rpm
AMD Radeon HD 6770M with 512MB

6 GB Mac Pro, One 3.2GHz Quad-Core Intel Xeon processor processors
Specifically: 6 GB/1 TB 7200-rpm / ATI Radeon HD 5770 with 1GB GDDR5

Memory for virtual machines can be configured with a wide array of settings. As a rule, both VMware Fusion's and Parallels Desktop's default memory for each configuration (of physical RAM and "guest" OS) were the same, and we made sure that was the case. Windows 7 and 8 virtual machines ran with 1 GB of virtual machine RAM (except for gaming). Lion and Mountain Lion (OS X) ran with 2 GB of virtual machine RAM. For gaming, we used 1.5 GB for 4 GB hardware, and 2 GB on hardware with 6 GB or more.

Similarly, for disk allocation we set up all of the virtual machines similarly. We used default sized (64 GB on Parallels, and 60 GB on VMware) expanding disks in a single file (e.g., not 2 GB chunks), but the disks were pre-expanded so that expanding actions wouldn't affect results. The disk location of the virtual hard drive was in a similar physical location between computers, as that can make a significant difference in disk performance.

The tests compared VMware Fusion 5.0.2 with Parallels Desktop for Mac 8.0.18314.813278, running on Mac OS X 10.8.2 with all updates. All "important," but not "optional," Windows updates also installed for Windows 7 and 8. By the time we got to testing games, we saw new versions released that could have impact, and therefore (games only) were tested on VMware Fusion 5.0.2 with Parallels Desktop for Mac 8.0.18354.823166, including additional updates to OS X 10.8.2.

Test Types

There are a variety of often referred to, and utilized benchmarking suites in the computer industry including SPEC tests, PCMark, WorldBench, Performance Test, Unixbench and others. Each of these tests uses a series of tests that measure in a consistent way to assess specific types of performances. The key to each of these tests is appropriateness, repeatability, and accuracy.

We are sometimes asked how we select the tests we have in benchmarks. The goal is to represent the types of actions that virtualization users are doing on a regular basis. In particular, we focus more on user tasks rather than installation or configuration (because they are typically only done once or infrequently).

In the PC market, PC World (magazine) uses WorldBench as their benchmarking tool of choice. If you don't know about WorldBench, you can see more at http://www.worldbench.com. WorldBench 7 uses a combination of applications bundled together to determine benchmarks. This may be a very good way for PC World to approach things, but obviously it's not reflecting the virtualization part of the experience or weighing towards the typical types of things people do in virtualization on the Mac.

There are a variety of other benchmarks available, and we've looked at many of them as possible tools to include in our mix. Often, we find their measurements simply don't reflect a true user experience. Other times, they don't reflect how a virtualization user may be using Windows, or even they simply report erroneous results. For example, in previous attempts to use PassMark, we found their graphics scores were not at all representative of real life graphics performance in games or other graphic uses. Those tests showed items as faster when, in fact, they were much slower. Benchmarks are only useful if they reflect the real world.

Rather than use WorldBench or others, we focus on the types of measurements we believe best represent the experience we see (in terms of speed). And while it takes practice and some skill, we test virtualization operations with a stopwatch—as the closest representation of how a real user would see it. CPU crunching ability, for example, is measured through tests like zip'ing files directly in Windows. For Office users, we use the most up to date version of Microsoft Office for Windows (2010), again with updates.

There are two exceptions to this: graphics/gaming and CPU utilization. In these two cases, we found that testing utilities not only work well, but also are necessary to give the most repeatable and concrete results. In the case of graphics benchmarking, we used 3DMark06 (the industry standard for tuning gaming systems). For game playing performance, we used FRAPS to measure frames per second. Finally, for CPU utilization we used "top" combined with Apple Remote Desktop.

Remember, benchmarks are not a review: they are benchmarks. They are simply meant to just tell you which runs faster. If you are interested in a specific feature, support, the user interface, or any of the other criteria for deciding on a product, that's a different article.




Virtualization is just one of the many critical topics at MacTech BootCamp II seminars
Packed schedule. Learn from the best.
Meet new people. Network with peers.


One day semimar for techs and consultants who support both small to medium size business, and larger organizations. Invest in yourself. Click here for more info.
Space is limited. Register today as early bird and save $200.
 

Community Search:
MacTech Search:

Software Updates via MacUpdate

The best scanner app on mobile
People always say that the best camera is the one you have with you. Well, the same is true with scanners, and your phone can be a pretty great tool for scanning receipts and other documents while you're on the go. [Read more] | Read more »
MARVEL Avengers Academy guide - How to g...
MARVEL Avengers Academy lets you build your own superhero school and fill it with heroes from the Marvel universe. It can be a little slow going to get your school's attendance up though, so we've gathered together somesome tips to help you do this... | Read more »
Shadow Blade: Reload guide - How to hack...
Shadow Blade: Reload is the kind of action-platformer that would have happily sucked up hours of your time on a console a few years back.Now, you can take it with you wherever you go, and its mobile conversion is not too shabby at all. To help you... | Read more »
Tomb of the Mask guide - How to increase...
Tomb of the Mask is a great endless arcade game from Happymagenta in which quick reflexes and a persistent attitude can go a long way toward earning a top score. Check out these tips to see if you can give yourself an edge on the leaderboards. [... | Read more »
Smooth Operator! (Games)
Smooth Operator! 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: Smooth Operator is a weird, weird two-player kissing game. Squeeze in for 2 player fun on a single iPad, creating awkward... | Read more »
Sinless: Remastered (Games)
Sinless: Remastered 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: | Read more »
_PRISM Guide - How to solve those puzzle...
_PRISM is a rather delightful puzzle game that’s been tailor made for touch screens. While part of the fun is figuring things out as you go along, we thought we’d offer you a helping hand at getting in the right mindset. Don’t worry about messing... | Read more »
Fractal Space (Games)
Fractal Space 1.3.1 Device: iOS Universal Category: Games Price: $.99, Version: 1.3.1 (iTunes) Description: Live the memorable experience of Fractal Space, a unique first person adventure & puzzle game by Haze Games! Will you... | Read more »
Set off on an adventure through the Cand...
Like match three puzzlers? If so, Jelly Blast, the innovative iOS and Android game which launched last year, is worth a look. Jelly Blast sees you head off on an epic adventure through the Candy Kingdom with your friends Lily, Mr. Hare, and Mr.... | Read more »
Ellipsis - Touch. Explore. Survive. (...
Ellipsis - Touch. Explore. Survive. 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: | Read more »

Price Scanner via MacPrices.net

cb Hardcase – Handmade and Premium Protective...
Baden-Baden, Germany based company cb innovations has introduced the new cb Hardcase for iPhone. Featuring fine Italian Premium leather that makes for a unique look and feel, the cb Hardcase... Read more
Sale! B&H Photo offers 12-inch Retina Mac...
B&H Photo has 12″ Retina MacBooks on sale for $300 off MSRP for a limited time. Shipping is free, and B&H charges NY tax only: - 12″ 1.1GHz Gray Retina MacBook: $999 $300 off MSRP - 12″ 1.... Read more
App Annie Reveals Future of the App Economy:...
App Annie, a San Francisco based mobile app data and insights platform, has launched its first comprehensive app economy forecast. This new offering will provide brands, agencies, investors and app... Read more
Apple restocks Certified Refurbished Mac mini...
Apple has restocked Certified Refurbished 2014 Mac minis, with models available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: - 1.4GHz Mac mini: $419 $... Read more
What iPad Pro Still Needs To Make It Truly Pr...
I love my iPad Air 2. So much that I’m grudgingly willing to put up with its compromises and limitations as a production tool in order to take advantage of its virtues. However, since a computer for... Read more
21-inch 3.1GHz 4K on sale for $1399, $100 off...
B&H Photo has the 21″ 3.1GHz 4K iMac on sale $1399 for a limited time. Shipping is free, and B&H charges NY sales tax only. Their price is $100 off MSRP: - 21″ 3.1GHz 4K iMac (MK452LL/A): $... Read more
Apple price trackers, updated continuously
Scan our Apple Price Trackers for the latest information on sales, bundles, and availability on systems from Apple’s authorized internet/catalog resellers. We update the trackers continuously: - 15″... Read more
Save up to $240 with Apple Certified Refurbis...
Apple is now offering Certified Refurbished 12″ Retina MacBooks for up to $240 off the cost of new models. Apple will include a standard one-year warranty with each MacBook, and shipping is free. The... Read more
Apple refurbished 13-inch Retina MacBook Pros...
Apple has Certified Refurbished 13″ Retina MacBook Pros available for up to $270 off the cost of new models. An Apple one-year warranty is included with each model, and shipping is free: - 13″ 2.7GHz... Read more
Apple refurbished Time Capsules available for...
Apple has certified refurbished Time Capsules available for $120 off MSRP. Apple’s one-year warranty is included with each Time Capsule, and shipping is free: - 2TB Time Capsule: $179, $120 off - 3TB... Read more

Jobs Board

Infrastructure Engineer - *Apple* /Mac - Rem...
…part of a team Requires proven problem solving skills Preferred Additional: Apple Certified System Administrator (ACSA) Apple Certified Technical Coordinator (ACTC) Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Simply Mac *Apple* Specialist- Service Repa...
Simply Mac is the largest premier retailer of Apple products in the nation. In order to support our growing customer base, we are currently looking for a driven Read more
*Apple* Reporter - Business Insider, Inc. (U...
Business Insider is looking for a reporter to cover Apple , the biggest and arguably most important company in tech. As our primary Apple reporter, you will: * Read more
Infrastructure Engineer - *Apple* /Mac - Rem...
…part of a team Requires proven problem solving skills Preferred Additional: Apple Certified System Administrator (ACSA) Apple Certified Technical Coordinator (ACTC) Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.