Office 2004 Benchmarks on Intel-based Macs
Volume Number: 22 (2006)
Issue Number: 5
Column Tag: MacTech Labs
Office 2004 Benchmarks on Intel-based Macs
How well does Office 2004 run on Rosetta technology?
by the MacTech Editorial Staff
The Big Question
If you are a Microsoft Office user on the Mac, there's likely a question on your mind if you are considering purchasing a new Mac based on the Intel processor. You see, currently, Microsoft Office is not "Universal." In other words, it runs on top of Rosetta (Apple's technology to dynamically translate PowerPC-based applications to work on Intel-based Macs). Now, Microsoft has announced that the next version of Office will be Universal, but no one expects this imminently. (It's a big job, Apple released their Intel-based machines earlier than expected, and Office isn't borne from Xcode.)
The big question is this: Does Microsoft Office 2004 run well enough on the new Intel-based Macs? Or should you delay your purchase of these machines?
The Test Bench
We chose three machines to compare. Our baseline machine is a PowerBook G4 15-inch, running a 1.5 GHz PowerPC G4, with 2 GB RAM, and an 80 GB/5400 rpm hard disk.
We compared a MacBook Pro 15-inch, running a 1.83 GHz Intel Core Duo, with 2 GB RAM, and a 120GB/5400 rpm hard disk.
And finally, we compared an iMac 20-inch, running a 2.0 GHz Intel Core Duo, 1.5 GB RAM, and a 250GB 7200-rpm Serial ATA hard disk.
For those interested in the benchmarking methodologies, see the more detailed testing information in Appendix A. For the detailed results of the tests used for the analysis, measured in seconds, see Appendix B. Both appendices are available on the MacTech web site.
Overview
We won't keep you in suspense. In general, Office 2004 under Rosetta works "well enough" to "very well," and in some cases, it's even faster than on the PowerPC machine.
To determine this, MacTech ran over a thousand tests across three models of Macs, and the four major Office applications: Word, Excel, PowerPoint and Entourage. And, since graphics code is shared between Office applications, we ran a suite of graphics tests as well. These are each covered in more detail below.
In one of the most critical set of tests, we specifically looked at whether the user could type or interact faster than Office could keep up, and even in the slowest of scenarios, we never found the user waiting for typing, or other interactions like selecting menus. Even when typing at over 100 wpm, Word was able to stay ahead of the user.
Of the four applications, PowerPoint, is the one that struggled the most. It appears this is due to Office graphics engine shared by all of the Office applications.
At the other end of the spectrum, Entourage was not only on par, it was faster in many cases than our PowerPC baseline. In fact, with the exception of launching the application, Entourage was faster across the board on the Intel iMac, while the MacBook Pro was about on par with the PowerBook G4 (slightly faster in some cases, slightly slower in others).
What was interesting is that even on those tests that were slow, they were still acceptable. For example, the test with the worst performance was inserting a large JPG (over 10mb) into a document was considerably slower. In most cases, a 10 megabyte JPEG would be larger than an 8.5x11 page. How often do users insert large JPEGs in a single sitting? And, when they do, does it matter when that something takes 10 seconds more? Not in our opinion ... and we use Word a lot at MacTech.
It appeared to us that the more modern the application, the better it did under Rosetta. Furthermore, the more that something used the underlying Mac OS, the better it did as well. For example, Entourage did very well with networking related items. Raw imports and opening did significantly better especially when a faster disk was involved.
And, the more times you did a function, the better it performed especially on the second iteration of a command. While it's difficult to confirm, this is due to a combination of code working smarter with caches, both in Rosetta as well as within the Office code base. An obvious benefit for those tasks that are most sensitive to time: the repetitive ones.
The Test Suite and Results
The tests used were selected specifically to give a real-world view of what Microsoft Office 2004 is like to run. We eliminated those tests that we ran that were so short a time frame (e.g., fast) that we could not create statistically significant results, or that had imperceivable differences.
We did one test suite for each of the four major applications in Office, as well as a series of tests focused on the graphics libraries. As you may or may not know, Office 2004 uses a common set of graphics libraries across the applications in Office. With that in mind, we considered a set of tests that would test these technologies, and they are a good representation across all the applications in Office. For example, when you import a graphic, it should be the same across the Office suite.
To give users a good idea of what it's like to use each of these apps, we came up with a list of tests that represented what we felt were the most relevant to regular use. For example, we ran tests on application launches, but did not focus on them since it's something that people tend to do only a couple of times a day. But, we did include the most repetitive of tasks within an application, as those affect productivity most.
The one thing that we did see across the board is that the Intel iMac is consistently faster than the MacBook Pro. Since the iMac had slightly less RAM, and the processor and front side bus speeds are the same, we looked further inside. The iMac has some pretty serious sub-systems design to make it a screaming machine (see <http://developer.apple.com/documentation/HardwareDrivers/Conceptual/iMac_06Jan/>. When you take into account that an iMac has SATA (instead of ATA), a faster hard drive (a 7200 vs. the MacBook Pro's 5400 rpm), and a variety of optimized sub-systems that would be more difficult to implement on a laptop, it's easy to see why the iMac is faster.
It's important to realize that many of the actions that users do when using these applications are so fast already, that even a degradation of 50% may not even be noticeable for most tasks. And in the non-repetitive tasks, they are nearly irrelevant. For example, if when launching an application, you have to wait several seconds, many users will notice, but it won't matter to their overall productivity.
Other areas, like repetitive tasks or editing actions, are far more important, and speed makes a great deal more difference in not only perception, but in productivity.
An overview of tests that best represents productivity is displayed in Figure 1. These metrics include a variety of tests normalized to give you an overview. Smaller bars are faster (better). Information on what is included in each of these overview metrics, as well as additional testing information and graphics are shown later for each application.
Figure 1: Office 2004, Average User Tests
Word
Of all the applications, Word is probably the most widely used by people. The most relevant tests to most Word users are those that are repeated throughout the use of Word. For productivity tests, we selected a variety of scrolling tests, saving, word count, find & replace, opening files, pasting and printing.
The end result is that, for productivity tasks, Word on MacBook Pro performed at 82% of the speed of a PowerBook G4, and on an Intel iMac at 92%. In a real life scenario, these numbers put Word at working pretty well. There's not much a perceptual difference for most tasks.
Our top level grouping included a application launching, variety of scrolling tests, saving, word counts, find & replace, opening documents, pasting and printing. The results are in Figure 2.
Figure 2: Word Benchmark Results
But, of course, the most important of the tests - typing - were so fast that we couldn't measure them. No matter what the computer model, Word was ahead of the typist (even at 100 wpm).
The areas that Word performed "well enough" in are the initial launch of the application launching and printing. Fortunately, these are things that most users do relatively infrequently and don't materially affect productivity.
The balance of the tests ran on par, or in some cases faster on the Intel iMac. As is consistent throughout, the Intel iMac was faster than the MacBook Pro, and in some cases (most notably saving and pasting), the iMac was faster than the PowerBook baseline.
Our one and only crash during the entire Office testing set was during a find and replace in Word. A quick re-launch, and everything proceeded as expected.
The end result is that while there are a couple of areas that a user may notice it's slower, Word works well on a MacBook Pro; and, on the Intel iMac, it works very well, under Rosetta.
Excel
For Excel, we wanted to focus again on the most repetitive tasks. For productivity tests, we selected fill range, scrolling vertically, zooming out, arranging windows, subtotals, auto-formatting, and editing in-cell. On average for the productivity tests, Excel performed at 83% of baseline for the MacBook Pro, and 91% for the Intel iMac.
Most users use relatively small spreadsheets, but it's important to take a look at what a more experienced spreadsheeter may experience. As a general rule, editing and calculations were very fast and in some cases, faster than on the baseline PowerPC.
For our top level testing, we looked at the common commands including fill range, vertical scroll, zoom out, arranging windows, subtotals, auto-format, and editing in-cell. The results are displayed in Figure 3.
Figure 3: Excel Benchmark Results
Scrolling did leave a bit to be desired, but it definitely worked well enough, even on the slower MacBook Pro. In our tests, we used a very large Excel document, and even so, the test ranges were 6.18 to 10.15 seconds. Certainly enough to perceive, but given the very large size of the test document, under normal circumstances, it's not enough to affect productivity.
The end result is that for Excel on a MacBook Pro, it works very well, and on the Intel iMac, even better. Users may notice slower scrolling, but that's about it as far as the most common tasks.
PowerPoint
For PowerPoint, we chose a variety of tests to represent productivity including opening, saving, scrolling slides in the sorter view, adding slides, viewing slide shows, slide transitions, text animations, printing, inserting images from the ClipArt Gallery, changing color schemes and applying templates. As we already mentioned, PowerPoint was the slowest of the Office suite, but in our most indicative tests for productivity, it did perform at 77% of the speed for the MacBook Pro, and nearly 90% for the Intel iMac.
In our top level testing, we took a look at launching, opening files, saving, scrolling slides in sorter view, adding slides, viewing slide transitions, viewing text animations, printing, changing color schemes and applying templates. The results are shown in Figure 4.
Figure 4: PowerPoint Benchmark Results
The areas PowerPoint was weakest were with complex files, printing and applying templates. That said, PowerPoint did pretty well with view slides and transitions, animations, and scrolling in the sorter view.
The bottom line is that even if you use PowerPoint regularly, it definitely works well enough for most people, and in the most important areas ... the presentation itself ... it works very well. If you are a heavy PowerPoint user, you will notice sluggishness, and you should take a closer look.
Entourage
For Entourage, the productivity tests included IMAP account sync, empty deleted items, opening messages, pop message download, opening folders, sending messages via SMTP, sorting and grouping 2000 messages, and searching the address book, mail and tasks. For the productivity tests, Entourage shined with the MacBook Pro performing at 92% of the baseline speed, and the Intel iMac performing 14% faster than the baseline PowerPC.
In our top level testing, we took a look at launch time, empty deleted items, POP message download, sending SMTP messages, sorting and grouping thousands of messages, and searching the address book, mail and tasks. Results are shown in Figure 5.
Figure 5: Entourage Benchmark Results
Like Word, the most important of the tests, typing, were so fast that we couldn't measure them regardless of which test machine.
Boot time was clearly slower - about double in fact. But in a game of seconds, and for something that you generally only do a couple of times a day, it's not particularly relevant.
Entourage is clearly the best performer in the suite, presumably because Entourage is the most modern code base of the Office 2004 suite, and because it relies the most on Mac OS X technologies that have already been made Universal.
The end result is that, across the board, Entourage under Rosetta performed terrifically. The MacBook Pro usually performed as good or better than the PowerBook. The Intel iMac was faster in almost every test.
Microsoft Office Graphics
The weakest part of Office under Rosetta is the graphics code shared across Microsoft Office. PowerPoint shows this most given its nature.
In reality, however, this affects the overall Office experience relatively little. Furthermore, for the graphics libraries, our tests were designed to point out weaknesses and give us measurable results. But let's be real: how often do you import a 10-megabyte JPEG? Most JPEG's that size would be much larger than a full page.
That said, we focused on the tests on inserting large graphics, opening, saving, printing, working with the WordArt features, and the ClipArt library.
The end result is that inserting JPEG's are slow, and we saw the impact across all the Office applications. Fortunately, most users don't tend to do this repetitively. Other formats like inserting EPS and PSD files did much better. In other words, you'll have no issues with your logos and template graphics.
Opening graphics files was ok, but fortunately, saving files was significantly faster (a good thing since users typically save much more often than open files). The ClipArt interfaces work very well including the importing and searching.
Figure 6: Office Graphics Benchmark Results
Conclusion
As we stated at the beginning of this article, in general, Office 2004 under Rosetta works "well enough" to "very well," and in some cases, it's even faster than on the PowerPC baseline machine.
Given the amount we were pushing these apps, we were thoroughly pleased with the stability of the entire Office suite of applications. Considering how complex Rosetta is, and how big a code base Office is, it's entirely remarkable how stable we found it. Furthermore, we expect to see Apple further optimize Rosetta, particularly in launch times.
While Microsoft has already announced that it will be making the next version of Office a "Universal" application, Mac users wishing to dive into Intel-based Macs now can rest assured in knowing they can move forward and be patient for the Universal version.
Appendix A: Benchmarking Methodology
Introduction
The purpose of this appendix is to outline the basic parameters for how MacTech Magazine performed benchmarking tests on Microsoft Office 2004 for Intel iMacs and the MacBook Pro compared to a PowerBook G4.
Benchmarking Methodology
Consistency
Since the tests involve both multiple machines and multiple pieces of software, the focus was on creating as much consistency across the tests as possible. MacTech accomplished this in several ways.
First, all timing of tests were performed by a single MacTech staff member so as to eliminate any of the natural inconsistencies that often occur across individuals.
All of the tests were performed on the same version of the Mac operating system across the different hardware. At the time of the tests, this was Mac OS X 10.4.5 and included the most up-to-date versions of Apple patches as prescribed through "Software Update" in Mac OS X.
All of the tests were done on "virgin" systems, i.e., freshly wiped hard disks, with fresh Mac OS X and Office installations, with no third party software installed beyond the standard Apple and Microsoft Office installations.
All of the tests were performed with the most up to date set of patches for Microsoft Office as prescribed by Microsoft's Auto-updater. At the time of the test, this included the Microsoft Office 2004 for Mac 11.2.3 Update.
Avoiding Interactions
While the tests covered the bulk of the Microsoft Office suite, all tests (where appropriate) were performed with only that single application within Microsoft Office open. In other words, to the extent possible, no other applications will be running. (Obviously excluding background and OS tasks that are part of a standard install of Mac OS X or Office.)
To avoid issues with a noisy network, the test machines were installed on what was considered a "quiet" network with minimal traffic. MacTech monitored the use of the network to make sure that the machine does have network access, but is not impacted by the network.
Measurements, Testing and Outliers
For timed tests with results under 15 seconds, tests were measured to within 1/10th of a second. For those over 15 seconds, tests were measured to within a second.
Most tests were performed at least three times per test and per machine, and up to 5 times depending on the test. On average, tests were conducted three to four times per machine per test case. Outliers indicating a testing anomaly were retested as appropriate.
In most cases, tester used successive tests, not "Adam" or "first" tests to better emulate the daily use of Office products.
Those tests that could be impacted by the size of the window, were tested with the same window size under all scenarios.
Some tests were eliminated because the machines simply performed too fast to get an accurate measurement. For example, typing always displayed faster on the machine than the tester could type (even at 100 wpm).
Appendix B: Testing Results
Introduction
The below chart contains the raw data for the tests selected to judge Office 2004 applications. All results are in seconds, and represent the best time for each test on each machine.
PowerBook G4 MacBook Pro Intel iMac
Word:
Scroll Real World Doc 8.12 9.47 8.34
Scroll Doc with Tables 3.47 4.13 3.91
Scroll Doc with Graphics 5.50 7.22 6.72
Save As Word Doc 1.19 1.22 1.12
Word Count 1.69 2.22 1.84
Find & Replace 0.85 0.93 0.88
Open Word Doc 1.28 1.59 1.34
Paste 1.84 1.94 1.75
Print a Word Doc 7.24 11.00 10.10
Excel:
Fill Range 1.41 1.66 1.68
Scroll Vertical 6.18 10.15 8.97
Zoom Out 1.25 1.56 1.47
Window Arrange 2.25 2.56 2.15
Subtotals 1.44 1.47 1.16
AutoFormat 0.94 1.06 1.00
Edit in-cell 1.44 1.57 1.47
PowerPoint:
Open typical simple file 1.47 1.78 1.56
Save complex file 1.40 2.10 1.78
Scroll slide in sorter view 2.72 3.25 3.00
Add slides in sorter view 2.16 2.56 2.47
View slide show 1.19 1.12 1.16
View slide transitions 0.82 0.85 0.88
View text animations 0.81 0.85 0.66
Print simple file 1.53 2.25 1.94
Insert from Clip Art Gallery 1.34 1.94 1.50
Change color scheme 0.88 1.13 0.72
Apply Template 2.59 4.88 4.38
Entourage:
IMAP Account Sync 1.84 2.31 1.57
Empty Deleted Items 5.25 6.50 4.06
Open Messages 3.03 4.46 3.65
POP Message Download 63.00 73.00 57.00
Open Folder 0.69 0.69 0.56
Sending Messages SMTP 1.35 1.19 1.16
Sorting 2000 messages 0.81 0.81 0.78
Grouping 2000 messages 1.78 1.65 1.50
Address Book Search 1.06 1.06 0.85
Mail Search 0.56 0.59 0.53
Task Search 0.68 0.62 0.47
Graphics:
Insert Large JPG 2.40 12.22 11.12
Insert Large EPS 25.56 37.81 21.87
Open Document 1.19 1.94 1.68
Save Document 2.81 3.59 2.07
Print Document 55.00 100.00 91.00
WordArt: Insert 0.84 1.16 1.06
ClipArt: Import 98.00 89.00 49.00
ClipArt: Search 0.50 0.53 0.44
