TweetFollow Us on Twitter

Major Releases, Benchmarks, and Feedback

Volume Number: 24 (2008)
Issue Number: 08
Column Tag: Industry

Major Releases, Benchmarks, and Feedback

Benchmarks and how the community contributes

By Neil Ticktin, Editor-in-Chief/Publisher

Complexities

Software is a complex beast these days, and it has been for a while. Just take a look at the storage on your local machine, and you will likely see hundreds of thousands of files and folders, if not a million or more.

OS X, like other OSes, has many different configurations - more than is realistically possible for anyone to test on even for the biggest of companies. As a result, developers have gone into the mode of trying to write the best quality software that they can, do realistic testing, and then focus enormous efforts on listening for problems, and reacting to them. Both Microsoft and Apple are great examples of having done this on the Mac.

But with this approach comes a lot of responsibility on the community, especially those that are technical in the market. As a MacTech reader you are at the core of this market). Like it or not, this responsibility lands largely on your shoulders.

Leopard and Microsoft Office 2008 are both giving us good examples of what to look for, what to expect, and what our responsibilities are.

Benchmarks

MacTech strives to give you quality benchmarks. We've done it for virtualization environments, Rosetta, hard drives, and Microsoft Office. In all of these cases, like anyone else that does high end testing, we do this in "virgin" or "clean" environments. It's the only way to do the testing, as you cannot penalize a product by putting it on a machine that needs to have a disk utility run on it, or has system problems, or a congested network.

Benchmarks are a great tool — they allow you to see, in an "apples to apples" comparison, how well a product performs. As the pros in the industry, you take those benchmarks and start to apply them to your real world setting, as only you know the setups you have. When major releases come out, the distance between a clean setup and yours may be that much more important, and here, your involvement grows.

Let's take Leopard and Microsoft Office 2008 as examples. Both of these are herculean efforts on behalf of their developers, and are massive releases. As anyone experienced will tell you, releases of this size are never perfect. There's no way for the developer to test all permutations of hardware, OS release and other software installed on the machine.

Community Feedback

So, what's a geek to do? Well, that's a good question. When major new releases come out, you should get a copy in your hands as fast as possible, and start to use it yourself. What you should not to is put yourself in a situation where you are supporting others using it, and furthermore, you should not yourself be in a position where you are dependent on it working.

As you are able to experiment with these new versions, you want to understand what, if any, pitfalls you fall into. You should be looking for bugs, incompatibilities and more.

Now here's the important part ...

you need to report them.

This is so much more important today than it was in the past. As the technical community, it's become our responsibility to report the problems so that they can be fixed. If we don't report them, they are not likely to be fixed. Griping about issues on public mailing lists and in blogs doesn't count. Even more important, the bug tracking and reporting tools have become sophisticated enough that they not only provide information on what you are experiencing, but give statistical information to help rank the importance of the problems within the community.

Apple and Mac OS X / Mac OS X Server

As many in the industry will tell you, Apple versions tend to stabilize on the .3 version. For example: Leopard will likely stabilize quite a bit on 10.5.3. It just takes that many interactions of versions and general usage to get the major kinks worked out. That's not to say that there won't be bugs with a .3 version; Historically, the .3 version is the one that generally runs well, and then has incremental improvements from there.

Every time you have an application that unexpectedly quits, a dialog appears asking for permission to send information to Apple. You should use this as much as you can, as it's not only fast, but it gives very specific machine generated information that is very helpful to Apple in understanding the situation that just happened.

In looking through Apple's online support, you'll see a way to give Apple feedback on whether an answer was helpful to you or not. Take the few seconds to let them know if it was helpful or not— over time, it makes a difference.

If you are looking for help on an issue, there are a couple of things that you can do. First, you can use MacTech's "Community Search Engine." Here, we've selected the best web sites in the community, including Apple's support sections, to be a part of the search. So, when you are looking for "rebuild degraded RAID," you'll get only those search results relevant to the Mac and OS X, and not the hundreds of thousands of answers you might see with a general Google search. See the left navigation bar on MacTech.com or http://www.mactech.com/advancedsearch.html — either way don't forget to press the "Mac Community" radio button for that type of search.

Apple uses their "Bug Reporter" system at http://bugreporter.apple.com/ as the primary way to report bugs to Apple. ADC Membership (free registration) is required to use this tool. And, there's a great article on AFP548.com that talks about the best ways to use Apple's system — see http://www.afp548.com/article.php?story=20051102133234150

Microsoft Office

Microsoft also uses a variety of ways to have people report in information. In fact, they appear to have broader methods of interacting than Apple does.

Microsoft has a "Send Feedback" command in the Help menu that redirects users to a page on the Mactopia web site. This information is generally used to plan future products for Microsoft.

See http://www.microsoft.com/mac/suggestions.mspx

Like Apple, the online Help sections on the Microsoft web site also asks, "Was this helpful?" Clicking the "No" or "Somewhat" buttons prompts for slightly more detail. When the community tells them why a topic isn't helpful, they can use that feedback to change the topic online, without having to wait for the next major release for offline content updates. Microsoft calls this "continuous publishing."

If you are looking for help on any of the Microsoft Office products, you should look to the newsgroup forums. Here, there are MVPs (not Microsoft employees) that know the products in depth, and will give you very candid answers. They are particularly good at answering "how to" questions. In addition, Microsoft read the posts looking for problem reports, and if appropriate will compare them to reported issues, or create a new one. Here are the URLs for the web interface into the newsgroup forums:

http://www.officeformac.com/ProductForums/Word/

http://www.officeformac.com/ProductForums/Excel/

http://www.officeformac.com/ProductForums/PowerPoint/

http://www.officeformac.com/ProductForums/Entourage/

http://www.officeformac.com/ProductForums/MacMessenger/

When you first install Office, it asks if you want to be part of the Customer Experience Improvement Program (aka CEIP) which gives Office permission to periodically send statistical data back to Microsoft about how the products are used. No personal information is collected. This is very helpful to Microsoft, if you are game. (The default is off, so you would need to turn it on.)

Finally, Microsoft uses Microsoft Error Reporting (MERP on the Mac, and Watson for Windows), which is for real time error reporting. Like Apple's reporter for Mac OS X, this is an excellent way to quickly send information that is machine generated about the problem that you are experiencing.

Statistical Benefits

What all the larger organizations have in their error reporting is the ability to gather significance statistically from the error reports. For example, there are patterns to problems, and the bug reports are done in a way that Microsoft (and presumably Apple), can compare two bug reports and see if they are the same issue or not. Once that happens, they can determine whether an issue has widespread impact, or is localized to just a few users. In other words, it helps them to prioritize which things need to be fixed first.

Another example is that they can see that certain problems may be on specific OS versions, categories of machines, or even specific models of machines. This not only helps in understanding how to fix the bug, but is also may give insight that it's not a vendors bug.

For example, in the next maintenance release of Leopard, Apple is rumored to be fixing a large number of items. Microsoft may be looking at bug reports for an issue, and may very well find that the problem in question is not one with Office, but with the Operating System instead. Of course, the same thing could happen in the opposite direction where Apple receives bug reports, and finds that it's a third party.

Near Term Expectations

Leopard's next update will likely be the one that fixes so much stuff that those that have been waiting, should now move forward. Naturally, Microsoft is working on updates for Office 2008 as well, presumably called SP1 (Service Pack 1, if they hold true to naming conventions). These will likely to fix many issues that widespread distribution saw, but MacTech may not have with the benchmarks on clean machines.

Why is this expected? Because both these companies have mechanisms in place to listen to customer experiences, problems, and bugs, research them down through statistical and other comparisons, and rank their priority accordingly.

But, all of that doesn't work unless we, the technical community, make sure to use the reporting mechanisms as much as possible.


Neil is the Editor-in-Chief and Publisher of MacTech Magazine. Neil has been in the Mac industry since 1985, has developed software, written documentation, been heading up the magazine since 1992. When Neil writes a review, he likes to put solutions into a real-life scenario and then write about that experience from the user point of view. That said, Neil has a reputation around the office for pushing software to its limits and crashing software/finding bugs. Drop him a line at publisher@mactech.com

 
AAPL
$517.96
Apple Inc.
-3.72
MSFT
$39.75
Microsoft Corpora
+0.57
GOOG
$536.44
Google Inc.
+3.92

MacTech Search:
Community Search:

Software Updates via MacUpdate

Starcraft II: Wings of Liberty 1.1.1.180...
Download the patch by launching the Starcraft II game and downloading it through the Battle.net connection within the app. Starcraft II: Wings of Liberty is a strategy game played in real-time. You... Read more
Sibelius 7.5.0 - Music notation solution...
Sibelius is the world's best-selling music notation software for Mac. It is as intuitive to use as a pen, yet so powerful that it does most things in less than the blink of an eye. The demo includes... Read more
Typinator 5.9 - Speedy and reliable text...
Typinator turbo-charges your typing productivity. Type a little. Typinator does the rest. We've all faced projects that require repetitive typing tasks. With Typinator, you can store commonly used... Read more
MYStuff Pro 2.0.16 - Create inventories...
MYStuff Pro is the most flexible way to create detail-rich inventories for your home or small business. Add items to MYStuff by dragging and dropping existing information, uploading new images, or... Read more
TurboTax 2013.r17.002 - Manage your 2013...
TurboTax guides you through your tax return step by step, does all the calculations, and checks your return for errors and overlooked deductions. It lets you file your return electronically to get... Read more
TrailRunner 3.8.769 - Route planning for...
Note: While the software is classified as freeware, it is actually donationware. Please consider making a donation to help support development. TrailRunner is the perfect companion for runners,... Read more
Flavours 1.1.10 - Create and apply theme...
Flavours is a Mac application that allow users to create, apply and share beautifully designed themes. Classy Give your Mac a gorgeous new look by applying delicious themes! Easy Unleash your... Read more
Spotify 0.9.8.296. - Stream music, creat...
Spotify is a streaming music service that gives you on-demand access to millions of songs. Whether you like driving rock, silky R&B, or grandiose classical music, Spotify's massive catalogue... Read more
SlingPlayer Plugin 3.3.20.475 - Browser...
SlingPlayer is the screen interface software that works hand-in-hand with the hardware inside the Slingbox to make your TV viewing experience just like that at home. It features an array of... Read more
S.M.A.R.T. for USB and FireWire Drives 0...
S.M.A.R.T. for USB and FireWire Drives is a kernel driver for OS X external usb or firewire drives. It extends the standard driver behaviour by providing access to drive smart data. The interface to... Read more

Latest Forum Discussions

See All

148Apps Live on Twitch: Pivvot’s Looper...
On our latest Twitch stream, we’ll be playing a pair of minimalist arcade games, one that just got a big content update in Pivvot, and another that was inspired by it in 15 Coins. Whitaker Trebella, creator of Pivvot, will discuss the new modes... | Read more »
Word Cubes Review
Word Cubes Review By Jordan Minor on April 15th, 2014 Our Rating: :: SQUARESVILLEUniversal App - Designed for iPhone and iPad Word Cubes is fine, but it is barely any different from any other word game.   | Read more »
PAX East 2014 – Desert Fox: The Battle o...
PAX East 2014 – Desert Fox: The Battle of El Alamein is Coming to iOS Soon Posted by Rob Rich on April 15th, 2014 [ permalink ] Shenandoah Studio has become one of the go-to developers for war games on iOS, with | Read more »
Tank of Tanks Review
Tank of Tanks Review By Carter Dotson on April 15th, 2014 Our Rating: :: TANKS A LOT!iPad Only App - Designed for the iPad This multiplayer game played on a single iPad is simple, chaotic fun.   | Read more »
PAX East 2014 – Dungeon of the Endless J...
PAX East 2014 – Dungeon of the Endless Just Might Have a Shot at an iPad Release Posted by Rob Rich on April 15th, 2014 [ permalink ] I think it’s fair to say that | Read more »
SideSwype Review
SideSwype Review By Carter Dotson on April 15th, 2014 Our Rating: :: ON YOUR SIDEUniversal App - Designed for iPhone and iPad SideSwype is a puzzler that takes inspiration from Threes, but becomes its own incredibly fun game.   | Read more »
PAX East 2014 – Bigfoot Hunter Invites P...
PAX East 2014 – Bigfoot Hunter Invites Players on a Wild and Wooly Photo Safari Posted by Rob Rich on April 15th, 2014 [ permalink ] Yeti. Sasquatch. Wendigo. | Read more »
Dungeon Quest Review
Dungeon Quest Review By Cata Modorcea on April 15th, 2014 Our Rating: :: NO STORY, BUT GOOD FUNUniversal App - Designed for iPhone and iPad Dungeon Quest does a lot of things right, but ultimately forgets about one of the core... | Read more »
Tempo AI and Speek Join Forces to “Kill...
Tempo AI and Speek Join Forces to “Kill the Conference Call PIN” Posted by Rob Rich on April 15th, 2014 [ permalink ] Today Tempo AI, makers of Tempo Smart Calendar, and | Read more »
Fighting Fantasy: Starship Traveller Rev...
Fighting Fantasy: Starship Traveller Review By Jennifer Allen on April 15th, 2014 Our Rating: :: A SIGNIFICANT VOYAGEUniversal App - Designed for iPhone and iPad Continuing the release of Fighting Fantasy titles, Starship Traveller... | Read more »

Price Scanner via MacPrices.net

Save $50 on Mac mini Server
B&H Photo has the 2012 Mac mini Server on sale for $949 including free shipping plus NY sales tax only. Their price is $50 off MSRP. Read more
PhatWare’s “Ultimate Writing App For iOS” Ren...
PhatWare Corp. has announced it has renamed its new WritePro word processing app for iPhone and iPad: WritePad Pro. The decision to change the app’s name to leverages the strong brand awareness and... Read more
Full Resolution Photo Editor Tint Mint 1.0 Re...
California based independent developer, Jeffrey Sun, creator of the iOS app Modern Editor, has released Tint Mint, a new photography app for editing enthusiasts. The app costs a dollar, and it packs... Read more
16GB iPad mini (Apple refurbished) available...
The Apple Store has refurbished 1st generation 16GB iPad minis available for $249 including free shipping. Both black and white models are available. Read more
Save $120 on the 27-inch 3.2GHz Haswell iMac
B&H Photo has the 27″ 3.2GHz iMac on sale for $1679.99 including free shipping plus NY sales tax only. Their price is about $120 off MSRP. Read more
Using a Mac Doesn’t Eliminate The Heartbleed...
Low End Mac’s Dan Knight notes that any time you visit a website with an https: prefix or see that secure lock icon on your browser, some type of security software is busy trying to protect your data... Read more
AirPrint Basics Tutorial Posted
A new Apple Knowledge Base article helps get you started using AirPrint, the Apple protocol that enables instant printing from iPad, iPhone, iPod touch, and Mac without the need to install drivers or... Read more
Speed Tips For Running LibreOffice On The Mac
LibreOffice is my favorite of several free, open-source application suites, and the one I have configured on my Mac as my default app for Word documents that one frequently has to deal with. It also... Read more
Snag a 15-inch Retina MacBook Pro for $115 of...
B&H Photo has 2013 15″ Retina MacBook Pros on sale for up to $115 off MSRP for a limited time. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.3GHz Retina MacBook Pro: $2489.99... Read more
MacBook Airs on sale for $50 to $100 off MSRP
Several resellers are offering $50-$100 discounts on 11″ and 13″ MacBook Airs today, including Amazon, Best Buy, B&H, and others. See the breakdown of deals on our MacBook Air Price Tracker,... Read more

Jobs Board

*Apple* Retail - Manager - SoHo - Apple (Uni...
Job SummaryKeeping an Apple Store thriving requires a diverse set of leadership skills, and as a Manager, you're a master of them all. In the store's fast-paced, dynamic Read more
Position Opening at *Apple* - Apple (United...
**Job Summary** Every day, business customers come to the Apple Store to discover what powerful, easy-to-use Apple products can do for them. As a Business Leader, Read more
Position Opening at *Apple* - Apple (United...
…challenges of developing individuals, building teams, and affecting growth across Apple Stores. You demonstrate successful leadership ability - focusing on excellence Read more
Position Opening at *Apple* - Apple (United...
…Summary** As a Specialist, you help create the energy and excitement around Apple products, providing the right solutions and getting products into customers' hands. You Read more
Position Opening at *Apple* - Apple (United...
**Job Summary** The Apple Store is a retail environment like no other - uniquely focused on delivering amazing customer experiences. As an Expert, you introduce people Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.