TweetFollow Us on Twitter

Major Releases, Benchmarks, and Feedback

Volume Number: 24 (2008)
Issue Number: 08
Column Tag: Industry

Major Releases, Benchmarks, and Feedback

Benchmarks and how the community contributes

By Neil Ticktin, Editor-in-Chief/Publisher

Complexities

Software is a complex beast these days, and it has been for a while. Just take a look at the storage on your local machine, and you will likely see hundreds of thousands of files and folders, if not a million or more.

OS X, like other OSes, has many different configurations - more than is realistically possible for anyone to test on even for the biggest of companies. As a result, developers have gone into the mode of trying to write the best quality software that they can, do realistic testing, and then focus enormous efforts on listening for problems, and reacting to them. Both Microsoft and Apple are great examples of having done this on the Mac.

But with this approach comes a lot of responsibility on the community, especially those that are technical in the market. As a MacTech reader you are at the core of this market). Like it or not, this responsibility lands largely on your shoulders.

Leopard and Microsoft Office 2008 are both giving us good examples of what to look for, what to expect, and what our responsibilities are.

Benchmarks

MacTech strives to give you quality benchmarks. We've done it for virtualization environments, Rosetta, hard drives, and Microsoft Office. In all of these cases, like anyone else that does high end testing, we do this in "virgin" or "clean" environments. It's the only way to do the testing, as you cannot penalize a product by putting it on a machine that needs to have a disk utility run on it, or has system problems, or a congested network.

Benchmarks are a great tool — they allow you to see, in an "apples to apples" comparison, how well a product performs. As the pros in the industry, you take those benchmarks and start to apply them to your real world setting, as only you know the setups you have. When major releases come out, the distance between a clean setup and yours may be that much more important, and here, your involvement grows.

Let's take Leopard and Microsoft Office 2008 as examples. Both of these are herculean efforts on behalf of their developers, and are massive releases. As anyone experienced will tell you, releases of this size are never perfect. There's no way for the developer to test all permutations of hardware, OS release and other software installed on the machine.

Community Feedback

So, what's a geek to do? Well, that's a good question. When major new releases come out, you should get a copy in your hands as fast as possible, and start to use it yourself. What you should not to is put yourself in a situation where you are supporting others using it, and furthermore, you should not yourself be in a position where you are dependent on it working.

As you are able to experiment with these new versions, you want to understand what, if any, pitfalls you fall into. You should be looking for bugs, incompatibilities and more.

Now here's the important part ...

you need to report them.

This is so much more important today than it was in the past. As the technical community, it's become our responsibility to report the problems so that they can be fixed. If we don't report them, they are not likely to be fixed. Griping about issues on public mailing lists and in blogs doesn't count. Even more important, the bug tracking and reporting tools have become sophisticated enough that they not only provide information on what you are experiencing, but give statistical information to help rank the importance of the problems within the community.

Apple and Mac OS X / Mac OS X Server

As many in the industry will tell you, Apple versions tend to stabilize on the .3 version. For example: Leopard will likely stabilize quite a bit on 10.5.3. It just takes that many interactions of versions and general usage to get the major kinks worked out. That's not to say that there won't be bugs with a .3 version; Historically, the .3 version is the one that generally runs well, and then has incremental improvements from there.

Every time you have an application that unexpectedly quits, a dialog appears asking for permission to send information to Apple. You should use this as much as you can, as it's not only fast, but it gives very specific machine generated information that is very helpful to Apple in understanding the situation that just happened.

In looking through Apple's online support, you'll see a way to give Apple feedback on whether an answer was helpful to you or not. Take the few seconds to let them know if it was helpful or not— over time, it makes a difference.

If you are looking for help on an issue, there are a couple of things that you can do. First, you can use MacTech's "Community Search Engine." Here, we've selected the best web sites in the community, including Apple's support sections, to be a part of the search. So, when you are looking for "rebuild degraded RAID," you'll get only those search results relevant to the Mac and OS X, and not the hundreds of thousands of answers you might see with a general Google search. See the left navigation bar on MacTech.com or http://www.mactech.com/advancedsearch.html — either way don't forget to press the "Mac Community" radio button for that type of search.

Apple uses their "Bug Reporter" system at http://bugreporter.apple.com/ as the primary way to report bugs to Apple. ADC Membership (free registration) is required to use this tool. And, there's a great article on AFP548.com that talks about the best ways to use Apple's system — see http://www.afp548.com/article.php?story=20051102133234150

Microsoft Office

Microsoft also uses a variety of ways to have people report in information. In fact, they appear to have broader methods of interacting than Apple does.

Microsoft has a "Send Feedback" command in the Help menu that redirects users to a page on the Mactopia web site. This information is generally used to plan future products for Microsoft.

See http://www.microsoft.com/mac/suggestions.mspx

Like Apple, the online Help sections on the Microsoft web site also asks, "Was this helpful?" Clicking the "No" or "Somewhat" buttons prompts for slightly more detail. When the community tells them why a topic isn't helpful, they can use that feedback to change the topic online, without having to wait for the next major release for offline content updates. Microsoft calls this "continuous publishing."

If you are looking for help on any of the Microsoft Office products, you should look to the newsgroup forums. Here, there are MVPs (not Microsoft employees) that know the products in depth, and will give you very candid answers. They are particularly good at answering "how to" questions. In addition, Microsoft read the posts looking for problem reports, and if appropriate will compare them to reported issues, or create a new one. Here are the URLs for the web interface into the newsgroup forums:

http://www.officeformac.com/ProductForums/Word/

http://www.officeformac.com/ProductForums/Excel/

http://www.officeformac.com/ProductForums/PowerPoint/

http://www.officeformac.com/ProductForums/Entourage/

http://www.officeformac.com/ProductForums/MacMessenger/

When you first install Office, it asks if you want to be part of the Customer Experience Improvement Program (aka CEIP) which gives Office permission to periodically send statistical data back to Microsoft about how the products are used. No personal information is collected. This is very helpful to Microsoft, if you are game. (The default is off, so you would need to turn it on.)

Finally, Microsoft uses Microsoft Error Reporting (MERP on the Mac, and Watson for Windows), which is for real time error reporting. Like Apple's reporter for Mac OS X, this is an excellent way to quickly send information that is machine generated about the problem that you are experiencing.

Statistical Benefits

What all the larger organizations have in their error reporting is the ability to gather significance statistically from the error reports. For example, there are patterns to problems, and the bug reports are done in a way that Microsoft (and presumably Apple), can compare two bug reports and see if they are the same issue or not. Once that happens, they can determine whether an issue has widespread impact, or is localized to just a few users. In other words, it helps them to prioritize which things need to be fixed first.

Another example is that they can see that certain problems may be on specific OS versions, categories of machines, or even specific models of machines. This not only helps in understanding how to fix the bug, but is also may give insight that it's not a vendors bug.

For example, in the next maintenance release of Leopard, Apple is rumored to be fixing a large number of items. Microsoft may be looking at bug reports for an issue, and may very well find that the problem in question is not one with Office, but with the Operating System instead. Of course, the same thing could happen in the opposite direction where Apple receives bug reports, and finds that it's a third party.

Near Term Expectations

Leopard's next update will likely be the one that fixes so much stuff that those that have been waiting, should now move forward. Naturally, Microsoft is working on updates for Office 2008 as well, presumably called SP1 (Service Pack 1, if they hold true to naming conventions). These will likely to fix many issues that widespread distribution saw, but MacTech may not have with the benchmarks on clean machines.

Why is this expected? Because both these companies have mechanisms in place to listen to customer experiences, problems, and bugs, research them down through statistical and other comparisons, and rank their priority accordingly.

But, all of that doesn't work unless we, the technical community, make sure to use the reporting mechanisms as much as possible.


Neil is the Editor-in-Chief and Publisher of MacTech Magazine. Neil has been in the Mac industry since 1985, has developed software, written documentation, been heading up the magazine since 1992. When Neil writes a review, he likes to put solutions into a real-life scenario and then write about that experience from the user point of view. That said, Neil has a reputation around the office for pushing software to its limits and crashing software/finding bugs. Drop him a line at publisher@mactech.com

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

OmniPlan Pro 3.6 - Professional-grade pr...
With OmniPlan Pro, you can create logical, manageable project plans with Gantt charts, schedules, summaries, milestones, and critical paths. Break down the tasks needed to make your project a success... Read more
Little Snitch 3.7.1 - Alerts you about o...
Little Snitch gives you control over your private outgoing data. Track background activity As soon as your computer connects to the Internet, applications often have permission to send any... Read more
iMazing 2.1.3 - Complete iOS device mana...
iMazing (was DiskAid) is the ultimate iOS device manager with capabilities far beyond what iTunes offers. With iMazing and your iOS device (iPhone, iPad, or iPod), you can: Copy music to and from... Read more
AppDelete 4.3.1 - $7.99
AppDelete is an uninstaller that will remove not only applications but also widgets, preference panes, plugins, and screensavers along with their associated files. Without AppDelete these associated... Read more
FileZilla 3.23.0.2 - Fast and reliable F...
FileZilla (ported from Windows) is a fast and reliable FTP client and server with lots of useful features and an intuitive interface. Version 3.23.0.2: Bug Fixes and Minor Changes Speed up icon... Read more
PDFpen 8.3 - $74.95
PDFpen allows users to easily edit PDF's. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Features... Read more
TunnelBear 3.0.8 - Subscription-based pr...
TunnelBear is a subscription-based virtual private network (VPN) service and companion app, enabling you to browse the internet privately and securely. Features Browse privately - Secure your data... Read more
Safari Technology Preview 10.1 - The new...
Safari Technology Preview contains the most recent additions and improvements to WebKit and the latest advances in Safari web technologies. And once installed, you will receive notifications of... Read more
Ableton Live 9.7.1 - Record music using...
Ableton Live lets you create and record music on your Mac. Use digital instruments, pre-recorded sounds, and sampled loops to arrange, produce, and perform your music like never before. Ableton Live... Read more
BetterTouchTool 1.963 - Customize Multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more

Latest Forum Discussions

See All

Bully: Anniversary Edition (Games)
Bully: Anniversary Edition 1.03.1 Device: iOS Universal Category: Games Price: $6.99, Version: 1.03.1 (iTunes) Description: *** PLEASE NOTE: This game is officially supported on the following devices: iPhone 5 and newer, iPod Touch... | Read more »
PINE GROVE (Games)
PINE GROVE 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: A pine grove where there are no footsteps of people due to continuous missing cases. The case is still unsolved and nothing has... | Read more »
Niantic teases new Pokémon announcement...
After rumors started swirling yesterday, it turns out there is an official Pokémon GO update on its way. We’ll find out what’s in store for us and our growing Pokémon collections tomorrow during the Starbucks event, but Niantic will be revealing... | Read more »
3 reasons why Nicki Minaj: The Empire is...
Nicki Minaj is as business-savvy as she is musically talented and she’s proved that by launching her own game. Designed by Glu, purveyors of other fine celebrity games like cult favorite Kim Kardashian: Hollywood, Nicki Minaj: The Empire launched... | Read more »
Clash of Clans is getting its own animat...
Riding on its unending wave of fame and success, Clash of Clans is getting an animated web series based on its Clash-A-Rama animated shorts.As opposed to the current shorts' 60 second run time, the new and improved Clash-A-Rama will be comprised of... | Read more »
Leaks hint at Pokémon GO and Starbucks C...
Leaked images from a hub for Starbucks employees suggests that a big Pokémon GO event with the coffee giant could begin this very week. The images appeared on Reddit and hint at some exciting new things to come for Niantic's smash hit game. | Read more »
Silent Depth Submarine Simulation (Game...
Silent Depth Submarine Simulation 1.0 Device: iOS Universal Category: Games Price: $7.99, Version: 1.0 (iTunes) Description: | Read more »
Enneas Saga lets you lead your own demon...
Defend the land of Enneas Continent from the forces of evil in the new fantasy MMORPG from Lyto Mobi: Enneas Saga. Can’t wait? No problem. It’s available to download now on Android devices. | Read more »
Great zombie games in the spirit of Dead...
Dead Rising 4 arrives tomorrow, giving enthusiasts a fresh chance to take selfies with zombies and get up to other ridiculous end-of-the-world shenanigans. To really get into the spirit of things, we've gone and gathered the best zombie games that... | Read more »
Amateur Surgeon 4 Guide: Advanced tips a...
Amateur Surgeon 4 is still tackling the competition at the top of the App Store charts, so if you haven't tried it out yet, you should probably do that right away. If you've been at it for a while, though, perhaps you're ready to start expanding... | Read more »

Price Scanner via MacPrices.net

Holiday sale: 12-inch Retina MacBook for $100...
B&H has 12″ Retina MacBooks on sale for $100 off MSRP as part of their Holiday sale. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.1GHz Space Gray Retina MacBook: $1199 $100... Read more
Apple refurbished 13-inch MacBook Airs availa...
Apple has Certified Refurbished 13″ MacBook Airs available starting at $849. An Apple one-year warranty is included with each MacBook, and shipping is free: - 13″ 1.6GHz/8GB/128GB MacBook Air: $849 $... Read more
Apple refurbished iMacs available for up to $...
Apple has Certified Refurbished 2015 21″ & 27″ iMacs available for up to $350 off MSRP. Apple’s one-year warranty is standard, and shipping is free. The following models are available: - 21″ 3.... Read more
Apple’s Education discount saves up to $300 o...
Purchase a new Mac or iPad using Apple’s Education Store and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free: -... Read more
Back in stock: Apple refurbished Mac minis fr...
Apple has Certified Refurbished Mac minis available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: - 1.4GHz Mac mini: $419 $80 off MSRP - 2.6GHz Mac... Read more
Twenty-Five Years Of Apple Laptops – A person...
Among many other things, the often tumultuous 16th year of the new century marked the 25th anniversary of Apple laptop computers, not counting the optimistically named 16-pound Mac Portable of 1989.... Read more
Landlordy iOS App Adds Support For Appliances...
Riga, Latvia based E-protect SIA is releasing major update (version 1.8) to its Landlordy app for managing rental business financials on the go. Landlordy is iPhone and iPad app designed for self-... Read more
Holiday sale, Apple iMacs for up to $200 off...
B&H Photo has 21″ and 27″ Apple iMacs on sale for up to $200 off MSRP including free shipping plus NY sales tax only: - 27″ 3.3GHz iMac 5K: $2099 $200 off MSRP - 27″ 3.2GHz/1TB Fusion iMac 5K: $... Read more
Holiday sale: Mac minis for $50 to $100 off M...
B&H Photo has Mac minis on sale for up to $100 off MSRP free shipping plus NY sales tax only: - 1.4GHz Mac mini: $449 $50 off MSRP - 2.6GHz Mac mini: $629 $70 off MSRP - 2.8GHz Mac mini: $899 $... Read more
Mac Pros on sale for up to $300 off MSRP, ref...
B&H Photo has Mac Pros on sale for up to $300 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3699, $... Read more

Jobs Board

*Apple* Retail - Multiple Positions- Philade...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Retail - Multiple Positions- San Ant...
Job Description:SalesSpecialist - Retail Customer Service and SalesTransform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Products Tester Needed - Apple (Unit...
…we therefore look forward to put out products to quality test for durability. Apple leads the digital music revolution with its iPods and iTunes online store, Read more
SW Engineer *Apple* TV Frameworks - Apple I...
The Apple TV team is looking for a software...create features that reflect the look and feel of Apple TV. Description: Were looking for someone who is Read more
Hardware Design Validation Engineer - *Apple...
The Apple Watch team is looking for a Hardware Design Validation Engineer. This person will be part of the Apple Watch hardware team with responsibilities for Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.