TweetFollow Us on Twitter

REALbasic Beta

Volume Number: 17 (2001)
Issue Number: 06
Column Tag: REALBasic

Beta Testers: What Do Programmers Want?

by G.D. Warner

The Basics of Testing Macintosh Software, Writing Bug Reports, and Keeping Developers Happy

"It was a bug, Dave."

While this article won't help developers produce bug-free code, it will help the people that test the programs developers write: the Beta Testers. We'll discuss the good (and bad) of bug reports, and take a brief look at MacsBug in a sidebar ("Quick and Dirty MacsBug"), where you'll learn the six commands one needs to use MacsBug effectively.

Your program is finished. Everything works (or appears to)... but before you release it to the (hopefully) paying public, it needs to be tested. That's where Beta Testers come in.

What does a Beta Tester do, anyway?

"Beta testers run the program and do pretty much everything possible in it," says a REALbasic developer who goes by the nickname of Squirrel. He continues:

"Click all the buttons, just mess around, and also use it like it is supposed to be used. If they find a bug, they report back what happened, what they were doing at the time of the error."

But ... what kind of information does a programmer need from his (or her) testers?

"Two words: Reproducible results," says developer Richard Theil. "An identified bug is an eliminated bug."

Words to live by.

"Exact instructions on how to get the program to misbehave will do the job," Richard continues. "If these can't be obtained, the next best thing would be a MacsBug StdLog."

StdLog is a text file produced by Motorola and Apple's MacsBug, a debugging tool for Motorola's 68K and the PowerPC chips.

If you've never seen it in action, MacsBug is as close to DOS as you'll ever get on the Mac (less PC emulators, Linux builds, and the terminal in OS X, of course). There are a couple rather lengthy manuals on the net that explain how to use it. For one of these, check here:

http://developer.apple.com/tools/debuggers/MacsBug/Documentation/MacsBugRef_6.2.sit.hqx

And here:

http://www.macfixit.com/reports/MacsBug.shtml

If you're testing a crash-prone program and need to produce a StdLog file but don't know how, check the Sidebar. It will save you some research time.

Thorsten Lemke, author of "Graphic Converter," has specific requirements for his testers:

"I ask my testers for the following: e-mail a screen shot, a MacsBug report, and e-mail the file that produced the error."

Obviously, it's hard to troubleshoot problems with a graphic without the original graphic.

RB developer Peter Jobs, (author of "Monica," "Hefty FTP," and "NewsFinder"), has a few suggestions on what he wants from his testers:

"I suppose the following sorts of things:

To actually run the program for some extended period; to check that the functions/screens do what they look like they are supposed to do; to check that the program corresponds to the docs; to hammer the program by trying to input invalid data and by running it under load; to think about the programs functions and interface and to suggest improvements. Obviously, to report your findings! Any other things you might find relevant-I can't think of everything!"

"All steps necessary to reproduce an error and the environment configuration," Thomas Engelmeier recommends. "Best way still is to provide a button that extracts 'submittable" data."

Interesting idea: software that can write its own bug report-or portions thereof, anyway. A developer who goes by the name of Lord Appolyon (or more simply, Rob) elaborates further on this idea:

"In my standard 'corpus' of libraries I use for development, one of the more mature things I've maintained is an interrupt-safe logging mechanism," he says. "When enabled at runtime, it can record every action (and even every internal "check condition" and other event of notice), for the purposes of forensic analysis. First and foremost, the logging system is designed to be as fast and as non-intrusive as possible-making users more apt to enable it when they exercise my program and also to be able to ferret out 'real-time' oriented bugs. My model was the UNIX syslog() service-which does not exist on MacOS (and the one product which bills itself as such needs events to function)."

"When I'm trying to fix a bug that's in a bug report, often I can spend 90% to 95% of my time trying to reproduce the bug in the debugging environment," says William Woody of PandaWave.

"That translates to about a day reproducing a bug which takes as little as a few minutes to a half an hour to actually fix," he continues. "While not all bugs are like this, most of them are. So any information I can get from the beta testers to help me reproduce the bug in a debugging environment will help *A LOT.*"

"Black Box testing (where testers do not have access to the source code, do not know how it is written/organized, etc.) needs to result in detailed steps to reproduce a found bug," says Shawn, a rather anonymous developer. "If the bug is not reproducible, even more detailed information is required (what other programs are running, System configuration, etc.) As a software developer who's worked with both in-house testing departments and end-users who report bugs, I believe this is the number one requirement/issue when reporting bugs."

"I want to see feedback basically. Most beta testers don't do anything at all," Theodore Smith of Elfdata Software says. "From those who do give feedback, I like to see someone who experiments with very many features of the program."

He continues: "I like to see determination to continually give reports, and to recheck very many things each time there is a new version. Obviously they can get away with checking less for a new version, than compared to the next time, but they should at least check out all the new features properly (and read the version history to find out the new ones!), test if the bugs they reported have disappeared."

Game programmers, of course, have slightly different requirements.

"Naturally, it helps to have bug reports. Most beta testers will do that much," says Brandon Ballinger of Flair Interactive.

He continues: "But, more importantly, programmers want game design advice and suggestions. Is the difficulty right? Was a piece of dialogue awkward? Does the user interface make sense? Is the player ever confused?"

Brandon offers a few good suggestions for game testers:

"Ideally, a Beta Tester would begin playing the game with a notepad by their side. Every time there was something wrong with the game (be it a bug or a design flaw), the tester should write it down. Likewise, if there was something particularly impressive, then the tester would write that down as well. I know I'd appreciate receiving an itemized list of specific good and bad points."

Andrew Welch, the self-described "el Presidente" of Ambrosia Software concurs with the thoughts of the other developers:

"Keep a notepad next to your computer to jot things down-deficiencies, bugs, or neat ideas," he suggests. "Put all that into a text file in a bulleted format and e-mail it to us." He continues: "Get the right tools for the job. Download and install MacsBug so you can generate standard logs for the developer to use in an effort to figure out why the game/utility crashed. You should also probably use a product like Snapz Pro 2, in the event that you need to take a screen capture of something to show a developer what is going on."

You can get Snapz Pro here:

http://www.SnapzPro.com

David Dunham, lead developer of A Sharp software adds an excellent recommendation for game testers:

"What I liked to see most was saved games (this was for King of Dragon Pass), since that frequently let me reproduce the problem. It was surprising how many people wouldn't bother."

James Wilson offers a few more suggestions from a slightly different perspective:

"Let me address this subject from a standard business application (no games, etc.).

Programmers should provide a standard way of submitting bug reports so that they then receive a standard response. Yes, do provide a large entry field for comments.

Programmers should request individual testers work on a specific area, user interface, documentation, entry fields, etc.

Testers should check edit field boundary and entry conditions. If a field should only have numbers what happens if letters are entered? What about special characters? If a field should only have a value within a certain range what happens if numbers outside the range are entered. What happens on values at the boundary?"

Those were some excellent things to watch for when testing a business application. They're even excellent suggestions for testing a programming environment like, say, REALbasic. Take a look at the latest version of RealBugs (Figure 1):


Figure 1: RealBugs

'Lots of room for comments' would be one way to describe the interface of RealBugs. I asked the folks at REAL Software about it:

"I believe Geoff (REAL's CEO) and Jason (head of support) came up with the idea," says Paul Scandariato of REAL Software. "It lets users send bug information to us in a structured way - once we receive it, we file it in a 4D database where it's later processed, assigned a bug number, and entered into our main database."

Sometimes a tester can go too far\0xC9telling the programmer how to write the code for his (or her) program is not a good idea, for instance.

"I mean when they obviously have no clue what they are talking about," says Jimbo. "Ninety-nine percent of our testers couldn't write "Hello World!" let alone disassemble anything. Their job is to test our software and report bugs, not play programmer for a day."

Richard Wesley of Electric Fish, Inc. has a slightly different slant on this subject:

"By the same token I am a lousy tester, and while I get annoyed with the boneheads who don't know the basics of their job (i.e. how to communicate results), I would find it similarly presumptuous to tell them how to do theirs. We can all agree on documentation protocols and testing methodologies, but isolation is an art, one that is not respected nearly enough."

\0xC9and then, of course, there are the, shall we say, "less than useful" bug reports.

Jimbo tells of a bad bug report he once found in his In box:

"What really gets me is a bug report like this:

'I found a huge bug in your app!! Click the main menu over here. Then pull this menu down. Look what happens!!!!'

Like I was looking over their shoulder as they did it."

Not even RealBugs can help with a report like that one.

William Woody of PandaWave tells a Tale of Bad Bug Reports:

"The *worse* bug reports are the ones which are like 'well, sometimes when I type in the document, it crashes.' Typed what? What happened before you typed? How did it crash? What files were open? What else was running? This is a useless bug report, as it requires me to either develop ESP or read the mind of the tester, or exercise *every possible combination of events* which involves typing\0xD0something which could take me years to do."

Bill had another nightmare to relate:

"Or the worse: once someone told me my application was completely unusable and not at all ready for shipping. I'm thinking the application won't even start on his machine, and, as it crashes, it's calling up his friends via modem and yelling audio-synthesized curse words. Turns out he was complaining about a typo in the startup splash window, and, on seeing the typo so early on, never bothered to actually exercise the program. After all, if there is such a simple mistake this early on, how many other bugs are there in the application? (*sigh*)."

Lord Appolyon's syslog() idea helps out in this area:

"It greatly diminishes the 'bs' factor of 'OK, what exactly did you do, etc?'\0xD0the log entries are timestamped with millisecond precision, so time ordering and sequencing are trivial to establish. At maximum loglevel, all routine entry/exit-points are logged with parameters and return values."

I think I'll have to talk to Lord Appolyon\0xD0er, Rob\0xD0 some more on this\0xC9

Owen Strain has a request for developers everywhere:

"Descriptive/Instructive error messages. If an error message is reporting a known bug, then it should tell me so. Error codes are good, because it makes it easy to be specific. I've seen a program that says 'WRITE THIS DOWN AND MAKE A BUG REPORT' which is good except that it was a finished program\0xC9"

Lord Appolyon also has a recommendation for developers everywhere:

"I *HIGHLY* advise developers to put some kind of logging service in their beta software (which can have logging levels increased/decreased or switched on/off at RUNTIME). This isn't a panacea for all kinds of problems; however in my experience it's allowed me to nail the vast majority of 'operational' bugs."

I test a lot of programs (sometimes unintentionally, like a lot of Windows users). Any time I crash while using the program, I create a StdLog file, write up an e-mail explaining what I did to produce the crash, and try to do it again after restarting with Extensions off at best, or with the Base Set for whichever version of the OS I'm using. If the program crashes again, I produce another StdLog (this I'll append to the original StdLog file) and e-mail all of this to the developer, along with my notes on what I was doing. When necessary, I'll include a screenshot.

"Wow! I wish my testers would do *half* the stuff you do," one programmer wrote to me on Usenet some time ago.

Hence, this article.

In summary, a good bug report should contain the following:

  • System configuration (i.e., "iMac DV SE 400, 192 MB of RAM, Mac OS X.")
  • A StdLog from MacsBug
  • A complete and well-written description of the bug (don't make the developer have to read your mind!)
  • (For games): a saved game
  • (For graphics programs): a copy of the problem graphic

When testing a new program, here are a few suggestions:

(1) Look at all the menus. Are all the menu items spelled correctly? Does anything appear under the wrong menu\0xD0'Open' under 'Edit,' for instance?

(2) Look at the user interface. Is everything spelled correctly? Are command-key equivalents improperly assigned (i.e., Command-A, C, N, O, P, V, W and Z used for anything other than 'Select All,' 'Copy,' 'New,' 'Open,' 'Print,' 'Paste,' 'Close Window,' and 'Undo')?

(3) If the program has fields, are they supposed to be 'numbers only'? 'Letters only'? Are they?

(4) Does the program have AppleScript support? If so, does it work properly?

(5) Does it conform to the Apple Human Interface Guidelines? Not sure? Go here:
http://www.devworld.apple.com/techpubs/mac/HIGuidelines/HIGuidelines-2.html

My hope for this article is that anyone who reads it and subsequently downloads a new program will look at that piece of software differently\0xC9 and, if necessary, will be able to submit to the developer a useful bug report.

It was a bug, Dave... "I feel much better admitting that now."

Uh, sure HAL, sure.


G.D. Warner is a Technical Writer, and has been using the Mac since 1992. He’s been downloading and testing software during that entire time, first from local BBSes, then from the internet, and toying with programming (HyperCard, AppleScript, FutureBasic, ObjectBasic, C, C++, Visual Basic, and, of course, RealBasic). E-mail him at mailto:gdwarner@ricochet.net.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Things 2.5.4 - Elegant personal task man...
Things is a task management solution that helps to organize your tasks in an elegant and intuitive way. Things combines powerful features with simplicity through the use of tags and its intelligent... Read more
NeoOffice 2014.10 - Mac-tailored, OpenOf...
NeoOffice is a complete office suite for OS X. With NeoOffice, users can view, edit, and save OpenOffice documents, PDF files, and most Microsoft Word, Excel, and PowerPoint documents. NeoOffice 3.x... Read more
iPhoto Library Manager 4.2 - Manage mult...
iPhoto Library Manager allows you to organize your photos among multiple iPhoto libraries, rather than having to store all of your photos in one giant library. You can browse the photos in all your... Read more
Web Snapper 3.3.8 - Capture entire Web p...
Web Snapper lets you capture Web pages exactly as they appear in your browser. You can send them to a file as images or vector-based, multi-page PDFs. It captures the whole Web page - eliminating the... Read more
TeamViewer 10.0.41404 - Establish remote...
TeamViewer gives you remote control of any computer or Mac over the Internet within seconds, or can be used for online meetings. Find out why more than 200 million users trust TeamViewer! Free for... Read more
Ableton Live 9.1.8 - Record music using...
Ableton Live lets you create and record music on your Mac. Use digital instruments, pre-recorded sounds, and sampled loops to arrange, produce, and perform your music like never before. Ableton Live... Read more
VOX 2.5 - Music player that supports man...
VOX is a beautiful music player that supports many filetypes. The beauty is in its simplicity, yet behind the minimal exterior lies a powerful music player with a ton of features & support for... Read more
OmniFocus 2.1.2 - GTD task manager with...
OmniFocus helps you manage your tasks the way that you want, freeing you to focus your attention on the things that matter to you most. Capturing tasks and ideas is always a keyboard shortcut away in... Read more
Adobe Flash Player 17.0.0.169 - Plug-in...
Adobe Flash Player is a cross-platform, browser-based application runtime that provides uncompromised viewing of expressive applications, content, and videos across browsers and operating systems.... Read more
iMazing 1.1.3 - Complete iOS device mana...
iMazing (was DiskAid) is the ultimate iOS device manager with capabilities far beyond what iTunes offers. With iMazing and your iOS device (iPhone, iPad, or iPod), you can: Copy music to and from... Read more

Chainsaw Warrior: Lords of the Night has...
It's time to put the Darkness back in its place now that Chainsaw Warrior: Lords of the Night has officially made it to iOS. | Read more »
A World of Ice and Fire Lets You Stalk 2...
George R. R. Martin’s A World of Ice and Fire, by Random House, is a mobile guide to the epic series. The new update gives you the Journeys map feture that will let you track the movements of 25 different characters. But don't worry, you can protect... | Read more »
Gameloft Announces Battle Odyssey, a New...
Battle Odyssey, Gameloft's newest puzzle RPG, is coming to the App Store next week. Set in the world of Pondera, you will need to control the power of the elements to defend the world from evil. You'll be able to entlist over 500 allies to aid you... | Read more »
Fusion - HDR Camera (Photography)
Fusion - HDR Camera 1.0.0 Device: iOS Universal Category: Photography Price: $1.99, Version: 1.0.0 (iTunes) Description: Fusion creates HDR (high dynamic range) photos by capturing different exposures and then combining them into one... | Read more »
Sago Mini Toolbox (Education)
Sago Mini Toolbox 1.1 Device: iOS Universal Category: Education Price: $2.99, Version: 1.1 (iTunes) Description: Come build with the Sago Mini friends! Use a wrench, try a saw, or hammer some nails. From sewing hand puppets to... | Read more »
You Should Probably Grab Hitman GO While...
Hitman GO is a surprisingly cool (yet also incredibly drastic) departure from the Hitman series. It's well worth playing for any puzzle game fans out there, and at the moment you can get your hands - or garrotte if you will - on it for a mere $0.99... | Read more »
IFTTT is Bringing Do Button and Do Note...
IFTTT has announced Do Button and Do Note for the Apple Watch. Do Button lets you make your own personalized button that can connect to things like your Google Drive, control the temperature in your home with Nest Thermostat, or turn the lights on... | Read more »
How Many Days, Hours, and Minutes Are Le...
Countdown, by Yves Tscherry, is now available on the App Store. The app keeps track of countdowns to your favorite things such as someones birthday or days till the New Year. You can display the time in seconds, minutes, hours, days, weeks, months,... | Read more »
The All-New Misfit 2.0 App is Available...
Misfit has just given their app a complete overhaul. Misfit 2.0 now has a brand new interface with a sleek design and is easier to navigate. You'll be able to sync your Misfit device and look up health and fitness information faster than ever before... | Read more »
Halo: Spartan Strike (Games)
Halo: Spartan Strike 1.0 Device: iOS Universal Category: Games Price: $5.99, Version: 1.0 (iTunes) Description: Delve into 30 challenging missions through cities and jungles using a devastating arsenal of weapons, abilities and... | Read more »

Price Scanner via MacPrices.net

Clearance 13-inch 2.6GHz Retina MacBook Pro a...
B&H Photo has clearance 2014 13″ 2.6GHz/128GB Retina MacBook Pros now available for $1099, or $200 off original MSRP. Shipping is free, and B&H charges NY sales tax only. Read more
Apple refurbished 2014 13-inch Retina MacBook...
The Apple Store has Apple Certified Refurbished 2014 13″ Retina MacBook Pros available for up to $400 off original MSRP, starting at $979. An Apple one-year warranty is included with each model, and... Read more
iMacs on sale for up to $205 off MSRP, NY tax...
B&H Photo has 21″ and 27″ iMacs on sale for up to $205 off MSRP including free shipping plus NY sales tax only: - 21″ 1.4GHz iMac: $1019 $80 off - 21″ 2.7GHz iMac: $1189 $110 off - 21″ 2.9GHz... Read more
Sale! 16GB iPhone 5S for $1 with service
Best Buy is offering 16GB iPhone 5Ss for $1.00 with 2-year activation at a participating cellular provider. Choose free home shipping and activation, or buy online and activate during in-store pickup... Read more
Apple refurbished 2014 MacBook Airs available...
The Apple Store has Apple Certified Refurbished 2014 MacBook Airs available starting at $679. An Apple one-year warranty is included with each MacBook, and shipping is free. These are currently the... Read more
27-inch 3.5GHz 5K iMac on sale for $2349, sav...
 Adorama has the 27″ 3.5GHz 5K iMac in stock today and on sale for $2349 including free shipping plus NY & NJ sales tax only. Their price is $150 off MSRP. For a limited time, Adorama will... Read more
Save up to $380 on an iMac with Apple refurbi...
The Apple Store has Apple Certified Refurbished iMacs available for up to $380 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 27″ 3.5GHz 5K iMac – $2119 $... Read more
iFixIt Teardown Awards 12-IInch Retina MacBoo...
iFixIt has posted its illustrated teardown of the new 12-inch MacBook Retina. They note that this new MacBook is less than half the thickness of the last Apple notebook called just “MacBook” back in... Read more
13-inch 2.5GHz MacBook Pro (refurbished) avai...
The Apple Store has Apple Certified Refurbished 13″ 2.5GHz MacBook Pros available for $829, or $270 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 13″ 2.... Read more
Faithful iPad 2 Gets A Second Career In Retir...
Finally, after four months’ transition, I handed my faithful old 2011 iPad 2 off to my wife at the end of March and switched whole-hog to using the iPad Air 2 I bought back in November. I’d found... Read more

Jobs Board

*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
*Apple* TV Live Streaming Frameworks Test En...
**Job Summary** Work and contribute towards the engineering of Apple 's state-of-the-art products involving video, audio, and graphics in Interactive Media Group (IMG) at Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.