Related Info: Memory Manager

QC: Test For Success

Integrated debugging & stress testing software

By Paul Robichaux, Fairgate Technologies, Harvest, AL

About the Author

Paul Robichaux - Paul Robichaux leads a dual life: by day, he builds Unix and Windows NT applications for a major computer company. During the rest of the time, he builds custom-engineered Mac applications and WWW pages for clients in a wide variety of fields. He welcomes reader e-mail at paul@fairgate.com, or visit his WWW page at http://www.iquest.com/~fairgate.

Taking the Sting
Out Of The “D” Word

Every programmer does it. Experienced programmers usually do less of it than neophytes, and programmers on some platforms do it more than others. What is it? Caffeine consumption? Swearing at operating system designers? Well, besides those... I’m talking about debugging. By some estimates, debugging can take up to 40% of development time, and studies have indicated that there can be as much as a twenty-to-one difference in productivity between a run-of-the-mill debugger and a really skilled debugger.[1]

The Macintosh operating system architecture offers unique challenges to debuggers. The combination of handles and memory-moving traps provide fertile ground for bugs related to the almost Brownian motion of relocatable blocks. Memory-related errors are made more difficult to isolate and fix because writing to memory that belongs to other applications, or writing beyond the bounds of a particular block, doesn’t always cause a crash. If a crash does occur, it can often be quite some time after the original bad write, so the cause of the bug can be masked by other paths taken through the source code.

There are a wide range of freely available tools available for memory & resource debugging. They were generally written because their authors needed them, and then saw benefit to giving them away so others could benefit from their work. That was good for everyone. However, the heroes who wrote and distributed them (people like Greg Marriott, who wrote EvenBetterBusError, DoubleTrouble, and DisposeHandle; and Bo3b Johnson, who wrote Leaks, Blat, and ZapHandles) have real jobs, and decided not to make a career of keeping the tools up to date and adding features. For example, Blat doesn’t work on very old or very new Mac CPUs. DisposeResource and DoubleTrouble slow system-wide performance since they are always on when installed. Still others reveal faults not only in your code, but in others’ as well. That’s fine if you have time to deal with other folks’ buggy software, but it sometimes gets in the way of debugging your own.[2]

Onyx Software has gone the extra mile and built QC™, a control panel which offers a variety of tools designed to help you build rock-solid code. QC allows you to selectively turn on a wide range of stress-testing features (including almost all of those provided by the above-mentioned tools) on a per-application basis. QC also includes a versatile API which allows you to embed calls to QC’s engine within your own code. The QC documentation says that QC is “specifically designed to make memory-related errors reproducible,” and it excels at that task.

What QC Is For

QC is designed to perform two key functions. First, it helps you find memory and resource-handling errors during development. Second, it provides a simple way for quality assurance testers to put unusual stress on an application to flush out any remaining errors or performance problems.

Most developers are already familiar with the two primary types of debuggers: low-level and source-level. Both types allow you to interactively examine the contents of memory and processor registers. The primary difference between them is how you interact with the code you’re debugging. Low-level debuggers, like Macsbug, typically require that you work with the disassembled contents of memory. You set breakpoints by absolute or relative memory address, and the contents of structures referenced by pointers may not always be identifiable by type. Source-level debuggers show you the source statements of your programs and allow you to set breakpoints and examine variables in the context of that, and some even allow you to execute function calls without interrupting execution.

QC doesn’t really fall into either category. It’s not a debugger in and of itself; it’s a debugging aid. You can’t manipulate or examine the contents of memory using QC. Instead, it can force your code to drop into the installed low-level debugger when it encounters an error, or it can simply beep. With the provided API, you can perform any operation expressible in code when an error occurs. The API also allows you to turn on a particular kind of debugging for any section in your code, which makes the beep option more useful than you might otherwise suspect.

Getting Started With QC

As you’d expect, installing the QC control panel is very, very simple: drag it to your system folder and reboot. When you open it the first time you’ll have to personalize it; after that, you’ll see the control panel’s main window:

The center list, called the “target list,” allows you to set debugging checks on an individual basis. Each application can have a unique combination of the 15 distinct checks that QC knows how to perform, and you can modify which checks should be performed while the application is being QCed. If you prefer, QC can automatically start testing when the application or code resource under study starts executing. The block at the window’s bottom allows you to specify a hot key to toggle QC at any time. Pressing the hot key causes QC to start checking the currently active application; pressing it again turns checking off. You can also use the auto-launch setting to start QC’s tests when an application starts or a code resource is loaded.

Figure 1: The QC control panel main window.

Double-clicking one of the items in the target list brings up a dialog of test options. List items with downward-pointing triangles have associated parameters which you can set either via the control panel or the API routines.

Figure 2: QC’s test configuration dialog.

You can choose QC’s behavior when it detects an error - QC will either beep or dump you into the installed low-level debugger. If you’re using the API routines, QC will not generate an error on its own; instead, it’ll return an error code so that you can do whatever reporting or cross-checking is appropriate. If you choose, your callback routine can let QC handle the error itself.

What QC Knows How To Check

QC has a total of 15 different tests. Any combination of tests can be enabled for any application or code resource. You can run the tests on both application and system memory and heaps. All but two of the tests run on 68k and PowerPC Macs; the “invalidate free memory” and “block bounds checking” tests are automatically disabled when running under the Modern Memory Manager.

I found that the tests tended to fall into one of three categories: memory-handling tests, Toolbox use tests, and stress tests. Some combinations of tests are particularly effective at flushing out errors.

Memory and Resource Tests

These tests verify that your code is staying inside its own heap and address space. In particular, these tests are useful for catching code which uses handles or pointers after they’ve been disposed.

Cross-reference master pointers

This test checks that every relocatable handle in your heap is pointed to by a master pointer within a master pointer block. As you might imagine, it’s not good if you have a relocatable block in your heap which isn’t pointed to by a master pointer.

Validate handles/pointers

This test validates each handle or pointer passed to Memory Manager calls to ensure that their addresses fall within the application’s heap and that the block to which the pointer or handle points is of the correct type. This is perhaps the most useful single test in QC’s arsenal, since it can detect a multiplicity of sins.

Detect writes to location zero

This very useful test sets a trap for code which uses dangling pointers or handles. When enabled, QC places a special tag value at memory location zero; at each trap call to see, QC checks to ensure that location zero still contains that tag.

Dereferencing zero

Like the “detect writes to location zero” option, this option salts location zero with a special value. In this case, the special value is one that causes a bus error (on 68020 or later CPUs) or an odd address error (on 68000 CPUs.) When your code attempts to dereference that tag, as it would when mistakenly trying to use a purged handle, you’ll immediately be dumped into the installed low-level debugger.

Toolbox Use Tests

The Mac OS offers developers a very powerful set of tools. This power carries with it some danger, because many Toolbox routines fail when given bad parameters, and some calls are dangerous when misused. These tests check for valid parameters and proper use of calls. They are not a substitute for using correct prototypes in your code.

Reasonable allocations

This test checks parameters to NewHandle() and NewPtr() to make sure that they’re positive values less than or equal to a user-specified size. On some compilers, it also works with library routines like malloc() which call the Toolbox routines to allocate memory.

Check dispose/release calls

Everyone was a Mac newcomer at some time, and this test catches a classic newcomer’s mistake: calling ReleaseResource() on a handle or DisposeHandle() on a resource. It takes some practice to make sure to do the right thing with handles which point to resources, and this test provides welcome reinforcement.

BlockMove bounds checking

The Toolbox BlockMove() call can fail rather spectacularly if you call it with addresses which span multiple blocks in the heap. This test makes QC check that the start and end addresses of the source and target blocks don’t cross block boundaries.

Block bounds checking

Some languages offer compile- or run-time bounds checking, but C doesn’t. Programs which write outside the bounds of an allocated block or array will sometimes crash, but sometimes they won’t. To make it easier to find the problem, this test causes QC to add a tag value to each allocated block. Your application doesn’t see the tag, but QC can. When it detects that the tag has been overwritten, it generates an error. This test is a lifesaver. It doesn’t work under the Modern Memory Manager, but you can run your tests using the original 32-bit memory manager to get the benefit of this test.

MemErr checking

Despite admonitions from Apple, many programs don’t check the value of MemErr , the low-memory global used to flag memory operation errors, after using memory-related Toolbox routines. When this test is active, QC will check MemErr after each Memory Manager trap.

Grow locked handle check

Inside Macintosh volume II has a warning: calling SetHandleSize() on a locked handle can fail if the block needs to be moved in order to grow. Unfortunately, MemErr is the only failure indication after such an occurrence. This test catches SetHandleSize() calls where the target handle is locked. [3]

Grow pointer checks

Nonrelocatable blocks, also known as pointer blocks, don’t move when a heap shuffles, but they come with their own special limitations. When you try to grow a nonrelocatable block in the heap, if there’s another nonrelocatable block “next” to it, the grow will fail. Like the “grow locked handle” test, this test will detect such failures and report them.

Stress Tests

Some applications behave well when they have plenty of RAM, but when forced to run in a smaller partition, they start to get flaky. Stress testing involves placing severe loads on the application by forcibly purging, scrambling, unloading, and otherwise molesting its heap. Well-written applications will be able to withstand such extreme conditions; other programs are bound to improve if their authors subject them to these tests.

Heap scramble

The state of an application’s heap varies from run to run and from machine to machine. This variation makes it difficult to find heap-related bugs, like use of a stale handle or reliance on a handle’s not moving. QC’s heap scramble, like the standard Macsbug ‘hs’ command, moves all handles in the application heap each time a memory-moving trap call is made. This test is great for general-purpose stress testing, since it creates a condition not likely to occur during ordinary use.

Heap purge

Once a resource or segment has been marked as ‘purgeable,’ it’s subject to being unloaded any time memory is allocated. If your code relies on the presence in memory of something which may have been purged, it can break when the Toolbox actually does a purge. The heap purge test causes QC to purge the heap after every memory-moving trap, meaning that anything marked purgeable will be flushed quickly.

Heap check

The heap check option tests the heap’s structure for corruption at each memory-moving trap. By checking on every memory-moving trap, you’re more likely to catch a heap-corrupting bug closer to the scene of the crime.

Invalidate free memory

This test puts a special salt value into all unallocated RAM within your application’s partition. If your code attempts to dereference the contents of free memory, the salt value will cause a bus error. This check is extra-useful when combined with the heap purge test; as items are purged from the heap, the memory they used to occupy gets salted. If your code reuses it, QC will catch it in the act. Unfortunately, this test doesn’t run under the Modern Memory Manager. Again, it runs fine on the PowerMac under the “classic” memory manager, so you can still use it for testing.

Eureka!

“Eureka” is Greek for “I have found it!” As mentioned above, the QC control panel will either beep or call _DebugStr when it encounters an error. The output string is very helpful in and of itself; it tells you what error was detected, what handles or parameters were involved, and the address of the last trap call in your code. QC only checks for errors after memory-related traps are called, so the “last trap called” field is useful since it can help you pinpoint the exact location of the error. All you need to do is search backwards from the address in the last trap field to the previous trap call.

Some tests may cause bus errors, so instead of a nicely formatted QC message you get a plain debugger prompt. This most commonly occurs with the “dereference zero” and “write to zero” tests. Finding the precise location of these errors is up to you. QC uses a unique salt value for each of these tests, so you can tell which type of misbehavior caused the fault.

For the other cases where QC does a controlled stop because it saw something noteworthy, you end up in the debugger with the program counter (PC) set to an address within QC’s address space. This poses a small problem for users of source-level debuggers. Since the source debugger doesn’t issue the breakpoint, there’s no way at break time to set a new breakpoint. To avoid this problem, you can use the API functions; if you write a small wrapper function which calls the QC routine of interest and breaks when an error occurs, you can break in the source-level debugger when the wrapper routine returns an error. Some debuggers also allow you to step out of the low-level debugger and back into your own function.

Using the API

The API is supplied as a C header file and three linkable 68K libraries, in MPW, Think, and CodeWarrior formats. The API contains calls to detect whether QC is active and/or installed, to activate or deactivate the current set of tests, to test the status of or activate/deactivate individual tests, and to instantly perform some types of tests.

Structurally the API is similar to most other Mac APIs; to use it, you use an enabling routine (QCActivate()) to start testing, then a series of get/set state calls to query, set, or clear various features and settings.

The API provides a great deal of flexibility, since you can easily turn tests on and off. By writing functions or macros to turn particular tests on and off, you can easily test at the function level. You can further configure your setup to run certain sets of tests at different points in the development cycle.

A minimal set of calls to use QC in your code might look like the code in Listing 1. Of course, in a real environment you should check error returns for all QC calls, since it’s possible for the API routines to return a variety of error codes.

Listing 1: QCTools.c

Sample utility routines for using QC within applications. StartQCTesting turns on QC testing if QC is installed but not active. StopQCTesting turns QC testing off, and SetDevelopmentTests configures QC to use a particular set of tests.

/* 1 */
QC tools

#include “QCAPI.h“

StartQCTesting
// turn on QC for testing
void StartQCTesting(void)
{
 if (QCInstalled())
 if (!QCIsActive())
 QCActivate(NULL);
}

StopQCTesting
// turn off QC when we’re done
void StopQCTesting(void)
{
 if (QCInstalled())
 if (QCIsActive())
 QCDeactivate(NULL);
}

SetDevelopmentTests 
// Turn on arbitrary set of tests depending on build type; we
// developed these test settings by guess and by gosh. You
// can define similar functions for other levels of testing.
void SetDevelopmentTests (void)
{
 QCErr testErr = noErr;
 testErr = QCSetTestState(qcValidateHandlePointers, TRUE);
 testErr = QCSetTestState(qcDetectWriteToZero, TRUE);
 testErr = QCSetTestState(qcDerefZeroCheck, TRUE);
 testErr = QCSetTestState(qcCheckDisposeRelease, TRUE);

 // make sure we see Macsbug error messages
 testErr = QCSetTestState(qcDebugBreaks, TRUE);
}

Callbacks

One really nifty feature of the QC API is that you can set custom callbacks that are invoked when QC detects an error. These callbacks can do extra testing or processing, depending on the error reported by QC. The QCInstallHandler() routine allows you to install a handler invoked each time an error occurs; within the callback routine, you can examine each error condition and you’re interested in.

Listing 2: QCCallback.c

Sample callback which logs the error to a file instead of dropping into the debugger. This particular routine logs errors using the string that QCreturns.

/* 2 */
QC callbacks

#include “QCAPI.h”

ErrorToFileCallback

// the callback routine is defined as “typedef long (*QCCallBack)(QCPBPtr)”; 

// the QCParamPtr tells us what type of test generated the error and 
what
// error occurred.
long ErrorToFileCallback(QCPBPtr *theParam)
{
 QCErr anErr = kQCNoErr;
 StringPtr s = NULL;

 // get the error message that would normally go to Macsbug
 anErr = QCGetErrorText(theParam->errorID, s);
 if (anErr == kQCNoErr) 
 {
 // log the message to our app log file
 LogEventMessage(kErrorEvt, s);
 }
 else
 LogEventMessage(kErrorEvent, ‘QC internal error’);
}

MyInstallQCCallBack 

// call this routine to make QC start using the callback.
// In its original form, this routine gets called right before
// the start of the main event loop. The second parameter
// to QCInstallHandler is a refcon, so you can pass data to the handler
// routine when it gets called.
QCErr MyInstallQCCallBack (void)
{
 return QCInstallHandler(ErrorToFileCallBack, OL);
}


MyUnInstallQCCallBack 

// call this routine to make QC stop using the callback.
// In its original form, this routine gets called right before ExitToShell()
QCErr MyUnInstallQCCallBack (void)
{
 return QCRemoveHandler();
}

Documentation and Support

It’s hard to effectively use a tool which has poor documentation or tech support. QC doesn’t suffer from either. Like Metrowerks’ CodeWarrior, QC comes packaged in a CD “jewel case” with no printed documentation. Instead, comprehensive DOCMaker-format documentation is included on the floppy; in addition to a “QuickStart” document, there’s a full user’s manual, plus a separate guide to the API. The disk also includes source code for “BadAPPL,” a small application designed to provide a showcase for all the different types of errors QC can detect.

Onyx Tech offers technical support via e-mail; all my questions have been promptly, cheerfully, and accurately answered. Updaters for new versions are released to the Internet and several commercial online services; QC is presently at version 1.1.1.

Onyx has also taken the pleasant step of releasing a demo version of QC. This trend is finally catching on among development tools, and it’s quite nice to be able to evaluate a tool in your own environment before laying down that hard-earned cash.

The Verdict

Since there are several free tools which incorporate various parts of QC’s functionality, why would you pay $99? Three reasons: ease of use, completeness, and versatility. QC is easy to install and use, even for nontechnical users who might be doing QA or testing on your application, and it doesn’t require any fluency in Macsbug. It incorporates a wider range of tests than any combination of other tools, and the ability to configure individual tests using the API is a terrific addition to an in-house development debugging suite.

Despite my enthusiasm, there are still a few things QC doesn’t do. It doesn’t find memory leaks; you’ll still need the “leaks” dcmd for that. It doesn’t give a precise cause for bus errors uncovered during the “dereference zero” and “write to zero” tests, so you’ll have to find those on your own. Finally, QC could potentially lead to sloppy programming after all, when you have such good tools for finding defects, your incentive not to make them in the first place may be reduced.

Onyx is planning leak detection, .SYM file support, and other user-requested improvements for their first post-PowerPC release, tentatively scheduled to ship in January or February of 1995.

For US$99, QC is a steal. Unless you’re writing software that no one else ever uses, every defect you catch in house is one defect your freeware, shareware, or commercial customer will never see and it’s one defect you won’t have to spend tech support or QA time finding and resolving. I give it two thumbs up.

Onyx Technology can be contacted via e-mail (OnyxTech@aol.com, D2238 on AppleLink, or 70550,1377 on CIS) or the more conventional routes: via phone at +1 813 795 7801, via fax at +1 813 792 5152, or via snail-mail at 7811 27th Avenue West; Bradenton, Florida, 34209 USA.

Bibliography and References

[1] McConnell, Steve. Code Complete, Microsoft Press, 1993; p. 625. I highly recommend this book for all software developers, especially the chapter on debugging.

[2] These tools are all available from ftp.apple.com in /pub/dts/mac.

[3] Inside Macintosh Vol. II, p. 34.