Volume Number: 15 (1999)
Issue Number: 6
Column Tag: Tools Of The Trade
by Daniel Jalkut
This is not your father's low-level debugger
MacsBug is an extremely powerful, low-level debugger which Apple maintains and distributes for free to developers of Mac OS software. From its humble beginnings as the Motorola Advanced Computer Systems Debugger in 1981, it has evolved into the debugger that many developers both inside and outside of Apple depend upon to get their jobs done.
The aim of this article is not so much to introduce MacsBug as it is to reintroduce it. Enough basic introductions to MacsBug have been written that I don't believe a repetition is necessary. If you are interested in a detailed description of MacsBug's basic features, I suggest you consult one of the publications mentioned at the end of this article. I suspect that most of you have heard of MacsBug, and a majority of you has probably used it to varying extents. I hope to provide something that each of you can use to further your use of MacsBug in the battle against buggy code.
In recent years, MacsBug has received less attention from Mac OS technical publications than it did in the past. Many developers have switched to using source-level debuggers like the one included with Metrowerks CodeWarrior. Those debuggers, while very convenient for the majority of bug diagnoses, have a limit to their usefulness. MacsBug continues to prosper because of the elegance with which it allows developers to overcome limitations of higher level debuggers. This article is divided into three sections, each of which discusses a different aspect of this elegance.
The first section is a high level discussion of the thinking that goes on when diagnosing a bug, and a description of some basic approaches you can take to work out the diagnosis with MacsBug. The second section extols the virtues of MacsBug's extensibility, a feature that should appeal to those of you with highly specialized or unpredictable debugging needs. Finally, the third section divulges some of the exciting features that MacsBug has gained in the past couple of years, which should convince the biggest MacsBug skeptic that something here deserves a second look.
Before you experiment with any of the functionality described in this article, I highly recommend downloading the most recent version of MacsBug from Apple's MacsBug web site:
Part One: The Debugging Mindset
I don't know anybody who has mastered MacsBug. Like Herman Hesse's Siddharta and his pursuit of nirvana, the mastery of MacsBug is a lifelong journey for which there is no apparent end. This humbling reality, while disheartening, must not prevent you from using what you do know to battle any bugs you encounter along the way! Like chess, the basic tools can be easily taught, but the secrets of winning are learned only by developing a mindset which allows you to apply simple techniques in new and exciting combinations. In this section, I will describe a basic approach to debugging in MacsBug, which you can expand upon as you develop your own debugging style.
MacsBug is typically encountered by one of two mechanisms: intentional or unintentional CPU exceptions. An exception is just what it sounds like - a deviation from the normal course of operations. Intentional exceptions occur when you, the programmer, premeditate an event that causes MacsBug to interrupt the execution of code and display the state of the computer on the screen. Unintentional exceptions occur when somebody's code crashes. The mentality for dealing with either case is different.
When you break into MacsBug intentionally, either by a breakpoint ("tvb", "atb", "br", "brp") or a programmed exception in your code (Debugger or DebugStr), you typically want to examine the state of things at a particular point in your code's execution. Typical reactions to an intentional MacsBug entry are to check that the parameters of a function look correct (e.g. "dm r3"), that the heap is not corrupt ("hc"), or to step through ("s", "t") code one instruction at a time and ensure that your code does what you think it should. MacsBug is perfectly suited to debugging in these circumstances, but since this type of debugging is especially well suited to a source-level debugger, I will focus only on unintentional interruptions (crashes).
When MacsBug is encountered as the result of a crash, you don't have the same luxury of understanding the circumstances as you do in an intentional interruption. Three very important questions to ask are: "What caused the crash?", "Where am I?", and finally "How did I get here?" The answers to these questions will greatly increase the odds of pinpointing the problem. Let's discuss some basic strategies for answering each question in turn.
What Caused the Crash?
MacsBug interrupts the execution of code when that code violates the rules of the CPU. If you are lucky, MacsBug's explanation of the crash will provide some insight into the problem. MacsBug displays the explanation immediately after it appears, and the explanation can be repeated by issuing the "how" command (a caveat is that MacsBug forgets this information as soon as you step or trace). Usually, the information provided is broadly useful, but doesn't specifically pin down the cause of the crash. Among the most common causes of crashes are memory exceptions and illegal instructions.
Memory exceptions occur when a piece of code tries to read or write from memory that either doesn't exist (is not part of the address space) or which exists but is off limits to the crashing code. When this is the cause of a crash, the basic approach is to look at the instruction that is causing the crash, and try to figure out where the bad address came from. If you're lucky, the address was the result of the last subroutine called, and you know the culprit immediately. If you're not so lucky, finding the origin of the address might involve tracing back through hundreds of lines of MacsBug disassembly, across several levels of subroutine calls.
Illegal instructions are sequences of bits that make no sense to the CPU. The CPU depends on receiving a stream of instructions that corresponds exactly to actions it knows how to perform. For instance, the hex value 0x38600000 means to the PowerPC processor "put the value zero in register r3". When the PowerPC processor receives a hex value like 0x11111111, it means nothing to the CPU so it causes an illegal instruction exception. When you reach MacsBug because of an illegal instruction, you are usually facing a bug where the program counter is pointing to data instead of code. There is a small chance that your compiled code contains illegal instructions, but this is unlikely unless there is a bug in the compiler, or you compiled for a specific CPU (Motorola 68030 for instance) and tried to run on a different CPU (Motorola 68000 for instance). Since the overwhelming majority of times you hit an illegal instruction will be because you are executing data instead of code, the solution to this problem requires answering the questions "Where am I?" and "How did I get here?".
Where Am I?
You know why the computer crashed, but this information is useless if you don't know the culprit code that caused it. There are three types of crashing code in this world: your code, other people's code, and data.
If you are building with symbols and traceback tables enabled, as you should be for pre-release builds, MacsBug will make it immediately clear whether you are executing your own code or somebody else's. If MacsBug prefaces the disassembly in the program counter display with a symbol, and you recognize the symbol as one of your routine names - congratulations your code crashed!
It Came from Beyond...
If there are no symbols, and the code doesn't scream out to you what it does (immediate recognition of assembly code's purpose is an uncommon, but real skill), you are going to have to dig deeper to find out exactly whose code it is. An immensely useful MacsBug command is the "wh" (short for "where") command, which identifies as best it can the characteristics of a location in memory. Commonly, this command is issued without a parameter, which causes it to give information about the current location of the program counter. The output of the "wh" command ranges from "very useful" to "not so useful, but I'll take it" to "pretty darn useless."
In the "very useful" category, when the code you are executing is part of a PowerPC code fragment, the name of the fragment will be displayed along with the offset into the fragment of the particular line of code. The name of a PowerPC code fragment can be either an application name, a library name, or the logical name given to any piece of PowerPC code that is stored in a fragment. Usually, the name will implicate your application, another application, or a piece of System Software code.
If the code is 68K code or is not part of a CFM library, the result of the command may be simply information about the Mac OS memory manager block in which the code resides. This is less immediately useful than a library name, but you can sometimes figure out whose code it is by examining the memory block for clues. In particular, if the memory block is being tracked by the Resource Manager, then additional information about the resource, including the file it came from, are included in the output of the command. A good second stab is to take a look at the first several bytes of the memory block, which sometimes contain symbols or character codes (e.g. 'PLUG', or '"Apple Menu Options") which will sometimes bring you closer to the identity of the code's owner.
In worst case scenarios, the code you crashed in is not even code that the Mac OS memory manager keeps track of, and all MacsBug can muster is that the address is "in RAM but not in a known heap." Aside from displaying memory around the program counter and hoping for a lucky find, you are pretty much lost. This doesn't mean you aren't going to find the bug, it just means you have to do so without the benefit of knowing what the crashing code is.
Since data and memory coexist in the same address space of the Macintosh, and since the Mac OS provides only limited memory protection (for file-mapped PowerPC libraries), it is entirely possible that the CPU will be handed a chunk of data and be asked to execute it as code. If you are at all familiar with the assembly language for the CPU you are debugging, it will usually be immediately clear whether you are in data or code memory. When you look at the code at and around the program counter ("ip"), does it look like assembly language or does it look like MacsBug trying to translate monkey talk to a rhinoceros? Common giveaways are if the disassembly contains a lot of "ORI.B" instructions (68K), or a lot of "dc.l"" instructions (PowerPC). This is because the hex representations of these instructions are 0x00000000, and data memory usually contains a lot of zeroed-out memory.
If the memory looks like data, you are one step further to solving your bug. Usually the data contains patterns or text which are characteristic of the code that owns it. Find the beginning of the memory block with the "wh" command, and display memory from the beginning of the block until you find something that looks incriminating. Whether you can cinch the owner or not, it is time to move on to the next stage.
How Did I Get Here?
If you have been lucky, you now know the fundamental cause of the crash, and you know whose code it is. That knowledge is mostly useful for providing clues to answer the ultimate question: "how did my cute fuzzy bunny code get coerced off its path into the cruel, crashing world?" Unless the nature of the crash or the location of the code was telling enough to reveal an obvious solution, you must now attempt to turn back the clock and study the sequence of events that led up to the crash. Fortunately, a time machine is built into most calling conventions in the form of stack based link addresses.
When a subroutine is called, the caller needs to communicate to the subroutine how it will return control to the caller. It is abundantly common for this information to be passed along on the stack. As each subroutine in turn calls its own subroutines, a bread crumb trail is left that snakes all the way back up the stack to the original caller. Examining this trail is referred to as performing a "stack crawl." If you were really hard up, you could display memory from the stack pointer for several hundred bytes, reason out the math implied by the values in the various positions, and determine for yourself which parts of the stack refer to return addresses. Fortunately, MacsBug does a fairly good job of doing this with its 'sc' command.
The "sc" command was originally only able to examine the stack for 68K calling conventions, but today it is capable of looking for both 68K and PowerPC subroutine calls, and listing them both in the same stack crawl listing. In the output, each line represents the location of an instruction that caused a subroutine to be called. The last entry displayed is the last subroutine call MacsBug could decipher from the stack. If you look at the list from bottom to top, you are examining the subroutine history, from most recent to oldest, that led to this point. Generally, you are looking for a piece of code that looks like it is yours. If you find it, disassemble the code at that address ("ip <address>") to see if you can figure out exactly which of your code it is. Once you determine this, your strategy shifts from unintentional to intentional debugging - move on to a source level debugger if you so desire. You can now set break points at the crashing subroutine call and reproduce the crash on your terms.
There's No Stack Crawl!
Sometimes, as you should expect by now, things do not work out quite so peachy. The stack and link registers are sometimes in a state such that MacsBug is of no help in examining the stack crawl. Situations such as these call for desperate actions. Even if MacsBug refuses to produce a useful stack crawl, you can search for clues by looking at the link register contents yourself. Remember the "wh" command? Try "wh lr" to get information about the address in the link register, and "ipp lr" to disassemble the code surrounding the address (and hopefully the code that got you where you are).
MacsBug also provides a second stack crawl command, "sc7", for situations where the first does not pan out. Its output is identical in structure to that of "sc", but it is much less finicky about what constitutes a "return address" on the stack. The "sc" command actually iterates backwards up the stack, examining the locations which algorithmically (based on the well-defined calling conventions) should point to code which called a subroutine in the call chain. The "sc7" command, on the other hand, will examine every address on the stack, and as long as the code an address points to appears to immediately follow an A-trap or subroutine call, it will list the address as a "possible return address." Thus, the results of the "sc7" command need to be taken with great skepticism, but as I said, this is a command for desperate situations.
I Mean There's No Stack Crawl!
If the stack crawl is not panning out in any way, shape, or form, there is another technique that may be of aid. For whatever reason, the code you crashed in is just not conducive to the usual stack crawl analysis. If the crash you are debugging, however, can be temporarily avoided, then it may bode well to circumvent the crash, trace until the code you are executing is returned from, and see if the stack at this point is in better shape. If not, and if your crash-workaround lasts, just keep tracing until you do get some place interesting!
Crashing code can be avoided either by jumping past it (directly setting the PC register to a later address, e.g. "PC = PC + #12"), or by manipulating the registers such that a crash does not occur (put a valid address like MacsBug's built-in "playmem" in the register that is causing a memory exception). As you become more comfortable with direct manipulation in MacsBug, you will find that your debugging options increase immensely. Be aware, though, that you can make a bad situation worse by trying to "fix" things yourself in MacsBug. Hopefully, if you are reading this article, you understand that the risks of data loss and corruption are higher than average on the machine of a programmer. I am usually more interested in getting the bug fixed than playing it safe with my computer, but then again, I do make regular backups!
Part Two: Custom MacsBug Features
MacsBug has always been evolving. Three important features of MacsBug - the ability to define macros, the ability to define templates, and the ability write dcmds - have enabled MacsBug to evolve, meeting the needs of programmers over the years with relatively few changes to the core of the program. I shall now give an overview of these features, because MacsBug simply can not be appreciated without understanding these gems.
A MacsBug macro, like a macro in other programs, is a word that is translated when it is evaluated to mean something else. A simple example is the "windlist" macro. You might type this and assume it is a command that is built into MacsBug, but it is in fact a macro! Macros are defined by resources of type 'mxbm', several of which are included in MacsBug's own resource fork. The list of macros that MacsBug knows about can be displayed by typing "mcd" into MacsBug. If you look at the resulting list, you will see the meaning of the "windlist" macro is defined as "dm @WindowList WindowRecord", where "dm" is a built-in MacsBug command, "WindowList" is another macro defined as the number 0x09D6, and "WindowRecord" is a template. This simple macro, then, means "display the memory pointed to by an address at location 0x09D6, and format the contents of the memory with the WindowRecord template."
Templates, packaged in resources of type 'mxwt', contain information about how to format memory that is displayed by MacsBug. Using the MacsBug "dm" command (display memory), a template is invoked by typing its name as a second parameter, after the memory address you wish to display from. Without a template, MacsBug uses a default output structure, which consists of lines of hex values with an ASCII translation to the right side. Often, a more readable format is preferable. Specifically, you would prefer a format as similar as possible to the structure definition in your code. In the case of WindowRecord, a simple memory display produces the following output:
Displaying memory from @09D6
005D45D0 0000 0071 DE40 C000 0071 DF04 0000 8000 ***qÞ@¿**qþ***Ä*
005D45E0 0000 0000 0094 0068 007D 5DD8 0000 2DBC *****î*h*}]ÿ**-º
005D45F0 0071 DE24 0000 0000 0000 FFFF FFFF FFFF *qÞ$******ùùùùùù
005D4600 0064 005C 0001 0001 0008 0058 73B0 005D *d*\*******XsÉ*]
005D4610 431C 0000 0001 0000 0001 0009 0000 0000 C***************
005D4620 0000 00FF 0000 0000 0000 0000 0000 0000 ***ù************
005D4630 0000 0000 0000 0000 0000 0000 FFF3 0101 ************ùÛ**
005D4640 0100 0071 DEE4 0071 DE38 0071 DE78 0000 ***qÞî*qÞ8*qÞx**
005D4650 2DC4 0000 0000 0071 DE5C 0041 0000 0000 -ü*****qÞ\*A****
While displaying the same memory with the WindowRecord template produces:
dm @windowlist WindowRecord
Displaying WindowRecord at 005D45D0
005D45E0 portRect #0 #0 #148 #104 (w #104, h #148)
005D45E8 visRgn 007D5DD8 -> #0 #0 #148 #104 [non-rect]
005D45EC clipRgn 00002DBC -> [wide-open region]
005D463C windowKind FFF3
005D463E visible true
005D463F hilited true
005D4640 goAwayFlag true
005D4641 spareFlag false
005D4642 strucRgn 0071DEE4 -> #13 #87 #181 #193 [non-rect]
005D4646 contRgn 0071DE38 -> #32 #188 #180 #192 [non-rect]
005D464A updateRgn 0071DE78 -> [empty]
005D464E windowDefProc 00002DC4 -> 008D9C00 ->
005D4652 dataHandle NIL
005D4656 titleHandle 0071DE5C -> 008DA0E0 -> "Calculator"
005D465A titleWidth 0041
005D465C controlList NIL
005D4660 nextWindow NIL
005D4664 windowPic NIL
005D4668 refCon 00000000
Instead of 136 bytes of hex spewed onto the screen, you get a fairly nice looking description of the window in the form of a WindowRecord structure. An added bonus of displaying memory via a template: a template definition can contain certain types which MacsBug knows are referred to by pointers in the memory being displayed. When MacsBug displays these elements, it follows the addresses to the data of interest, and displays that in a readable format. Great examples in the listing above are the "visRgn" and "titleHandle" fields.
"Debugger commands", or dcmds, are independent pieces of code which MacsBug allows you to execute from the command line while you are debugging. These commands, packaged in 'dcmd' resources, are the most significant contributors to MacsBug's extensibility. There are limitations to the types of code that dcmds can execute, and output is restricted to text displayed via MacsBug's text screen, but for most of your specialized debugging needs, the facilities of the dcmd architecture will be perfect. Like macros and templates, much of the behavior taken as "built-in" to MacsBug is actually implemented in the various dcmds that come pre-installed with MacsBug. To get a list of the dcmds available to you from MacsBug, type the "dcmd" command itself. The result is a list of the commands at your disposal. To get help on any of them, type "help [name of dcmd]" (or just "? [name of dcmd]"), which will cause the dcmd itself to explain its own functionality.
Roll Your Own Debugger
Obviously the power of extensibility is most appreciated when you have occasion to require a custom addition to the built-in functionality of MacsBug. Some MacsBug enhancements can be found in the public domain, or in friends' private collections. These enhancements are easily installed by dragging the resource files in which they were delivered into a folder called "MacsBug Preferences" inside your Preferences folder. MacsBug will retrieve the resources from all such resource files when it initializes itself early in the boot process. You could also use ResEdit to copy and paste the resources into MacsBug itself, but by leaving them outside of MacsBug, you can easily keep the enhancements when you update to future releases of MacsBug.
If you can't find the enhancements you need on the Internet or from your geeky friends, you will have to resort to writing your own. Fortunately, writing templates and macros is very simple, and programming dcmds takes only a bit more effort. To get started on templates and macros, open up MacsBug with ResEdit and look at some of the built in examples. You can copy the 'TMPL' resources from MacsBug to your own resource file, which will allow you to edit macros and templates with ResEdit's template (not to be confused with MacsBug templates) editor. A sample dcmd project is included in both CodeWarrior and MPW format in the MacsBug release archive. Once you've learned to concoct your own MacsBug enhancements, you will be free to dream up completely new ways of solving your debugging problems.
Part Three: MacsBug Enters the Millennium
Now it is 1999, and the MacsBug available on Apple's web site is much different than the MacsBug first rolled off the line in 1981. Here are some of the features of MacsBug that might surprise you if you haven't been paying attention to the most recent MacsBug releases.
MacsBug supports simple operations involving the Mouse. Yes, break into MacsBug and move the mouse around. It's not a bug - click on some items in MacsBug's display to get a feel for the effects. Mouse support is handy when you don't want to retype an address that is already displayed on the screen into a MacsBug command expression. Another popular use is for changing the height of the PC section of MacsBug's display - just click and drag!
CFM Symbol Execution
How many times have you wanted to execute a particular Mac OS API call from within MacsBug, but you didn't have a dcmd to do it for you? If you are an especially adept MacsBug user, you might have done this in the past by extreme MacsBug direct manipulation. You would write down or save in macros the contents of the registers, switch the PC to a free spot in memory, type out the hexadecimal code that prepares the stack and calls an A-Trap, and trace over the A-Trap before finally restoring the state of registers, the PC, and the stack to their original values.
Thankfully, MacsBug now possesses an amazing functionality that obviates the need for this kind of hacking. You can ask MacsBug to execute any routine that is exported by a CFM library, directly from the MacsBug command line. Since almost every Mac OS API routine is exported through some library (even if it's only glue code to get to the 68K implementation), you have access to almost any Mac OS API call that could assist in debugging your code! To execute an exported routine in MacsBug, simply preface the routine name with the © symbol (option-G), put the parameter list in parentheses as you would in a C function call, and press return. The return value of the function is the MacsBug expression value returned by the command. For example, if you type the MacsBug command:
©MoveWindow(@WindowList, #100, #100, 0)
The window that is front-most in the current application is moved to location (100, 100) on the screen, and MacsBug returns the following output:
©MoveWindow(@09D6, #100, #200, 0) = $000000FF #255 #255 '***ù'
The value of the expression in this case should be ignored, because MoveWindow does not have a defined return value. Another simple example:
Causes MacsBug to return the following output:
©HMGetBalloons() = $00000000 #0 #0 '****'
Which tells me that Balloon Help is currently inactive. Clearly, there is enormous potential for this feature, and you will only know how handy it is when you remember that it's there in the midst of a really tough bug.
The feature does have some drawbacks, of course. First, you must phrase all of the parameters to the routine as explicit values. MacsBug doesn't know how to translate the text "true" into a valid numeric value for the function you are calling. Second, if any of parameters are pointers that receive an output value (VAR parameters), you are responsible for passing an address to a legitimate memory location. I am in the habit of using playmem for such things. For example:
©GetIndString(playmem, #128, 1)
Which gets the first string from 'STR#' resource 128 and copies the string to the "play" memory reserved by MacsBug. You can now display the contents of the string by typing "dm playmem". Last, and most importantly, MacsBug will not police the safety of the CFM call you are making! Calling routines at certain times and with certain parameters may spell death for your debugging session. Among the factors to consider are: the interrupt level, the reentrancy of code that is currently being executed, and the state of the file system if the routine you're calling will access files. In general, this feature falls into the "desperate actions" category, but as you come to depend on it, you may enjoy being desperate!
PowerPC Watch Points
A number of MacsBug's features evolved during the years when Macintosh computers ran solely on Motorola's 680x0 series of CPU. Since the introduction of the PowerPC based Macintoshes, MacsBug has continued to evolve, albeit sometimes slowly, implementing functionality for PowerPC that matches what we grew accustomed to on 680x0 computers.
One very useful feature for 680x0 code is called "step spying", and it can still be activated by using the "ss" command. Basically, what step spying does is to execute instructions one at a time, checking at each step whether a particular range of memory was written to. This is very useful when you have found a bug that is caused by the wrong data being written to a location in memory, and you want to find the culprit. You set a step spy on the address, wait for the computer to sluggishly execute instructions one by one, and hopefully when you break into MacsBug again, you are staring at the code that writes the bad value to that address.
Since step spying is implemented by stepping, instruction by instruction, through 680x0 code, it is completely useless when the offending code is in PowerPC format. MacsBug users who have needed step spy style functionality for PowerPC code have been essentially out of luck for quite some time. Steadily, however, a new functionality in MacsBug called "watch points" has been developing. With release 6.5.4a6 of MacsBug, this functionality has become remarkably more functional and substantially less likely to crash your machine.
To set a watch point in MacsBug, type "wp" followed by one or two addresses. If you specify one address, MacsBug will watch the 4 bytes at that location. If you specify two addresses, the range of memory bounded by the two addresses will be watched. Watch points are implemented by using the memory protection feature of the PowerPC processor. When you specify a watch point, the memory pages that contain the given range are write protected. When some piece of code tries to write to one of the protected pages, MacsBug's exception handler is called, which triggers a break into MacsBug if the attempted write was in the desired range. Watch points stay in effect until you clear them by using the "wpc" command.
The watch points functionality is in its infancy. There are a number of limitations that are described in the release notes:
- To use the watch points functionality, you must install the Emulator Update file from MacsBug's distribution archive into your System Folder.
- The emulator update only works on Power Mac 9500 and newer machines.
- No more than one watch point can be set at a time.
- When a watch point is hit, you may not safely trace or step until clearing the watch point.
- Watch points can not be set on memory locations on the stack, memory that will be addressed with an atomic instruction ("lwarx" or "stwcx"), or the memory location zero.
The limitations are significant, but those of you who have been waiting patiently for a successor to step spying will be able to look past them and appreciate the mere existence of this feature. I can see your head nodding along right now, "I don't care if it's buggy, I need that feature!" Well, there you have it.
MacsBug is a very powerful debugger. If you didn't believe so before reading this article, I hope you will now agree, and I hope you will find occasion to make use of its power in your own debugging pursuits. For those of you who didn't need to be convinced, I hope that the new features will refocus your attention on the intrepid debugger that, to this day, is critical to Mac OS development everywhere. Remember, the most powerful way to grow the functionality and evolve this debugger into the next century is by writing and distributing your own dcmds, macros, and templates to the Mac OS developer community. MacsBug's evolution is not controlled by one person, one company, or one country - it's up to you to continue improvising and revolutionizing the functionality of a very simple, very complicated development tool.
The following are good references for those of you who need a more detailed review of MacsBug's functionality.
Apple Computer, MacsBug Reference and Debugging Guide, available online at
Othmer, Konstantin and Jim Straus. Debugging Macintosh software with MacsBug, Addison Wesley Publishing Company, Inc.
Thanks to Dave Lyons and Jim Murphy for reviewing this article, and for being the primary caretakers of MacsBug for the past several years.
Daniel Jalkut is a software engineer in the Mac OS System Software group at Apple Computer. In his spare time, Daniel works on solo projects under the moniker Red Sweater Software, and enjoys being a pedestrian in the beautiful city of San Francisco. You can contact him firstname.lastname@example.org, or view his home page at http://www.sirius.com/~jalkut