Xtrap, A Hardware Debugger

By Mike Scanlin, Mountain View, CA

A New Macintosh Debugging Tool

Software debuggers have come a long way since the Mac was first introduced. Among other improvements, symbolic and source-level debugging are awesome. However, there are times when it is difficult or impossible to debug software by using software. Sometimes the observing software (the debugger) interferes with the software being observed. Other times there are situations where you just cannot do what you need with a software debugger. Have you ever tried to set a breakpoint in ROM or in a VBL? Have you ever tried to debug a time-sensitive piece of code such as a driver? Have you ever wondered if somewhere, deep within the core of your program, you were unknowingly reading or writing to memory location zero? Have you ever had a bug that wedged your machine so bad that your debugger was out to lunch until you rebooted? These are all examples of the types of problems that the new Xtrap hardware debugger from HotWire Labs can help you solve.

Real-Time Trace

In a nutshell, Xtrap is a hardware device that monitors and records all of the activity on the address bus and data bus of the 68000 and allows you to play back that activity at a later time. It allows you to set your own hardware-defined breakpoint conditions (for example, “stop recording cycles as soon as there is a read or write cycle to memory location zero”) and to play back the last 32K worth of cycles before or after the breakpoint conditions are met. This makes it easy to see how your program got into a particular state that you’ve defined and/or what it did after it got to that point. Because this is a hardware device monitoring the processor it is totally transparent -- your program will run at it’s normal speed and nothing will prevent Xtrap from recording cycles (including a munged debugger and/or operating system).

A Simple Example

Here is a simple example to give you a feel for what Xtrap is all about. Let’s say you have a program that is causing an illegal instruction and that by the time you see the error the PC is in a random location (and you don’t know how it got there). So you configure Xtrap to record cycles until you get an illegal instruction. You do this by asking Xtrap to record cycles until there is a data read cycle to the illegal instruction exception vector (memory location $10). Once it gets one, you ask Xtrap to show you what happened before that illegal instruction cycle. You could end up with this output:

-00039:1B572E:4EBA:     :              ;Jsr     BugProc
-00038:1B5730:FFD0:oe   :
-00037:1B5700:48E7:     :BugProc       ;Movem.L D5-D3,-(A7)
-00036:1BDCDA:001B:w’••’:
-00035:1BDCDC:5732:w’W2':
-00034:1B5702:1C00:oe   :BugProc+0002
-00033:1B5704:61E4:     :BugProc+0004  ;Bsr.S   TrashRegs
-00032:1BDCD8>0000<w’••’:
-00031:1BDCD6>0000<w’••’:
-00030:1BDCD4>0000<w’••’:
-00029:1BDCD2:0000:w’••’:
-00028:1BDCD0>0000<w’••’:
-00027:1BDCCE>0000<w’••’:
-00026:1B5706:4CDF:sof  :
-00025:1BDCCA>001B<w’••’:
-00024:1BDCCC>5706<w’W•’:
-00023:1B56EA:7603:     :TrashRegs     ;Moveq   #$03,D3
-00022:1B56EC:7804:     :TrashReg+0002 ;Moveq   #$04,D4
-00021:1B56EE:7A05:     :TrashReg+0004 ;Moveq   #$05,D5
-00020:1B56F0:4E75:     :TrashReg+0006 ;Rts
-00019:1B56F2:8009:sof  :
-00018:1BDCCA>001B<r’••’:
-00017:1BDCCC>5706<r’W•’:
-00016:1B5706:4CDF:     :BugProc+0006  ;Movem.L (A7)+,D3/D5
-00015:1B5708:0028:oe   :BugProc+0008
-00014:1B570A:4E75:     :BugProc+000A  ;Rts
-00013:1BDCCE>0000<r’••’:
-00012:1BDCD0>0000<r’••’:
-00011:1BDCD2:0000:r’••’:
-00010:1BDCD4>0000<r’••’:
-00009:1BDCD6>0000<r’••’:
-00008:1B570C:8007:sof  :
-00007:1BDCD6>0000<r’••’:
-00006:1BDCD8>0000<r’••’:
-00005:000000:00F8:     :              ;????
-00004:000002:0000:oe   :
-00003:1BDCD8>0000<w’••’:
-00002:1BDCD4>2000<w’ •’:
-00001:1BDCD6>0000<w’••’:
 00000:000010:0039:r’•9':
 00001:000012:9E08:r’••’:

The first column is for the cycle index. Negative indexes are cycles that occurred before the breakpoint conditions we specified were met. Positive indexes occurred after the breakpoint conditions were met. As you will see in the next example, you can tell Xtrap what percentage (0..99) of the trace buffer you want reserved for cycles that occur after the breakpoint conditions have been met. Normally you’ll be interested in recording cycles before the breakpoint conditions are met so you can look backwards and see how you got into the state you’ve defined, but the ability to look forwards from that point comes in handy, too.

The second column is the value of the address bus for that cycle. Notice that for cycle 0 the address bus is $000010, the illegal instruction vector. Once Xtrap saw this value on the address bus it stopped recording cycles into the trace buffer (because we asked it to record cycles until we got an access to the illegal instruction vector) and generated a non-maskable interrupt to put us in MacsBug. By definition, cycle 0 is always the cycle within the trace buffer (it’s a cyclical buffer) than meets the breakpoint conditions that we’ve defined.

The third column is the value of the data bus for that cycle. The “>” and “<“ that you see around the data bus value in some cycles means that the value at disassembly time at that address is different than the value at trace record time at that address.

The fourth column is the cycle type:

 blank  = opcode cycle
 oe=  opcode extension cycle
 sof  = spurious opcode fetch cycle
 w’xx’  = data write cycle, ‘xx’ is the value written
 r’xx’  = data read cycle, ‘xx’ is the value read

The fifth column is the symbolic address column. Xtrap supports all MacsBug-style symbols and, in addition, allows you to add your own symbols if you want (more on this later).

The sixth column is the disassembly column.

The “????” at cycle -5 is the instruction that caused the illegal instruction exception. How did we get there? Well, if you look at cycle -39 you see that there is a Jsr to BugProc. The first thing BugProc does is save registers D5-D3 (the reason it is written this way instead of D3-D5 is because when Xtrap disassembles a Movem instruction it displays the registers in the order that they are pushed or popped -- this is useful for determining the registers’ values by looking at the corresponding data write or data read cycles). BugProc then calls TrashRegs at cycle -33. Cycles -32 to -27 are the data write cycles corresponding to the Movem.L D5-D3,-(SP) at cycle -37. They occur here, instead of immediately after the Movem instruction, because of the 68000’s pipelining.

Okay, now we’re getting close to the bug in this example code. Notice when TrashRegs returns to BugProc that BugProc only restores registers D3 and D5 (and not D4). That means it’s popping one less register than it pushed -- a sure way to end up with a random location for the PC when the Rts instruction is executed. Looking in the source you will see that you need to replace “D3/D5” with “D3-D5” to fix this bug.

Pretty cool, huh? For comparison with the Xtrap output, here’s the original source code:

 TrashRegsPROC
 Moveq  #3,D3
 Moveq  #4,D4
 Moveq  #5,D5
 Rts
 DC.B $80,’TrashRegs’
 DC.W $0000
 ENDP

 BugProcPROC
 Movem.LD3-D5,-(SP)
 Bsr.S  TrashRegs
 Movem.L(SP)+,D3/D5
 Rts
 DC.B $80,’BugProc’
 DC.W $0000
 ENDP

Installation

Currently, Xtrap only works on a Mac SE (although a Mac II class machine version is in the works). It comes with two pieces of hardware: one is a circuit board that goes in the SE’s internal slot and the other is a 5" x 6" x 1" external control panel that hooks onto the internal board via a 5' cable. The control panel has 5 buttons and 15 LEDs:

The manual gives two sets of installation instructions; one for people who are familiar with installing Mac SE boards and a more detailed one for those who need a little help with removing the Mac SE’s case and motherboard. The complete installation takes about ten minutes.

Interface

You communicate with Xtrap (i.e. define breakpoints and get disassemblies of the trace buffer) through MacsBug dcmds (TMON Pro support is planned and may be available by the time you read this). To install the dcmds you paste the provided resources into the Debugger Prefs file in your System Folder and reboot. They will be loaded automatically the next time MacsBug installs itself.

Accessing Memory Location Zero

What follows is a complete example debug session with Xtrap that illustrates how you would check if your program is reading or writing memory location zero (which is not altogether illegal, but certainly is not very good programming practice). Interestingly enough, both MPW Shell and Finder do this. I discovered this within five minutes of hooking up Xtrap for the first time.

(Note: Xtrap is far more powerful than the Mr. Bus Error INIT in terms of checking for location zero access. Mr. Bus Error (which stuffs $00F0F0F1 into location zero) only catches dereferenced NIL pointer errors; it won’t tell you if you are reading a byte from location zero (NIL PString pointer, for instance), nor will it tell you if you are writing a byte, word or long to location zero.)

Let’s say you’ve installed the Xtrap hardware and software and have launched MPW Shell (version 3.0). The first thing you do is press the programmer’s switch (or the NMI button on the Xtrap control panel; they are equivalent) to get into MacsBug.

You can define up to four separate breakpoint conditions. Each breakpoint has an address value, an address mask, a data value, a data mask and a cycle type. You can specify things like: any access to a certain address (read access, write access, or either), any read or write of a specific data value (data read or write, opcode fetch, or either), any cycle type (read, write, supervisor, user or interrupt acknowledge) or any combination of these. An example of when you’d want to use a mask is where you use a data mask to mask off all but the hi bit of the MemErr low memory global -- you could then record cycles until a negative value (an error value) was written there.

Xtrap will continuously record cycles in a 32K cyclical buffer until the first condition is met and any one of the last three conditions are met. To set the breakpoint conditions that will tell Xtrap to stop recording cycles after location zero is accessed, the commands you issue are:

Type in MacsBug: Xb 1 o
Type in MacsBug: Xb 2 @0
Type in MacsBug: Xt 50
Push the “Arm” button on the Xtrap control panel
Type in MacsBug: G

“Xb” is the “Xtrap breakpoint” command used to set breakpoint conditions. The first parameter is the breakpoint number (1 to 4). The second parameter is the breakpoint condition. In this case the first breakpoint it set to “o” for “opcode”, which is Xtrap’s version of a NOP breakpoint (because it will match any instruction). This is necessary because Xtrap requires that the first breakpoint condition be met before breakpoint conditions 2, 3 or 4 are checked (and in our example, we only want one real breakpoint condition). The second breakpoint above is set for any access to memory location zero. The table of breakpoints that Xb echoes out every time you change a breakpoint condition now looks like this:

T Adrs Mask Data Mask Type

1 :XXXXXX:FFFFFF:XXXX:FFFF:Opcode

2 :000000:000000:XXXX:FFFF:

3 :XXXXXX:FFFFFF:XXXX:FFFF:Disabled

4 :XXXXXX:FFFFFF:XXXX:FFFF:Disabled

“Xt” is the “Xtrap trace” command which starts Xtrap recording cycles. The “50” means that 50% of the 32K trace buffer will contain cycles from before the breakpoint conditions are met and 50% will be from cycles after the breakpoint conditions are met. The reason why you would want to record cycles after the breakpoint conditions are met instead of or in addition to before the breakpoint conditions are met is because sometimes it’s easier to define a breakpoint condition that happens just before the piece of code you are examining is executed. You could put some unique instruction there and then configure Xtrap to record the first 32K cycles after that unique instruction was executed. This may seem strange but in practice I found this feature useful.

Okay, after the “G” command above you’re back in the Shell and Xtrap is recording cycles and looking for any read or write access to location zero (it does this by comparing the address bus on each cycle to the zero value we’ve specified).

I discovered by accident that if you choose Select All when in MPW Shell then certain LEDs on the Xtrap control panel light up indicating that something just read or wrote to location zero (and another LED lights up telling you that Xtrap has stopped recording cycles).

Now if you ask Xtrap to disassemble the cycles preceding the memory location zero access (by using the “Xd” for “Xtrap disassemble” command) you would get this listing:

-00087:1D4C96:7000:     :getSelec+0082 ;Moveq   #$00,D0
-00086:1D4C98:4CEE:     :getSelec+0084 ;Movem.L
 $FFEC(A6),D6/D7/A3/A4
-00085:1D4C9A:18C0:oe   :
-00084:1D4C9C:FFEC:oe   :
-00083:1D4C9E:4E5E:     :getSelec+008A ;Unlk    A6
-00082:336CE2:0000:r’••’:
-00081:336CE4:0001:r’••’:
-00080:336CE6:0000:r’••’:
-00079:336CE8:0101:r’••’:
-00078:336CEA:0033:r’•3':
-00077:336CEC:6D48:r’mH’:
-00076:336CEE:001D:r’••’:
-00075:336CF0:9B6E:r’•n’:
-00074:336CF2:6D06:r’m•’:
-00073:1D4CA0:4E75:     :getSelec+008C ;Rts
-00072:336CF6:0033:r’•3':
-00071:336CF8:6D68:r’mh’:
-00070:1D4CA2:8C67:sof  :
-00069:336CFA:0031:r’•1':
-00068:336CFC:D9B4:r’••’:
-00067:31D9B4:508F:     :SetFileM+01B6 ;Addq.L  #8,A7
-00066:31D9B6:2F00:     :SetFileM+01B8 ;Move.L  D0,-(A7)
-00065:31D9B8:4EBA:     :SetFileM+01BA ;Jsr     fixSelectSize
-00064:31D9BA:FD80:oe   :
-00063:336D04:0000:w’••’:
-00062:336D02:0000:w’••’:
-00061:31D73A:4E56:     :fixSelectSize ;Link    A6,#$FFFC
-00060:336CFE:0031:w’•1':
-00059:336D00:D9BC:w’••’:
-00058:31D73C:FFFC:oe   :
-00057:31D73E:48E7:     :fixSelec+0004 ;Movem.L A4/A3/D7,-(A7)
-00056:336CFA:0033:w’•3':
-00055:336CFC:6D68:w’mh’:
-00054:31D740:0118:oe   :
-00053:31D742:2E2E:     :fixSelec+0008 ;Move.L  $000C(A6),D7
-00052:336CF4:9B6E:w’•n’:
-00051:336CF2:001D:w’••’:
-00050:336CF0:6D48:w’mH’:
-00049:336CEE:0033:w’•3':
-00048:336CEC:0101:w’••’:
-00047:336CEA:0000:w’••’:
-00046:31D744:000C:oe   :
-00045:31D746:266E:     :fixSelec+000C ;Move.L  $0008(A6),A3
-00044:336D06:0000:r’••’:
-00043:336D08:0040:r’•@’:
-00042:31D748:0008:oe   :
-00041:31D74A:4AAE:     :fixSelec+0010 ;Tst.L   $0010(A6)
-00040:336D02:0000:r’••’:
-00039:336D04:0000:r’••’:
-00038:31D74C:0010:oe   :
-00037:31D74E:6778:     :fixSelec+0014 ;Beq.S   fixSelec+008E
-00036:336D0A:0000:r’••’:
-00035:336D0C:0001:r’••’:
-00034:31D750:700D:     :fixSelec+0016 ;Moveq   #$0D,D0
-00033:31D752:2F00:     :fixSelec+0018 ;Move.L  D0,-(A7)
-00032:31D754:2F0B:     :fixSelec+001A ;Move.L  A3,-(A7)
-00031:31D756:4EAD:     :fixSelec+001C ;Jsr     $026A(A5)
-00030:336CE8:000D:w’••’:
-00029:336CE6:0000:w’••’:
-00028:31D758:026A:oe   :
-00027:336CE4:0000:w’••’:
-00026:336CE2:0000:w’••’:
-00025:33BDF0:4EF9:     :              ;Jmp     strchr
-00024:336CDE:0031:w’•1':
-00023:336CE0:D75A:w’•Z’:
-00022:33BDF2:001B:oe   :
-00021:33BDF4:F9FC:oe   :
-00020:1BF9FC:4E56:     :strchr        ;Link    A6,#$0000
-00019:1BF9FE:0000:oe   :
-00018:1BFA00:48E7:     :strchr+0004   ;Movem.L A3/D7,-(A7)
-00017:336CDA:0033:w’•3':
-00016:336CDC:6CFA:w’l•’:
-00015:1BFA02:0110:oe   :
-00014:1BFA04:2E2E:     :strchr+0008   ;Move.L  $000C(A6),D7
-00013:336CD8:0000:w’••’:
-00012:336CD6:0000:w’••’:
-00011:336CD4:0040:w’•@’:
-00010:336CD2:0000:w’••’:
-00009:1BFA06:000C:oe   :
-00008:1BFA08:266E:     :strchr+000C   ;Move.L  $0008(A6),A3
-00007:336CE6:0000:r’••’:
-00006:336CE8:000D:r’••’:
-00005:1BFA0A:0008:oe   :
-00004:1BFA0C:1013:     :strchr+0010   ;Move.B  (A3),D0
-00003:336CE2:0000:r’••’:
-00002:336CE4:0000:r’••’:
-00001:1BFA0E:4880:     :strchr+0012   ;Ext.W   D0
 00000:000000:00  :r’•’ :
 00001:1BFA10:48C0:     :strchr+0014   ;Ext.L   D0
 00002:1BFA12:1207:     :strchr+0016   ;Move.B  D7,D1
 00003:1BFA14:4881:     :strchr+0018   ;Ext.W   D1
 00004:1BFA16:48C1:     :strchr+001A   ;Ext.L   D1
 00005:1BFA18:B280:     :strchr+001C   ;Cmp.L   D0,D1
 00006:1BFA1A:6604:     :strchr+001E   ;Bne.S   strchr+0024
 00007:1BFA1C:200B:sof  :
 00008:1BFA20:4A1B:     :strchr+0024   ;Tst.B   (A3)+
 00009:1BFA22:66E8:     :strchr+0026   ;Bne.S   strchr+0010
 00010:000000:00  :r’•’ :
 00011:1BFA24:7000:     :strchr+0028   ;Moveq   #$00,D0
 00012:1BFA26:4CEE:     :strchr+002A   ;Movem.L $FFF8(A6),D7/A3
 00013:1BFA28:0880:oe   :
 00014:1BFA2A:FFF8:oe   :
 00015:1BFA2C:4E5E:     :strchr+0030   ;Unlk    A6
 00016:336CD2:0000:r’••’:
 00017:336CD4:0040:r’•@’:
 00018:336CD6:0000:r’••’:
 00019:336CD8:0000:r’••’:
 00020:336CDA:0033:r’•3':
 00021:1BFA2E:4E75:     :strchr+0032   ;Rts
 00022:336CDA:0033:r’•3':
 00023:336CDC:6CFA:r’l•’:
 00024:1BFA30:8673:sof  :
 00025:336CDE:0031:r’•1':
 00026:336CE0:D75A:r’•Z’:
 00027:31D75A:4A80:     :fixSelec+0020 ;Tst.L   D0

Fortunately, MPW Shell has labels in it so we are given a clue as to what this code is doing. Basically, SetFileMenu is calling getSelection and fixSelectSize. FixSelectSize calls strchr which is the routine that is accessing location zero (in cycles 0 and 10). Cycle 0 is the data read cycle corresponding to the Move.B (A3),D0 instruction at cycle -4. So, we know A3 contains $0 at that point. Looking at the previous instruction, Move.L 8(A6),A3, we can see that the string pointer passed to strchr is the culprit. Where does it come from? Well, at cycle -32 we can see that A3 is pushed on the stack. At cycle -45 A3 is set equal to the first parameter of fixSelectionSize. If we look above to cycle -66 we see that SetFileMenu passes the result from getSelection to fixSelectSize. And if we look at cycle -87 we can see that getSelection is indeed returning $0 in D0.

I’m not going to do it here, but you could continue looking backwards from this point and determine what it was that caused getSelection to return zero. Maybe it’s by design and MPW Shell is just using location zero as a scratch area. Then again, maybe this is a subtle bug that doesn’t have any ill side-effects. In any case I think you can see that Xtrap has the ultimate “watchpoint” feature. It doesn’t slow down program execution and because it is monitoring the address bus directly nothing will get past it. You can optionally ask Xtrap to only flag read accesses to a certain location or to only flag write accesses (or both, as in this example).

For those who are interested, the Finder (version 6.1.5) writes a zero byte to location zero when you close any window (?).

Symbolic Debugging

Xtrap supports symbolic disassembly. It comes configured with symbols for all of the low memory globals and ROM addresses. MacsBug symbols are supported. In addition, you can add your own symbols (this is key for people like me who use TMON’s resource-relative label mechanism). Xtrap stores all of its symbols in its private 512K write-protected RAM so that they are preserved across warm boots. This is extremely useful if your program crashes the system or trashes MacsBug and forces you to reboot -- the symbols and trace buffer are still in tact so you can review what happened before you had to reboot.

Trace Filtering

There are times when you don’t want EVERY cycle recorded. For instance, if you are debugging a piece of code and it makes a call to BlockMove you probably don’t want the entire trace buffer to get filled with BlockMove’s instructions. To get around this problem, Xtrap provides a filtering mechanism where you can specify that you don’t want ROM cycles (any cycle where the PC is in ROM) recorded in the trace buffer.

The trace buffer contains 32K worth of cycles, not instructions. If you are interested in maximizing the number of instructions that are saved in the buffer then you can specify that Xtrap should record opcode cycles only (and not record data cycles). Of course, the downside of not recording data cycles is that it is much harder (and most of the time impossible) to reverse engineer what the values of the registers are at each instruction. Still, it will allow you to see if the flow of your program is what you expected.

Xdump

An MPW tool, Xdump, is provided with Xtrap. When executed it will dump the entire 32K trace buffer to a text file (in disassembled format like the above examples). This is useful if you want to munge on the data provided by Xtrap in some special way. Xtrap does have an Xf (Xtrap Find) dcmd that allows you to look for specific cycles in the buffer while in MacsBug but Xdump still comes in handy (especially if you want to diff two different traces).

Serial Port

Xtrap has a built in serial port (300 to 9.6K baud full duplex, RS-232C) that allows you to operate Xtrap from an external terminal, without using the target Mac’s screen, keyboard or mouse. I did not have the opportunity to test this feature.

Expandability

Because Xtrap uses Xilinx reprogrammable circuits, it is upgradable without hardware modifications. This makes it possible for HotWire Labs to write new features and send out the new resources for pasting into your Debugger Prefs file. You won’t need to upgrade your hardware. Also, the complete source code (in MPW C and assembly) is shipped with Xtrap so if, for example, you don’t like the format of the Xd disassemblies you are free to change it.

One particularly interesting possibility for the future is some kind of performance measurement feature. It would be more accurate than current performance measurement tools and would not slow down your program while measuring it. This hasn’t been written yet but is certainly feasible given the hardware.

Another cool idea is a “derived register values” feature where each cycle would have the values of the registers involved listed. This isn’t implemented in the version of Xtrap that I reviewed but HotWire has said that it may be a reality soon.

Summary

In terms of design and quality of product I give Xtrap a big thumb’s up. It is a clean, bug-free, well thought-out product. HotWire Labs has been extremely helpful for Xtrap tech support and very responsive to ideas for future improvements. They are truly interested in making the product fit the needs of Macintosh programmers.

In terms of general usefulness I would say that Xtrap is a must-have for full-time Mac developers and probably a justifiable tool for anyone who is a serious Mac hacker. In addition to being the only debugging tool available able to track down certain types of bugs, I have found Xtrap to be useful for tracking down bugs in general, too. It has more than paid for itself in the two months that I’ve been using it in terms of the time I’ve saved debugging. In addition, when you consider that typical 68000 logic analyzers cost between $4000 and $6000, have a smaller trace buffer, don’t have as many features and aren’t expandable, Xtrap is a very good deal indeed.

Xtrap Information

Version reviewed: 1.1

Price: $2000. 30-day money-back guarantee.

Requirements and availability: A Mac SE running MacsBug 6.2 version is available now. A Mac II class machine version should be available sometime in 1991. TMON Pro support is planned (and may be available by the time you read this).

HotWire Labs

P.O. Box 17731

Wichita, KS 67217

AppleLink: D4527

(316) 838-8849