Related Info: Memory Manager

Playing with Blocks

Memory manager structures - a software autopsy of your application's death!

By Brooks Bell, Bradenton, Florida

Let’s set the ‘Way Back’ machine for 1985. Programmers of the day complained heavily of the information that Apple was keeping to itself. Inside Macintosh just did not provide enough details.

The chapter on the Memory Manager, for example, was fuzzy on the meaning of certain fields in the Zone header. SparePtr, allocPtr even the ever-popular flags field were defined only as ‘used internally’. Even worse, fields such as cntRel, maxRel, and cntHandles were described as ‘not used’, when any debugger showed that they clearly contained valid data.

Of course, we have all by now learned that these things are hidden from programmers because they are fair game for change. What works for a 512K “Fat” Mac isn’t necessarily applicable to a Quadra with 64MB of RAM. We have grown accustomed to the trade off: Apple shields us from some OS details in exchange for blessed future hardware compatibility.

That is why it is something of a surprise to turn a few pages further in IM II and discover a very detailed description of the old 24 bit memory manager block structure. In retrospect, this knowledge led directly to the problems of 32 bit uncleanliness. Here is how to set the Master Pointer ‘lock’ bit and bypass the HLock trap overhead. Lo3Bytes is described - who needs StripAddress? Everything that you need to make a 32 bit dirty application is right here in black and white.

With the 32 bit memory manager and Inside Macintosh VI, having learned from their mistakes, Apple decided not to publish the Memory Manager block structures. Well written applications will use HGetState, HLock, etc There is no need to risk compatibility by accessing these structures directly.

However, when you are in the middle of debugging some code errors can crop up that cause an invalid heap. Because the Mac ROMs do not check parameters for errors, it is easy to pass a trap a bad value and have it write garbage all over the heap. It is also easy to move data into a disposed handle, store data beyond the bounds of an array, maybe even call BlockMove directly and blast data all over creation. Lacking sophisticated memory protection, the Macintosh heap is easily destroyed.

Perhaps the worst aspect of such damage is that fatal errors may not appear for some time. When they do show up in the form of the friendly ‘illegal instruction’, ‘Stack has collided with the heap’, ‘Bus Error’ or whatever the program may have crawled, bleeding, thousands of instructions away from the offending line of code.

It is at this point that the knowledge of memory manger structures can often be put to good use. If you are lucky enough to still have a low level debugger functioning, knowing the structure of the heap can provide vital information in “cracking the case”. Think of it as a software autopsy to determine the cause of your application’s death.

Healthy, Happy Heaps

At the highest level, the heap is made up of three basic components. There is one Zone Header, one Zone Trailer, and in between the two lie any number of blocks. The blocks occupy all the space in the heap and are divided into three categories: free, non-relocatable, and relocatable. This article will focus on the internal structure of these blocks.

Figure 1

With the recent addition of the new Inside Macintosh Memory volume Apple has decided to publish the 32 bit heap structures they left out of IM VI. Note that using these structures in a program guarantees incompatibility when Apple changes the memory manager. Precisely when that will happen is anyone’s guess, but with PowerPC looming on the horizon I’d imagine this information has a useful shelf life of less than two years.

Both 24 bit and 32 bit memory manager blocks have the same general structure. Blocks always begin with a Block Header, followed by any amount of logical data. A variable sized padding field may be tacked onto the end. The size of the entire structure is called the block’s physical size. Applications never deal with this size. The size of the variable length logical data field is called the logical size. It is this logical size that is returned when your application issues a GetHandleSize of GetPtrSize trap.

Figure 2

The blocks are arranged one after another in the heap. There is no empty space separating blocks: every byte in the heap between the zone header and zone trailer “belongs” to one and only one block. What an application thinks of as “free” memory is contained in “free” blocks.

Of the three fields in the Memory Manager Block, only the contents of the Block Header is of interest to us. The padding field exists to keep the blocks long word aligned on 68020, 68030 and 68040 class machines and to enforce a minimum block size of twelve bytes on all machines. The padding field’s contents are unused.

The logical data field’s contents are application specific - you can put whatever you like into Handles and Pointers that you allocate. In a free block, the logical data field will contain whatever garbage is remaining in memory at the time the block is created.

The structure of the block header depends on whether you are running the 24 or 32 bit memory manager. The 24 bit version is 8 bytes long, while the 32 bit version is 12. The 32 bit version has expanded several of the old fields and added one new field to accommodate larger memory models.

Figure 3

Defined in C, these two structures are as follows:

/* 1 */

typedef struct {
 long   blockType:2; // See defines below
 long   unused:2;
 long   sizeCorrection:4; 
 long   physSize:24;
 VariantData     v;
}Block24Header;

typedef struct {
 short  blockType:2;
 short  unused:6;
 short  flagBits:8;
 Byte   unused;
 Byte   sizeCorrection; 
 long   physSize;
 VariantData     v;
}Block32Head;

The first field is the block type. These two bits indicate whether a block is free, nonrelocatable, or relocatable (i.e., unused, a pointer, or a handle). In C, the defines for these values are:

/* 2 */

 #definefreeType 0x00
 #definenonRelocType 0x01
 #definerelocType0x02

The size correction field determines the size of the padding field in the block structure, while the physSize field contains the size of the entire block. To get the logical size of the block, take the physical size and subtract the sizeCorrection and the block header size (8 bytes for the 24 bit memory manager and 12 bytes for the 32 bit memory manager). In the 32 bit memory manager the physSize field has been widened to accommodate larger block sizes.

To allow for full 32 bit addresses, it was also necessary to change the location of the Master Pointer flags. In the 24 bit memory manager, these flags (lock, purge, and resource) occupied the high byte of each master pointer, while the lower 24 bits addressed a relocatable block. In the 32 bit memory manager, all 32 bits of the master pointer are significant in determining the address of a relocatable block. Routines such as HLock have been recoded to look for these flag in the flagBits field of the 32 bit block header. The bit usage within the flagBits field is exactly the same as it was for the high byte of the 24 bit Memory Manager’s master pointers: lock is the most significant bit, followed by purge and then resource. The other bits of the flagBits byte are reserved.

Variant Data

The variant data in either block header is a 32 bit value whose meaning depends on the block type:

/* 3 */

typedef union {
 // relocatable block: Offset from zone to master ptr
 long   relHand;
 // non-relocatable block: address of heap zone
 THz    itsZone;
 // free block
 long   unused;
}VariantData;

For a free block, the information is unused. For a non-relocatable block, this field contains the address of the heap zone that “owns” the block. For a relocatable block, the variant data contains an offset from the start of the heap to this block’s master pointer.

MacsBug Templates

In the examples that follow, I’ll be poking around with MacsBug in a heap while running the 32 bit memory manager. A MacsBug template and macro will come in handy. Open up the Debugger Prefs file (found in the System Folder) using ResEdit and create these mxwt and mxbm resources:

Figure 4

Once these are installed in the Debugger Prefs file, make sure that the Memory control panel is set to 32 bit addressing and reboot to let MacsBug load these resources.

Examining the Heap

After rebooting I enter MacsBug via the Programmer’s switch. The heap dump command will give us a view of the current heap (of course, your addresses will vary):

hd
 Displaying the Application heap at 012F7904
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N
•012F7A50 000026F4+00R  012F7A38 L CODE0003  0BC2
•012FA150 00000024+00R  012F7A0C L acur1964  0BC2
•012FA180 00000044+00R  012F7A08 L CURS1964  0BC2
etc 

The block at 012F7944 is the first block in the heap following the Zone header. Let’s use our new macro on the second block. First we have to display some memory to equate the MacsBug ‘.’ pseudo-register with location 012F7A50.

dm 12F7A50
 Displaying memory from 12f7a50
  012F7A50  0318 0001 4EFA 26BE  0000 0000 0000 6100  

This is the logical data that an application program would see. Now we can invoke our macro to display the preceding 12 bytes with our block header template:

Block32
 Displaying Block32Header at 012F7A44
  012F7A44  blockType          80 
  012F7A45  flagBits           A0 
  012F7A46  unused             00 
  012F7A47  sizeCorrection     00 
  012F7A48  PhysSize           00002700 
  012F7A4C  Variant            00000134 

Note that 012F7A44 is twelve bytes before the start of the logical data. The blockType of $80 is %1000 0000 in binary. Recall that the first two bits designate the block type and that a 0x02 value in this field signifies a relocatable block. Although the next six bits are reserved for future use, I have never seen anything in them other than zeros. As a human convenience, we can interpret our template blockType field as follows: $80 = relocatable, $40 = non-relocatable, $00 = relocatable. So the $80 value agrees with the Tag R designation in the MacsBug display.

The flagBits field contains A0. This is analogous to having a 24 bit Master Pointer’s high byte set to $A0. In other words, the lock bit and resource bits are set ($80+$20). Again, this agrees with the MacsBug display: the Lock column is set and the resource link (CODE 0003 0BC2) establishes this as a resource.

The sizeCorrection field is set to $00, so no size correction was necessary. MacsBug shows this as +00 next to the size field of 000026F4+00. Our physSize field shows 00002700. MacsBug displays the logical size as $26F4. Convert physical size to logical size by subtracting the block header size (a constant 12 decimal bytes) and the size correction (in this case, zero):

$2700- $C - $0 = $26F4

The next block in the heap should be found “physSize” bytes beyond the start of this blocks header:

$012F7A44 + $2700 = 012FA144
dm 12FA144 Block32Header
 Displaying Block32Header at 012FA144
  012FA144  blockType          80 
  012FA145  flagBits           A0 
  012FA146  unused             00 
  012FA147  sizeCorrection     00 
  012FA148  PhysSize           00000030 
  012FA14C  Variant            00000108 

That’s the next block, all right (the logical data starts at 012F7A50, which agrees with the MacsBug hd command output we received earlier).

Back to the block at $012F7A44. The Variant portion of the block header has a value of $00000134. This is the offset from the start of the heap to this block’s master pointer. From the MacsBug hd command we know we are looking at an application heap at 012F7904. Adding $00000134 gives:

$012F7904 + $00000134 = $012F7A38

Is this our master pointer? Examining the contents shows that it does indeed point back to the logical start of our block ($012F7A50):

dl 12F7A38
 Long at 012F7A38 = $012F7A50  #19888720  #19888720   '•/zP'

The master pointer itself can be anywhere within the logical portion of a non-relocatable block. To find the start of the block containing this Master Pointer, we’ll use the MacsBug where command:

wh 012F7A38
Address 012F7A38 is in the Application heap at 012F7904 
 It is 000000F4 bytes into this heap block:
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N

We can use our macro to look at the master pointer block’s header:

Block32
 Displaying Block32Header at 012F7938
  012F7938  blockType          40 
  012F7939  flagBits           00 
  012F793A  unused             00 
  012F793B  sizeCorrection     00 
  012F793C  PhysSize           0000010C 
  012F7940  Variant            012F7904 

Note that this time, the blockType is $40, indicating a non-relocatable block. Again there is no size correction necessary. The Variant portion of the non-relocatable block contains the address of the heap: $012F7904.

Lets examine the zone header at this address in more detail:

dm 012F7904 Zone
 Displaying Zone at 012F7904
  012F7904  bkLim              0133D818 ->  
  012F7908  purgePtr           012F7938 ->  
  012F790C  hFstFree           012FA740 ->  
  012F7910  zcbFree            000080DC 
  012F7914  gzProc             0006F264 ->  
  012F7918  moreMast           0040 
  012F791A  flags              0000 
  012F7922  heapType           01 
  012F792C  purgeProc          0004503C ->  
  012F7930  sparePtr           4080ED12 ->  
  012F7934  allocPtr           01302140 ->  

Block32
 Displaying Block32Header at 012F78F8
  012F78F8  blockType          80 
  012F78F9  flagBits           80 
  012F78FA  unused             00 
  012F78FB  sizeCorrection     00 
  012F78FC  PhysSize           00055C0C 
  012F7900  Variant            01244508 

Hmm this heap zone is itself a locked relocatable block. MultiFinder (or in this case, System 7) allocates application heaps within its own heap as locked relocatable blocks. Where is the MultiFinder heap? A where command on the physical block location will let us know:

wh 012F78F8
 Address 012F78F8 is in the heap at 0010B5EC 
 It is FFFFFFF4 bytes into this heap block (in the block header):
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  •     012F7904
 00055C00+00R  0134FAF4 L

The MultiFinder heap is at address $0010B5EC (ignore that FFFFFFF4 stuff, MacsBug is reporting the negative blocksize offset from the start of our original heap). You can easily see the MultiFinder heap in the MacsBug hz command - all heaps contained within it are indented beneath it.

Conclusion

It is easy to see why some “invalid heap” errors can be so hard to detect. Suppose your program is writing data to a handle and mistakenly oversteps the handle size by one byte. If the block in question has padding bytes, whose value is meaningless, no symptoms will occur. However, if your handle was allocated as a multiple of four (and is larger than four bytes), this wayfaring byte will overwrite the blockType tag byte of the following block! Will you be alerted? Not necessarily: since just two bits of the tag are significant only a value of %11 in the top two bits of the byte will cause the next block to be invalid. Unfortunately, a value of %00 could “free” the next block, causing havoc later when a heap compaction moves other blocks into this “free” space.

We can use our knowledge of block structures to help in debugging. A locked block that is somehow being unlocked can be detected through use of a checksum. Knowing where the lock bit is stored, you can place a checksum on that byte (in MacsBug, use the step spy ‘ss’ command), causing the debugger to break on the offending line of code.

Future Tools

Last year Devon Hubbard and I wrote a tool called HeapQC that isolated many of the problems that can crop up in a Macintosh heap. This tool relied on very fast heap scrambling to catch “dangling pointer” problems, purge routines to find mismanagement of purgeable blocks, free memory invalidation to uncover “wild” pointers, and sophisticated heap checking that verified all of the linkages described in this article.

We grew disenchanted with the company that distributed the product (read: we were not paid) and decided to create a new company, Onyx Technology to pursue contract work. Before long, we began working on a stress testing tool designed to be order of magnitude more thorough and convenient than anything else on the market.

This new tool, QCPro, is being written from scratch to answer our own in-house needs for quality assurance on the contract jobs we undertake. The plan calls for MMU protection, trap discipline, leak detection, and variable frequency heap examinations (ranging from validating the heap after every instruction to validating it after every trap) as well as a host of more esoteric checks. As developers, we feel the Mac has needed more powerful error detection for some time.

Anyone who has ever chased down a memory bug for the better part of a day (week?) will find this tool invaluable. We are already using it to test itself. If you’d like more information or have suggestions, please don’t hesitate to drop me a line at AppleLink: D2238, America Online: B.Bell5, or CompuServe: 70550,137. We have gotten quite a few good ideas from people online and will try to incorporate as many of them as possible into the final product.