TweetFollow Us on Twitter

Playing with Blocks
Volume Number:8
Issue Number:7
Column Tag:Debugging

Related Info: Memory Manager

Playing with Blocks

Memory manager structures - a software autopsy of your application's death!

By Brooks Bell, Bradenton, Florida

Note: Source code files accompanying article are located on MacTech CD-ROM or source code disks.

Let’s set the ‘Way Back’ machine for 1985. Programmers of the day complained heavily of the information that Apple was keeping to itself. Inside Macintosh just did not provide enough details.

The chapter on the Memory Manager, for example, was fuzzy on the meaning of certain fields in the Zone header. SparePtr, allocPtr even the ever-popular flags field were defined only as ‘used internally’. Even worse, fields such as cntRel, maxRel, and cntHandles were described as ‘not used’, when any debugger showed that they clearly contained valid data.

Of course, we have all by now learned that these things are hidden from programmers because they are fair game for change. What works for a 512K “Fat” Mac isn’t necessarily applicable to a Quadra with 64MB of RAM. We have grown accustomed to the trade off: Apple shields us from some OS details in exchange for blessed future hardware compatibility.

That is why it is something of a surprise to turn a few pages further in IM II and discover a very detailed description of the old 24 bit memory manager block structure. In retrospect, this knowledge led directly to the problems of 32 bit uncleanliness. Here is how to set the Master Pointer ‘lock’ bit and bypass the HLock trap overhead. Lo3Bytes is described - who needs StripAddress? Everything that you need to make a 32 bit dirty application is right here in black and white.

With the 32 bit memory manager and Inside Macintosh VI, having learned from their mistakes, Apple decided not to publish the Memory Manager block structures. Well written applications will use HGetState, HLock, etc There is no need to risk compatibility by accessing these structures directly.

However, when you are in the middle of debugging some code errors can crop up that cause an invalid heap. Because the Mac ROMs do not check parameters for errors, it is easy to pass a trap a bad value and have it write garbage all over the heap. It is also easy to move data into a disposed handle, store data beyond the bounds of an array, maybe even call BlockMove directly and blast data all over creation. Lacking sophisticated memory protection, the Macintosh heap is easily destroyed.

Perhaps the worst aspect of such damage is that fatal errors may not appear for some time. When they do show up in the form of the friendly ‘illegal instruction’, ‘Stack has collided with the heap’, ‘Bus Error’ or whatever the program may have crawled, bleeding, thousands of instructions away from the offending line of code.

It is at this point that the knowledge of memory manger structures can often be put to good use. If you are lucky enough to still have a low level debugger functioning, knowing the structure of the heap can provide vital information in “cracking the case”. Think of it as a software autopsy to determine the cause of your application’s death.

Healthy, Happy Heaps

At the highest level, the heap is made up of three basic components. There is one Zone Header, one Zone Trailer, and in between the two lie any number of blocks. The blocks occupy all the space in the heap and are divided into three categories: free, non-relocatable, and relocatable. This article will focus on the internal structure of these blocks.

Figure 1

With the recent addition of the new Inside Macintosh Memory volume Apple has decided to publish the 32 bit heap structures they left out of IM VI. Note that using these structures in a program guarantees incompatibility when Apple changes the memory manager. Precisely when that will happen is anyone’s guess, but with PowerPC looming on the horizon I’d imagine this information has a useful shelf life of less than two years.

Both 24 bit and 32 bit memory manager blocks have the same general structure. Blocks always begin with a Block Header, followed by any amount of logical data. A variable sized padding field may be tacked onto the end. The size of the entire structure is called the block’s physical size. Applications never deal with this size. The size of the variable length logical data field is called the logical size. It is this logical size that is returned when your application issues a GetHandleSize of GetPtrSize trap.

Figure 2

The blocks are arranged one after another in the heap. There is no empty space separating blocks: every byte in the heap between the zone header and zone trailer “belongs” to one and only one block. What an application thinks of as “free” memory is contained in “free” blocks.

Of the three fields in the Memory Manager Block, only the contents of the Block Header is of interest to us. The padding field exists to keep the blocks long word aligned on 68020, 68030 and 68040 class machines and to enforce a minimum block size of twelve bytes on all machines. The padding field’s contents are unused.

The logical data field’s contents are application specific - you can put whatever you like into Handles and Pointers that you allocate. In a free block, the logical data field will contain whatever garbage is remaining in memory at the time the block is created.

The structure of the block header depends on whether you are running the 24 or 32 bit memory manager. The 24 bit version is 8 bytes long, while the 32 bit version is 12. The 32 bit version has expanded several of the old fields and added one new field to accommodate larger memory models.

Figure 3

Defined in C, these two structures are as follows:

/* 1 */

typedef struct {
 long   blockType:2; // See defines below
 long   unused:2;
 long   sizeCorrection:4; 
 long   physSize:24;
 VariantData     v;
}Block24Header;

typedef struct {
 short  blockType:2;
 short  unused:6;
 short  flagBits:8;
 Byte   unused;
 Byte   sizeCorrection; 
 long   physSize;
 VariantData     v;
}Block32Head;

The first field is the block type. These two bits indicate whether a block is free, nonrelocatable, or relocatable (i.e., unused, a pointer, or a handle). In C, the defines for these values are:

/* 2 */

 #definefreeType 0x00
 #definenonRelocType 0x01
 #definerelocType0x02

The size correction field determines the size of the padding field in the block structure, while the physSize field contains the size of the entire block. To get the logical size of the block, take the physical size and subtract the sizeCorrection and the block header size (8 bytes for the 24 bit memory manager and 12 bytes for the 32 bit memory manager). In the 32 bit memory manager the physSize field has been widened to accommodate larger block sizes.

To allow for full 32 bit addresses, it was also necessary to change the location of the Master Pointer flags. In the 24 bit memory manager, these flags (lock, purge, and resource) occupied the high byte of each master pointer, while the lower 24 bits addressed a relocatable block. In the 32 bit memory manager, all 32 bits of the master pointer are significant in determining the address of a relocatable block. Routines such as HLock have been recoded to look for these flag in the flagBits field of the 32 bit block header. The bit usage within the flagBits field is exactly the same as it was for the high byte of the 24 bit Memory Manager’s master pointers: lock is the most significant bit, followed by purge and then resource. The other bits of the flagBits byte are reserved.

Variant Data

The variant data in either block header is a 32 bit value whose meaning depends on the block type:

/* 3 */

typedef union {
 // relocatable block: Offset from zone to master ptr
 long   relHand;
 // non-relocatable block: address of heap zone
 THz    itsZone;
 // free block
 long   unused;
}VariantData;

For a free block, the information is unused. For a non-relocatable block, this field contains the address of the heap zone that “owns” the block. For a relocatable block, the variant data contains an offset from the start of the heap to this block’s master pointer.

MacsBug Templates

In the examples that follow, I’ll be poking around with MacsBug in a heap while running the 32 bit memory manager. A MacsBug template and macro will come in handy. Open up the Debugger Prefs file (found in the System Folder) using ResEdit and create these mxwt and mxbm resources:

Figure 4

Once these are installed in the Debugger Prefs file, make sure that the Memory control panel is set to 32 bit addressing and reboot to let MacsBug load these resources.

Examining the Heap

After rebooting I enter MacsBug via the Programmer’s switch. The heap dump command will give us a view of the current heap (of course, your addresses will vary):

hd
 Displaying the Application heap at 012F7904
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N
•012F7A50 000026F4+00R  012F7A38 L CODE0003  0BC2
•012FA150 00000024+00R  012F7A0C L acur1964  0BC2
•012FA180 00000044+00R  012F7A08 L CURS1964  0BC2
etc 

The block at 012F7944 is the first block in the heap following the Zone header. Let’s use our new macro on the second block. First we have to display some memory to equate the MacsBug ‘.’ pseudo-register with location 012F7A50.

dm 12F7A50
 Displaying memory from 12f7a50
  012F7A50  0318 0001 4EFA 26BE  0000 0000 0000 6100  

This is the logical data that an application program would see. Now we can invoke our macro to display the preceding 12 bytes with our block header template:

Block32
 Displaying Block32Header at 012F7A44
  012F7A44  blockType          80 
  012F7A45  flagBits           A0 
  012F7A46  unused             00 
  012F7A47  sizeCorrection     00 
  012F7A48  PhysSize           00002700 
  012F7A4C  Variant            00000134 

Note that 012F7A44 is twelve bytes before the start of the logical data. The blockType of $80 is %1000 0000 in binary. Recall that the first two bits designate the block type and that a 0x02 value in this field signifies a relocatable block. Although the next six bits are reserved for future use, I have never seen anything in them other than zeros. As a human convenience, we can interpret our template blockType field as follows: $80 = relocatable, $40 = non-relocatable, $00 = relocatable. So the $80 value agrees with the Tag R designation in the MacsBug display.

The flagBits field contains A0. This is analogous to having a 24 bit Master Pointer’s high byte set to $A0. In other words, the lock bit and resource bits are set ($80+$20). Again, this agrees with the MacsBug display: the Lock column is set and the resource link (CODE 0003 0BC2) establishes this as a resource.

The sizeCorrection field is set to $00, so no size correction was necessary. MacsBug shows this as +00 next to the size field of 000026F4+00. Our physSize field shows 00002700. MacsBug displays the logical size as $26F4. Convert physical size to logical size by subtracting the block header size (a constant 12 decimal bytes) and the size correction (in this case, zero):

$2700- $C - $0 = $26F4

The next block in the heap should be found “physSize” bytes beyond the start of this blocks header:

$012F7A44 + $2700 = 012FA144
dm 12FA144 Block32Header
 Displaying Block32Header at 012FA144
  012FA144  blockType          80 
  012FA145  flagBits           A0 
  012FA146  unused             00 
  012FA147  sizeCorrection     00 
  012FA148  PhysSize           00000030 
  012FA14C  Variant            00000108 

That’s the next block, all right (the logical data starts at 012F7A50, which agrees with the MacsBug hd command output we received earlier).

Back to the block at $012F7A44. The Variant portion of the block header has a value of $00000134. This is the offset from the start of the heap to this block’s master pointer. From the MacsBug hd command we know we are looking at an application heap at 012F7904. Adding $00000134 gives:

$012F7904 + $00000134 = $012F7A38

Is this our master pointer? Examining the contents shows that it does indeed point back to the logical start of our block ($012F7A50):

dl 12F7A38
 Long at 012F7A38 = $012F7A50  #19888720  #19888720   '•/zP'

The master pointer itself can be anywhere within the logical portion of a non-relocatable block. To find the start of the block containing this Master Pointer, we’ll use the MacsBug where command:

wh 012F7A38
Address 012F7A38 is in the Application heap at 012F7904 
 It is 000000F4 bytes into this heap block:
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N

We can use our macro to look at the master pointer block’s header:

Block32
 Displaying Block32Header at 012F7938
  012F7938  blockType          40 
  012F7939  flagBits           00 
  012F793A  unused             00 
  012F793B  sizeCorrection     00 
  012F793C  PhysSize           0000010C 
  012F7940  Variant            012F7904 

Note that this time, the blockType is $40, indicating a non-relocatable block. Again there is no size correction necessary. The Variant portion of the non-relocatable block contains the address of the heap: $012F7904.

Lets examine the zone header at this address in more detail:

dm 012F7904 Zone
 Displaying Zone at 012F7904
  012F7904  bkLim              0133D818 ->  
  012F7908  purgePtr           012F7938 ->  
  012F790C  hFstFree           012FA740 ->  
  012F7910  zcbFree            000080DC 
  012F7914  gzProc             0006F264 ->  
  012F7918  moreMast           0040 
  012F791A  flags              0000 
  012F7922  heapType           01 
  012F792C  purgeProc          0004503C ->  
  012F7930  sparePtr           4080ED12 ->  
  012F7934  allocPtr           01302140 ->  

Block32
 Displaying Block32Header at 012F78F8
  012F78F8  blockType          80 
  012F78F9  flagBits           80 
  012F78FA  unused             00 
  012F78FB  sizeCorrection     00 
  012F78FC  PhysSize           00055C0C 
  012F7900  Variant            01244508 

Hmm this heap zone is itself a locked relocatable block. MultiFinder (or in this case, System 7) allocates application heaps within its own heap as locked relocatable blocks. Where is the MultiFinder heap? A where command on the physical block location will let us know:

wh 012F78F8
 Address 012F78F8 is in the heap at 0010B5EC 
 It is FFFFFFF4 bytes into this heap block (in the block header):
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  •     012F7904
 00055C00+00R  0134FAF4 L

The MultiFinder heap is at address $0010B5EC (ignore that FFFFFFF4 stuff, MacsBug is reporting the negative blocksize offset from the start of our original heap). You can easily see the MultiFinder heap in the MacsBug hz command - all heaps contained within it are indented beneath it.

Conclusion

It is easy to see why some “invalid heap” errors can be so hard to detect. Suppose your program is writing data to a handle and mistakenly oversteps the handle size by one byte. If the block in question has padding bytes, whose value is meaningless, no symptoms will occur. However, if your handle was allocated as a multiple of four (and is larger than four bytes), this wayfaring byte will overwrite the blockType tag byte of the following block! Will you be alerted? Not necessarily: since just two bits of the tag are significant only a value of %11 in the top two bits of the byte will cause the next block to be invalid. Unfortunately, a value of %00 could “free” the next block, causing havoc later when a heap compaction moves other blocks into this “free” space.

We can use our knowledge of block structures to help in debugging. A locked block that is somehow being unlocked can be detected through use of a checksum. Knowing where the lock bit is stored, you can place a checksum on that byte (in MacsBug, use the step spy ‘ss’ command), causing the debugger to break on the offending line of code.

Future Tools

Last year Devon Hubbard and I wrote a tool called HeapQC that isolated many of the problems that can crop up in a Macintosh heap. This tool relied on very fast heap scrambling to catch “dangling pointer” problems, purge routines to find mismanagement of purgeable blocks, free memory invalidation to uncover “wild” pointers, and sophisticated heap checking that verified all of the linkages described in this article.

We grew disenchanted with the company that distributed the product (read: we were not paid) and decided to create a new company, Onyx Technology to pursue contract work. Before long, we began working on a stress testing tool designed to be order of magnitude more thorough and convenient than anything else on the market.

This new tool, QCPro, is being written from scratch to answer our own in-house needs for quality assurance on the contract jobs we undertake. The plan calls for MMU protection, trap discipline, leak detection, and variable frequency heap examinations (ranging from validating the heap after every instruction to validating it after every trap) as well as a host of more esoteric checks. As developers, we feel the Mac has needed more powerful error detection for some time.

Anyone who has ever chased down a memory bug for the better part of a day (week?) will find this tool invaluable. We are already using it to test itself. If you’d like more information or have suggestions, please don’t hesitate to drop me a line at AppleLink: D2238, America Online: B.Bell5, or CompuServe: 70550,137. We have gotten quite a few good ideas from people online and will try to incorporate as many of them as possible into the final product.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

BBEdit 11.6.1 - Powerful text and HTML e...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more
OmniGraffle Pro 6.6.1 - Create diagrams,...
OmniGraffle Pro helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use... Read more
OmniGraffle 6.6.1 - Create diagrams, flo...
OmniGraffle helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use Graffle to... Read more
Dropbox 8.4.21 - Cloud backup and synchr...
Dropbox is an application that creates a special Finder folder that automatically syncs online and between your computers. It allows you to both backup files and keep them up-to-date between systems... Read more
BetterTouchTool 1.84 - Customize Multi-T...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
ScreenFlow 6.1 - Create screen recording...
ScreenFlow is powerful, easy-to-use screencasting software for the Mac. With ScreenFlow you can record the contents of your entire monitor while also capturing your video camera, microphone and your... Read more
f.lux 37.7 - Adjusts the color of your d...
f.lux makes the color of your computer's display adapt to the time of day, warm at night and like sunlight during the day. Ever notice how people texting at night have that eerie blue glow? Or wake... Read more
BetterTouchTool 1.84 - Customize Multi-T...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
Dropbox 8.4.21 - Cloud backup and synchr...
Dropbox is an application that creates a special Finder folder that automatically syncs online and between your computers. It allows you to both backup files and keep them up-to-date between systems... Read more
OmniGraffle Pro 6.6.1 - Create diagrams,...
OmniGraffle Pro helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use... Read more

3 best fantasy football apps to get you...
Last season didn't go the way you wanted it to in fantasy football. You were super happy following your drafts or auctions, convinced you had outsmarted everyone. You were all set to hustle on the waiver wire, work out some sweet trades, and make... | Read more »
Pokemon GO update: Take me to your leade...
The Team Leaders in Pokemon GO have had it pretty easy up until now. They show up when players reach level 5, make their cases for joining their respective teams, and that's pretty much it. Light work, as Floyd Mayweather might say. [Read more] | Read more »
Ruismaker FM (Music)
Ruismaker FM 1.0 Device: iOS Universal Category: Music Price: $4.99, Version: 1.0 (iTunes) Description: Following up on the success of Ruismaker, here's her crazy twin-sister, designed for people who want to design their own... | Read more »
Space Marshals 2 (Games)
Space Marshals 2 1.0.15 Device: iOS iPhone Category: Games Price: $5.99, Version: 1.0.15 (iTunes) Description: The sci-fi wild west adventure in outer space continues with Space Marshals 2. This tactical top-down shooter puts you in... | Read more »
Dungeon Warfare (Games)
Dungeon Warfare 1.0 Device: iOS Universal Category: Games Price: $3.99, Version: 1.0 (iTunes) Description: Dungeon Warfare is a challenging tower defense game where you become a dungeon lord to defend your dungeon against greedy... | Read more »
Solitairica (Games)
Solitairica 1.0.7 Device: iOS Universal Category: Games Price: $3.99, Version: 1.0.7 (iTunes) Description: Solitairica takes RPG combat and challenging rogue-like progression to a fresh new place—the world of solitaire! | Read more »
Bowmasters tips, tricks and hints
At least for this writer, archery was one of the more pleasant surprises of the 2016 Rio Olympics. As opposed to target shooting with guns, which was dreadfully boring, watching people shoot arrows at targets was pretty darn cool. [Read more] | Read more »
Best apps for watching live TV
The Olympics have come and gone, leaving nearly everyone in a temporary state of "What the heck am I going to watch on TV right now?" Besides old reruns of Golden Girls, but that goes without saying. [Read more] | Read more »
What is Flip Diving, and why has it take...
Move over Pokemon GO. There's a new king in town, and it's "the world's #1 cliff diving game." [Read more] | Read more »
5 places where Pokemon GO is still numbe...
In the U.S., the bloom is off the Pokemon Go rose ever so slightly. It's still doing great, sitting atop the top grossing chart as it has for some time, but it's no longer among the top 10 free apps in downloads, possibly because darn near... | Read more »

Price Scanner via MacPrices.net

Apple refurbished 13-inch MacBook Airs availa...
Apple has Certified Refurbished 2016 and 2015 13″ MacBook Airs now available starting at $849. An Apple one-year warranty is included with each MacBook, and shipping is free: - 2016 13″ 1.6GHz/8GB/... Read more
Apple refurbished iPad mini 2s available for...
Apple is offering Certified Refurbished iPad mini 2s for up to $80 off the cost of new minis. An Apple one-year warranty is included with each model, and shipping is free: - 16GB iPad mini 2 WiFi: $... Read more
Save up to $600 with Apple refurbished Mac Pr...
Apple has Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The following... Read more
Mac Pros on sale for $200 off MSRP
B&H Photo has Mac Pros on sale for $200 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3799, $200... Read more
Will We See A 10.5″ iPad Pro in 2017? – The ‘...
A MacRumors report, cites a research note from KGI Securities analyst Ming-Chi Kuo, saying a new size iPad model is in the works. According to the highly respected Cho, who has a strong track record... Read more
IOGEAR USB-C Docking Station Transforms Lapto...
IOGEAR has announced the launch of its innovative USB-C Docking Station with Power Delivery which turns USB-C enabled laptops into desktop workstations. The new IOGEAR USB-C Docking Station features... Read more
12-inch Retina MacBooks on sale for up to $10...
Amazon has 2016 12″ Apple Retina MacBooks on sale for $100 off MSRP. Shipping is free: - 12″ 1.1GHz Space Gray Retina MacBook: $1199 $100 off MSRP - 12″ 1.1GHz Silver Retina MacBook: $1224.99 $75 off... Read more
13-inch 2.5GHz MacBook Pro (Apple refurbished...
Apple has Certified Refurbished 13″ 2.5GHz MacBook Pros available for $829, or $270 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 13″ 2.5GHz MacBook Pros... Read more
21-inch iMacs on sale for up to $120 off MSRP
B&H Photo has 21″ iMacs on sale for up to $120 off MSRP including free shipping plus NY sales tax only: - 21″ 3.1GHz iMac 4K: $1379 $120 off MSRP - 21″ 2.8GHz iMac: $1199.99 $100 off MSRP - 21″ 1... Read more
Typinator 6.10 comes with 50 improvements – G...
Ergonis Software today announced release of Typinator 6.10, a new version of their text expander utility for macOS. Typinator 6.10 comes with 50 improvements, including new features, compatibility... Read more

Jobs Board

*Apple* Mobile Master - Best Buy (United Sta...
What does a Best Buy Apple Mobile Master do? At Best Buy, our mission is to leverage the unique talents and passions of our employees to inspire, delight, and enrich Read more
*Apple* Retail - Multiple Positions Akron, O...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Simply Mac *Apple* Specialist- Repair Techn...
…The Technician is a master at working with our customers to diagnose and repair Apple devices in a manner that exceeds the expectations set forth by Apple Read more
*Apple* Retail - Multiple Positions Germanto...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Professional Learning Specialist - A...
# Apple Professional Learning Specialist Job Number: 51234379 Portland, Maine, Maine, United States Posted: Aug. 18, 2016 Weekly Hours: 40.00 **Job Summary** The Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.