TweetFollow Us on Twitter

Playing with Blocks
Volume Number:8
Issue Number:7
Column Tag:Debugging

Related Info: Memory Manager

Playing with Blocks

Memory manager structures - a software autopsy of your application's death!

By Brooks Bell, Bradenton, Florida

Note: Source code files accompanying article are located on MacTech CD-ROM or source code disks.

Let’s set the ‘Way Back’ machine for 1985. Programmers of the day complained heavily of the information that Apple was keeping to itself. Inside Macintosh just did not provide enough details.

The chapter on the Memory Manager, for example, was fuzzy on the meaning of certain fields in the Zone header. SparePtr, allocPtr even the ever-popular flags field were defined only as ‘used internally’. Even worse, fields such as cntRel, maxRel, and cntHandles were described as ‘not used’, when any debugger showed that they clearly contained valid data.

Of course, we have all by now learned that these things are hidden from programmers because they are fair game for change. What works for a 512K “Fat” Mac isn’t necessarily applicable to a Quadra with 64MB of RAM. We have grown accustomed to the trade off: Apple shields us from some OS details in exchange for blessed future hardware compatibility.

That is why it is something of a surprise to turn a few pages further in IM II and discover a very detailed description of the old 24 bit memory manager block structure. In retrospect, this knowledge led directly to the problems of 32 bit uncleanliness. Here is how to set the Master Pointer ‘lock’ bit and bypass the HLock trap overhead. Lo3Bytes is described - who needs StripAddress? Everything that you need to make a 32 bit dirty application is right here in black and white.

With the 32 bit memory manager and Inside Macintosh VI, having learned from their mistakes, Apple decided not to publish the Memory Manager block structures. Well written applications will use HGetState, HLock, etc There is no need to risk compatibility by accessing these structures directly.

However, when you are in the middle of debugging some code errors can crop up that cause an invalid heap. Because the Mac ROMs do not check parameters for errors, it is easy to pass a trap a bad value and have it write garbage all over the heap. It is also easy to move data into a disposed handle, store data beyond the bounds of an array, maybe even call BlockMove directly and blast data all over creation. Lacking sophisticated memory protection, the Macintosh heap is easily destroyed.

Perhaps the worst aspect of such damage is that fatal errors may not appear for some time. When they do show up in the form of the friendly ‘illegal instruction’, ‘Stack has collided with the heap’, ‘Bus Error’ or whatever the program may have crawled, bleeding, thousands of instructions away from the offending line of code.

It is at this point that the knowledge of memory manger structures can often be put to good use. If you are lucky enough to still have a low level debugger functioning, knowing the structure of the heap can provide vital information in “cracking the case”. Think of it as a software autopsy to determine the cause of your application’s death.

Healthy, Happy Heaps

At the highest level, the heap is made up of three basic components. There is one Zone Header, one Zone Trailer, and in between the two lie any number of blocks. The blocks occupy all the space in the heap and are divided into three categories: free, non-relocatable, and relocatable. This article will focus on the internal structure of these blocks.

Figure 1

With the recent addition of the new Inside Macintosh Memory volume Apple has decided to publish the 32 bit heap structures they left out of IM VI. Note that using these structures in a program guarantees incompatibility when Apple changes the memory manager. Precisely when that will happen is anyone’s guess, but with PowerPC looming on the horizon I’d imagine this information has a useful shelf life of less than two years.

Both 24 bit and 32 bit memory manager blocks have the same general structure. Blocks always begin with a Block Header, followed by any amount of logical data. A variable sized padding field may be tacked onto the end. The size of the entire structure is called the block’s physical size. Applications never deal with this size. The size of the variable length logical data field is called the logical size. It is this logical size that is returned when your application issues a GetHandleSize of GetPtrSize trap.

Figure 2

The blocks are arranged one after another in the heap. There is no empty space separating blocks: every byte in the heap between the zone header and zone trailer “belongs” to one and only one block. What an application thinks of as “free” memory is contained in “free” blocks.

Of the three fields in the Memory Manager Block, only the contents of the Block Header is of interest to us. The padding field exists to keep the blocks long word aligned on 68020, 68030 and 68040 class machines and to enforce a minimum block size of twelve bytes on all machines. The padding field’s contents are unused.

The logical data field’s contents are application specific - you can put whatever you like into Handles and Pointers that you allocate. In a free block, the logical data field will contain whatever garbage is remaining in memory at the time the block is created.

The structure of the block header depends on whether you are running the 24 or 32 bit memory manager. The 24 bit version is 8 bytes long, while the 32 bit version is 12. The 32 bit version has expanded several of the old fields and added one new field to accommodate larger memory models.

Figure 3

Defined in C, these two structures are as follows:

/* 1 */

typedef struct {
 long   blockType:2; // See defines below
 long   unused:2;
 long   sizeCorrection:4; 
 long   physSize:24;
 VariantData     v;
}Block24Header;

typedef struct {
 short  blockType:2;
 short  unused:6;
 short  flagBits:8;
 Byte   unused;
 Byte   sizeCorrection; 
 long   physSize;
 VariantData     v;
}Block32Head;

The first field is the block type. These two bits indicate whether a block is free, nonrelocatable, or relocatable (i.e., unused, a pointer, or a handle). In C, the defines for these values are:

/* 2 */

 #definefreeType 0x00
 #definenonRelocType 0x01
 #definerelocType0x02

The size correction field determines the size of the padding field in the block structure, while the physSize field contains the size of the entire block. To get the logical size of the block, take the physical size and subtract the sizeCorrection and the block header size (8 bytes for the 24 bit memory manager and 12 bytes for the 32 bit memory manager). In the 32 bit memory manager the physSize field has been widened to accommodate larger block sizes.

To allow for full 32 bit addresses, it was also necessary to change the location of the Master Pointer flags. In the 24 bit memory manager, these flags (lock, purge, and resource) occupied the high byte of each master pointer, while the lower 24 bits addressed a relocatable block. In the 32 bit memory manager, all 32 bits of the master pointer are significant in determining the address of a relocatable block. Routines such as HLock have been recoded to look for these flag in the flagBits field of the 32 bit block header. The bit usage within the flagBits field is exactly the same as it was for the high byte of the 24 bit Memory Manager’s master pointers: lock is the most significant bit, followed by purge and then resource. The other bits of the flagBits byte are reserved.

Variant Data

The variant data in either block header is a 32 bit value whose meaning depends on the block type:

/* 3 */

typedef union {
 // relocatable block: Offset from zone to master ptr
 long   relHand;
 // non-relocatable block: address of heap zone
 THz    itsZone;
 // free block
 long   unused;
}VariantData;

For a free block, the information is unused. For a non-relocatable block, this field contains the address of the heap zone that “owns” the block. For a relocatable block, the variant data contains an offset from the start of the heap to this block’s master pointer.

MacsBug Templates

In the examples that follow, I’ll be poking around with MacsBug in a heap while running the 32 bit memory manager. A MacsBug template and macro will come in handy. Open up the Debugger Prefs file (found in the System Folder) using ResEdit and create these mxwt and mxbm resources:

Figure 4

Once these are installed in the Debugger Prefs file, make sure that the Memory control panel is set to 32 bit addressing and reboot to let MacsBug load these resources.

Examining the Heap

After rebooting I enter MacsBug via the Programmer’s switch. The heap dump command will give us a view of the current heap (of course, your addresses will vary):

hd
 Displaying the Application heap at 012F7904
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N
•012F7A50 000026F4+00R  012F7A38 L CODE0003  0BC2
•012FA150 00000024+00R  012F7A0C L acur1964  0BC2
•012FA180 00000044+00R  012F7A08 L CURS1964  0BC2
etc 

The block at 012F7944 is the first block in the heap following the Zone header. Let’s use our new macro on the second block. First we have to display some memory to equate the MacsBug ‘.’ pseudo-register with location 012F7A50.

dm 12F7A50
 Displaying memory from 12f7a50
  012F7A50  0318 0001 4EFA 26BE  0000 0000 0000 6100  

This is the logical data that an application program would see. Now we can invoke our macro to display the preceding 12 bytes with our block header template:

Block32
 Displaying Block32Header at 012F7A44
  012F7A44  blockType          80 
  012F7A45  flagBits           A0 
  012F7A46  unused             00 
  012F7A47  sizeCorrection     00 
  012F7A48  PhysSize           00002700 
  012F7A4C  Variant            00000134 

Note that 012F7A44 is twelve bytes before the start of the logical data. The blockType of $80 is %1000 0000 in binary. Recall that the first two bits designate the block type and that a 0x02 value in this field signifies a relocatable block. Although the next six bits are reserved for future use, I have never seen anything in them other than zeros. As a human convenience, we can interpret our template blockType field as follows: $80 = relocatable, $40 = non-relocatable, $00 = relocatable. So the $80 value agrees with the Tag R designation in the MacsBug display.

The flagBits field contains A0. This is analogous to having a 24 bit Master Pointer’s high byte set to $A0. In other words, the lock bit and resource bits are set ($80+$20). Again, this agrees with the MacsBug display: the Lock column is set and the resource link (CODE 0003 0BC2) establishes this as a resource.

The sizeCorrection field is set to $00, so no size correction was necessary. MacsBug shows this as +00 next to the size field of 000026F4+00. Our physSize field shows 00002700. MacsBug displays the logical size as $26F4. Convert physical size to logical size by subtracting the block header size (a constant 12 decimal bytes) and the size correction (in this case, zero):

$2700- $C - $0 = $26F4

The next block in the heap should be found “physSize” bytes beyond the start of this blocks header:

$012F7A44 + $2700 = 012FA144
dm 12FA144 Block32Header
 Displaying Block32Header at 012FA144
  012FA144  blockType          80 
  012FA145  flagBits           A0 
  012FA146  unused             00 
  012FA147  sizeCorrection     00 
  012FA148  PhysSize           00000030 
  012FA14C  Variant            00000108 

That’s the next block, all right (the logical data starts at 012F7A50, which agrees with the MacsBug hd command output we received earlier).

Back to the block at $012F7A44. The Variant portion of the block header has a value of $00000134. This is the offset from the start of the heap to this block’s master pointer. From the MacsBug hd command we know we are looking at an application heap at 012F7904. Adding $00000134 gives:

$012F7904 + $00000134 = $012F7A38

Is this our master pointer? Examining the contents shows that it does indeed point back to the logical start of our block ($012F7A50):

dl 12F7A38
 Long at 012F7A38 = $012F7A50  #19888720  #19888720   '•/zP'

The master pointer itself can be anywhere within the logical portion of a non-relocatable block. To find the start of the block containing this Master Pointer, we’ll use the MacsBug where command:

wh 012F7A38
Address 012F7A38 is in the Application heap at 012F7904 
 It is 000000F4 bytes into this heap block:
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N

We can use our macro to look at the master pointer block’s header:

Block32
 Displaying Block32Header at 012F7938
  012F7938  blockType          40 
  012F7939  flagBits           00 
  012F793A  unused             00 
  012F793B  sizeCorrection     00 
  012F793C  PhysSize           0000010C 
  012F7940  Variant            012F7904 

Note that this time, the blockType is $40, indicating a non-relocatable block. Again there is no size correction necessary. The Variant portion of the non-relocatable block contains the address of the heap: $012F7904.

Lets examine the zone header at this address in more detail:

dm 012F7904 Zone
 Displaying Zone at 012F7904
  012F7904  bkLim              0133D818 ->  
  012F7908  purgePtr           012F7938 ->  
  012F790C  hFstFree           012FA740 ->  
  012F7910  zcbFree            000080DC 
  012F7914  gzProc             0006F264 ->  
  012F7918  moreMast           0040 
  012F791A  flags              0000 
  012F7922  heapType           01 
  012F792C  purgeProc          0004503C ->  
  012F7930  sparePtr           4080ED12 ->  
  012F7934  allocPtr           01302140 ->  

Block32
 Displaying Block32Header at 012F78F8
  012F78F8  blockType          80 
  012F78F9  flagBits           80 
  012F78FA  unused             00 
  012F78FB  sizeCorrection     00 
  012F78FC  PhysSize           00055C0C 
  012F7900  Variant            01244508 

Hmm this heap zone is itself a locked relocatable block. MultiFinder (or in this case, System 7) allocates application heaps within its own heap as locked relocatable blocks. Where is the MultiFinder heap? A where command on the physical block location will let us know:

wh 012F78F8
 Address 012F78F8 is in the heap at 0010B5EC 
 It is FFFFFFF4 bytes into this heap block (in the block header):
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  •     012F7904
 00055C00+00R  0134FAF4 L

The MultiFinder heap is at address $0010B5EC (ignore that FFFFFFF4 stuff, MacsBug is reporting the negative blocksize offset from the start of our original heap). You can easily see the MultiFinder heap in the MacsBug hz command - all heaps contained within it are indented beneath it.

Conclusion

It is easy to see why some “invalid heap” errors can be so hard to detect. Suppose your program is writing data to a handle and mistakenly oversteps the handle size by one byte. If the block in question has padding bytes, whose value is meaningless, no symptoms will occur. However, if your handle was allocated as a multiple of four (and is larger than four bytes), this wayfaring byte will overwrite the blockType tag byte of the following block! Will you be alerted? Not necessarily: since just two bits of the tag are significant only a value of %11 in the top two bits of the byte will cause the next block to be invalid. Unfortunately, a value of %00 could “free” the next block, causing havoc later when a heap compaction moves other blocks into this “free” space.

We can use our knowledge of block structures to help in debugging. A locked block that is somehow being unlocked can be detected through use of a checksum. Knowing where the lock bit is stored, you can place a checksum on that byte (in MacsBug, use the step spy ‘ss’ command), causing the debugger to break on the offending line of code.

Future Tools

Last year Devon Hubbard and I wrote a tool called HeapQC that isolated many of the problems that can crop up in a Macintosh heap. This tool relied on very fast heap scrambling to catch “dangling pointer” problems, purge routines to find mismanagement of purgeable blocks, free memory invalidation to uncover “wild” pointers, and sophisticated heap checking that verified all of the linkages described in this article.

We grew disenchanted with the company that distributed the product (read: we were not paid) and decided to create a new company, Onyx Technology to pursue contract work. Before long, we began working on a stress testing tool designed to be order of magnitude more thorough and convenient than anything else on the market.

This new tool, QCPro, is being written from scratch to answer our own in-house needs for quality assurance on the contract jobs we undertake. The plan calls for MMU protection, trap discipline, leak detection, and variable frequency heap examinations (ranging from validating the heap after every instruction to validating it after every trap) as well as a host of more esoteric checks. As developers, we feel the Mac has needed more powerful error detection for some time.

Anyone who has ever chased down a memory bug for the better part of a day (week?) will find this tool invaluable. We are already using it to test itself. If you’d like more information or have suggestions, please don’t hesitate to drop me a line at AppleLink: D2238, America Online: B.Bell5, or CompuServe: 70550,137. We have gotten quite a few good ideas from people online and will try to incorporate as many of them as possible into the final product.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

PopChar 7.1 - Floating window shows avai...
We're also selling a 5-license family pack for only $25.99! PopChar helps you get the most out of your font collection. With its crystal-clear interface, PopChar X provides a frustration-free way to... Read more
BBEdit 11.1.1 - Powerful text and HTML e...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more
Picasa 3.9.139 - Organize, edit, and sha...
Picasa and Picasa Web Albums allows you to organize, edit, and upload your photos to the Web from your computer in quick, simple steps. Arrange your photos into folders and albums and erase their... Read more
Mac DVDRipper Pro 5.0.5 - Copy, backup,...
Mac DVDRipper Pro is the DVD backup solution that lets you protect your DVDs from scratches, save your batteries by reading your movies from your hard disk, manage your collection with just a few... Read more
NetShade 6.2 - Browse privately using an...
This promotion is for NetShade and 1 year of Proxy and VPN services NetShade is an anonymous proxy and VPN app+service for Mac. Unblock your Internet through NetShade's high-speed proxy and VPN... Read more
CrossOver 14.1.3 - Run Windows apps on y...
CrossOver can get your Windows productivity applications and PC games up and running on your Mac quickly and easily. CrossOver runs the Windows software that you need on Mac at home, in the office,... Read more
Little Snitch 3.5.3 - Alerts you about o...
Little Snitch gives you control over your private outgoing data. Track background activity As soon as your computer connects to the Internet, applications often have permission to send any... Read more
OmniGraffle Pro 6.2.3 - Create diagrams,...
OmniGraffle Pro helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use... Read more
OmniFocus 2.2 - GTD task manager with iO...
OmniFocus helps you manage your tasks the way that you want, freeing you to focus your attention on the things that matter to you most. Capturing tasks and ideas is always a keyboard shortcut away in... Read more
1Password 5.3.2 - Powerful password mana...
1Password is a password manager that uniquely brings you both security and convenience. It is the only program that provides anti-phishing protection and goes beyond password management by adding Web... Read more

MooVee - Your Movies Guru (Entertainmen...
MooVee - Your Movies Guru 1.0 Device: iOS iPhone Category: Entertainment Price: $1.99, Version: 1.0 (iTunes) Description: MooVee helps you effortlessly manage your movies, on your iPhone. | Read more »
Geometry Wars 3: Dimensions (Games)
Geometry Wars 3: Dimensions 1.0.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.0 (iTunes) Description: Enjoy the next chapter in the award-winning Geometry Wars franchise and enjoy stunning, console-quality... | Read more »
CHAOS RINGS Ⅲ (Games)
CHAOS RINGS Ⅲ 1.0.0 Device: iOS Universal Category: Games Price: $19.99, Version: 1.0.0 (iTunes) Description: The newest addition to the popular smartphone RPG series is finally here! ・CHAOS RINGS Overview | Read more »
The Popular Insight Series of Travel Gui...
Getting around in a country when you can't understand the primary language can be tough. Fortunately there are several options available to help wold travellers with the important stuff like giving directions to a cab driver or asking where the... | Read more »
Desktop Dungeons is Now on the iPad Desp...
Desktop Dungeons has been a well-loved roguelike on PC for quite some time, and now it's finally available for the iPad! Just the iPad, though. Sorry iPhone users. [Read more] | Read more »
Moleskine Timepage – Calendar for iCloud...
Moleskine Timepage – Calendar for iCloud, Google & Exchange 1.0 Device: iOS iPhone Category: Productivity Price: $4.99, Version: 1.0 (iTunes) Description: The most elegant calendar for your pocket and wrist, Timepage is a... | Read more »
QuizUp Gets Social in its New Update
Plain Vanilla Corp has released a new and improved version of their popular trivia game, QuizUp. The app now emphasizes social play so you can challenge friends from all over the world. [Read more] | Read more »
The Deep (Games)
The Deep 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Swipe Controls Delve into the deep in this retro rogue-like! Swipe to move your diver around and keep away from the enemies as you... | Read more »
Sproggiwood (Games)
Sproggiwood 1.2.8 Device: iOS Universal Category: Games Price: $9.99, Version: 1.2.8 (iTunes) Description: Sproggiwood was developed for devices with at least 1GB of RAM. We recommend you only download Sproggiwood if your device... | Read more »
Battle of Gods: Ascension (Games)
Battle of Gods: Ascension 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: TURN-BASED TACTICAL COMBATFight tactical battles against the forces of Hades! In Battle of Gods: Ascension you play... | Read more »

Price Scanner via MacPrices.net

Apple refurbished 2014 13-inch Retina MacBook...
The Apple Store has Apple Certified Refurbished 2014 13″ Retina MacBook Pros available for up to $400 off original MSRP, starting at $979. An Apple one-year warranty is included with each model, and... Read more
What Would the ideal Apple Productivity Platf...
For the past four years I’ve kept a foot in both the Mac and iPad camps respectively. my daily computing hours divided about 50/50 between the two devices with remarkable consistency. However, there’... Read more
PageMeUp 1.2.1 Ten Dollar Page Layout Applica...
Paris, France-based Softobe, an OS X software development company, has announced that their PageMeUp v. 1.2.1, is available on the Mac App Store for $9.99. The license can be installed on up to 5... Read more
Eight New Products For USB Type-C Application...
Fresco Logic, specialists in advanced connectivity technologies and ICs, has introduced two new product families targeting the Type-C connector recently introduced across a number of consumer... Read more
Scripps National Spelling Bee Launches Buzzwo...
Scripps National Spelling Bee fans can monitor the action at the 2015 Spelling Bee with the new Buzzworthy app for iOS, Android and Windows mobile devices. The free Buzzworthy app provides friendly... Read more
13-inch 2.5GHz MacBook Pro on sale for $120 o...
B&H Photo has the 13″ 2.5GHz MacBook Pro on sale for $979 including free shipping plus NY sales tax only. Their price is $120 off MSRP, and it’s the lowest price for this model (except for Apple’... Read more
27-inch 3.3GHz 5K iMac on sale for $1899, $10...
B&H Photo has the new 27″ 3.3GHz 5K iMac on sale for $1899.99 including free shipping plus NY tax only. Their price is $100 off MSRP. Read more
Save up to $50 on iPad Air 2, NY tax only, fr...
B&H Photo has iPad Air 2s on sale for up to $50 off MSRP including free shipping plus NY sales tax only: - 16GB iPad Air 2 WiFi: $469 $30 off - 64GB iPad Air 2 WiFi: $549.99 $50 off - 128GB iPad... Read more
Updated Mac Price Trackers
We’ve updated our Mac Price Trackers with the latest information on prices, bundles, and availability on systems from Apple’s authorized internet/catalog resellers: - 15″ MacBook Pros - 13″ MacBook... Read more
New 13-inch 2.9GHz Retina MacBook Pro on sale...
B&H Photo has the 13″ 2.9GHz/512GB Retina MacBook Pro on sale for $1699.99 including free shipping plus NY tax only. Their price is $100 off MSRP, and it’s the lowest price for this model from... Read more

Jobs Board

Program Manager, *Apple* Community Support...
**Job Summary** Apple Support Communities ( discussions. apple .com) helps customers get the most from their Apple products and services by providing access to Read more
Senior Data Scientist, *Apple* Retail - Onl...
**Job Summary** Apple Retail - Online sells Apple products to customers around the world. In addition to selling Apple products with unique services such as iPad Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
*Apple* Watch SW Application Project Manager...
**Job Summary** The Apple Watch software team is looking for an Application Engineering Project Manager to work on new projects for Apple . The successful candidate Read more
Engineering Manager for *Apple* Maps on the...
…the Maps App Team get to take part in just about any new feature in Apple Maps, often contributing a majority of the feature work. In our day-to-day engineering work, we Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.