TweetFollow Us on Twitter

Playing with Blocks
Volume Number:8
Issue Number:7
Column Tag:Debugging

Related Info: Memory Manager

Playing with Blocks

Memory manager structures - a software autopsy of your application's death!

By Brooks Bell, Bradenton, Florida

Note: Source code files accompanying article are located on MacTech CD-ROM or source code disks.

Let’s set the ‘Way Back’ machine for 1985. Programmers of the day complained heavily of the information that Apple was keeping to itself. Inside Macintosh just did not provide enough details.

The chapter on the Memory Manager, for example, was fuzzy on the meaning of certain fields in the Zone header. SparePtr, allocPtr even the ever-popular flags field were defined only as ‘used internally’. Even worse, fields such as cntRel, maxRel, and cntHandles were described as ‘not used’, when any debugger showed that they clearly contained valid data.

Of course, we have all by now learned that these things are hidden from programmers because they are fair game for change. What works for a 512K “Fat” Mac isn’t necessarily applicable to a Quadra with 64MB of RAM. We have grown accustomed to the trade off: Apple shields us from some OS details in exchange for blessed future hardware compatibility.

That is why it is something of a surprise to turn a few pages further in IM II and discover a very detailed description of the old 24 bit memory manager block structure. In retrospect, this knowledge led directly to the problems of 32 bit uncleanliness. Here is how to set the Master Pointer ‘lock’ bit and bypass the HLock trap overhead. Lo3Bytes is described - who needs StripAddress? Everything that you need to make a 32 bit dirty application is right here in black and white.

With the 32 bit memory manager and Inside Macintosh VI, having learned from their mistakes, Apple decided not to publish the Memory Manager block structures. Well written applications will use HGetState, HLock, etc There is no need to risk compatibility by accessing these structures directly.

However, when you are in the middle of debugging some code errors can crop up that cause an invalid heap. Because the Mac ROMs do not check parameters for errors, it is easy to pass a trap a bad value and have it write garbage all over the heap. It is also easy to move data into a disposed handle, store data beyond the bounds of an array, maybe even call BlockMove directly and blast data all over creation. Lacking sophisticated memory protection, the Macintosh heap is easily destroyed.

Perhaps the worst aspect of such damage is that fatal errors may not appear for some time. When they do show up in the form of the friendly ‘illegal instruction’, ‘Stack has collided with the heap’, ‘Bus Error’ or whatever the program may have crawled, bleeding, thousands of instructions away from the offending line of code.

It is at this point that the knowledge of memory manger structures can often be put to good use. If you are lucky enough to still have a low level debugger functioning, knowing the structure of the heap can provide vital information in “cracking the case”. Think of it as a software autopsy to determine the cause of your application’s death.

Healthy, Happy Heaps

At the highest level, the heap is made up of three basic components. There is one Zone Header, one Zone Trailer, and in between the two lie any number of blocks. The blocks occupy all the space in the heap and are divided into three categories: free, non-relocatable, and relocatable. This article will focus on the internal structure of these blocks.

Figure 1

With the recent addition of the new Inside Macintosh Memory volume Apple has decided to publish the 32 bit heap structures they left out of IM VI. Note that using these structures in a program guarantees incompatibility when Apple changes the memory manager. Precisely when that will happen is anyone’s guess, but with PowerPC looming on the horizon I’d imagine this information has a useful shelf life of less than two years.

Both 24 bit and 32 bit memory manager blocks have the same general structure. Blocks always begin with a Block Header, followed by any amount of logical data. A variable sized padding field may be tacked onto the end. The size of the entire structure is called the block’s physical size. Applications never deal with this size. The size of the variable length logical data field is called the logical size. It is this logical size that is returned when your application issues a GetHandleSize of GetPtrSize trap.

Figure 2

The blocks are arranged one after another in the heap. There is no empty space separating blocks: every byte in the heap between the zone header and zone trailer “belongs” to one and only one block. What an application thinks of as “free” memory is contained in “free” blocks.

Of the three fields in the Memory Manager Block, only the contents of the Block Header is of interest to us. The padding field exists to keep the blocks long word aligned on 68020, 68030 and 68040 class machines and to enforce a minimum block size of twelve bytes on all machines. The padding field’s contents are unused.

The logical data field’s contents are application specific - you can put whatever you like into Handles and Pointers that you allocate. In a free block, the logical data field will contain whatever garbage is remaining in memory at the time the block is created.

The structure of the block header depends on whether you are running the 24 or 32 bit memory manager. The 24 bit version is 8 bytes long, while the 32 bit version is 12. The 32 bit version has expanded several of the old fields and added one new field to accommodate larger memory models.

Figure 3

Defined in C, these two structures are as follows:

/* 1 */

typedef struct {
 long   blockType:2; // See defines below
 long   unused:2;
 long   sizeCorrection:4; 
 long   physSize:24;
 VariantData     v;
}Block24Header;

typedef struct {
 short  blockType:2;
 short  unused:6;
 short  flagBits:8;
 Byte   unused;
 Byte   sizeCorrection; 
 long   physSize;
 VariantData     v;
}Block32Head;

The first field is the block type. These two bits indicate whether a block is free, nonrelocatable, or relocatable (i.e., unused, a pointer, or a handle). In C, the defines for these values are:

/* 2 */

 #definefreeType 0x00
 #definenonRelocType 0x01
 #definerelocType0x02

The size correction field determines the size of the padding field in the block structure, while the physSize field contains the size of the entire block. To get the logical size of the block, take the physical size and subtract the sizeCorrection and the block header size (8 bytes for the 24 bit memory manager and 12 bytes for the 32 bit memory manager). In the 32 bit memory manager the physSize field has been widened to accommodate larger block sizes.

To allow for full 32 bit addresses, it was also necessary to change the location of the Master Pointer flags. In the 24 bit memory manager, these flags (lock, purge, and resource) occupied the high byte of each master pointer, while the lower 24 bits addressed a relocatable block. In the 32 bit memory manager, all 32 bits of the master pointer are significant in determining the address of a relocatable block. Routines such as HLock have been recoded to look for these flag in the flagBits field of the 32 bit block header. The bit usage within the flagBits field is exactly the same as it was for the high byte of the 24 bit Memory Manager’s master pointers: lock is the most significant bit, followed by purge and then resource. The other bits of the flagBits byte are reserved.

Variant Data

The variant data in either block header is a 32 bit value whose meaning depends on the block type:

/* 3 */

typedef union {
 // relocatable block: Offset from zone to master ptr
 long   relHand;
 // non-relocatable block: address of heap zone
 THz    itsZone;
 // free block
 long   unused;
}VariantData;

For a free block, the information is unused. For a non-relocatable block, this field contains the address of the heap zone that “owns” the block. For a relocatable block, the variant data contains an offset from the start of the heap to this block’s master pointer.

MacsBug Templates

In the examples that follow, I’ll be poking around with MacsBug in a heap while running the 32 bit memory manager. A MacsBug template and macro will come in handy. Open up the Debugger Prefs file (found in the System Folder) using ResEdit and create these mxwt and mxbm resources:

Figure 4

Once these are installed in the Debugger Prefs file, make sure that the Memory control panel is set to 32 bit addressing and reboot to let MacsBug load these resources.

Examining the Heap

After rebooting I enter MacsBug via the Programmer’s switch. The heap dump command will give us a view of the current heap (of course, your addresses will vary):

hd
 Displaying the Application heap at 012F7904
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N
•012F7A50 000026F4+00R  012F7A38 L CODE0003  0BC2
•012FA150 00000024+00R  012F7A0C L acur1964  0BC2
•012FA180 00000044+00R  012F7A08 L CURS1964  0BC2
etc 

The block at 012F7944 is the first block in the heap following the Zone header. Let’s use our new macro on the second block. First we have to display some memory to equate the MacsBug ‘.’ pseudo-register with location 012F7A50.

dm 12F7A50
 Displaying memory from 12f7a50
  012F7A50  0318 0001 4EFA 26BE  0000 0000 0000 6100  

This is the logical data that an application program would see. Now we can invoke our macro to display the preceding 12 bytes with our block header template:

Block32
 Displaying Block32Header at 012F7A44
  012F7A44  blockType          80 
  012F7A45  flagBits           A0 
  012F7A46  unused             00 
  012F7A47  sizeCorrection     00 
  012F7A48  PhysSize           00002700 
  012F7A4C  Variant            00000134 

Note that 012F7A44 is twelve bytes before the start of the logical data. The blockType of $80 is %1000 0000 in binary. Recall that the first two bits designate the block type and that a 0x02 value in this field signifies a relocatable block. Although the next six bits are reserved for future use, I have never seen anything in them other than zeros. As a human convenience, we can interpret our template blockType field as follows: $80 = relocatable, $40 = non-relocatable, $00 = relocatable. So the $80 value agrees with the Tag R designation in the MacsBug display.

The flagBits field contains A0. This is analogous to having a 24 bit Master Pointer’s high byte set to $A0. In other words, the lock bit and resource bits are set ($80+$20). Again, this agrees with the MacsBug display: the Lock column is set and the resource link (CODE 0003 0BC2) establishes this as a resource.

The sizeCorrection field is set to $00, so no size correction was necessary. MacsBug shows this as +00 next to the size field of 000026F4+00. Our physSize field shows 00002700. MacsBug displays the logical size as $26F4. Convert physical size to logical size by subtracting the block header size (a constant 12 decimal bytes) and the size correction (in this case, zero):

$2700- $C - $0 = $26F4

The next block in the heap should be found “physSize” bytes beyond the start of this blocks header:

$012F7A44 + $2700 = 012FA144
dm 12FA144 Block32Header
 Displaying Block32Header at 012FA144
  012FA144  blockType          80 
  012FA145  flagBits           A0 
  012FA146  unused             00 
  012FA147  sizeCorrection     00 
  012FA148  PhysSize           00000030 
  012FA14C  Variant            00000108 

That’s the next block, all right (the logical data starts at 012F7A50, which agrees with the MacsBug hd command output we received earlier).

Back to the block at $012F7A44. The Variant portion of the block header has a value of $00000134. This is the offset from the start of the heap to this block’s master pointer. From the MacsBug hd command we know we are looking at an application heap at 012F7904. Adding $00000134 gives:

$012F7904 + $00000134 = $012F7A38

Is this our master pointer? Examining the contents shows that it does indeed point back to the logical start of our block ($012F7A50):

dl 12F7A38
 Long at 012F7A38 = $012F7A50  #19888720  #19888720   '•/zP'

The master pointer itself can be anywhere within the logical portion of a non-relocatable block. To find the start of the block containing this Master Pointer, we’ll use the MacsBug where command:

wh 012F7A38
Address 012F7A38 is in the Application heap at 012F7904 
 It is 000000F4 bytes into this heap block:
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N

We can use our macro to look at the master pointer block’s header:

Block32
 Displaying Block32Header at 012F7938
  012F7938  blockType          40 
  012F7939  flagBits           00 
  012F793A  unused             00 
  012F793B  sizeCorrection     00 
  012F793C  PhysSize           0000010C 
  012F7940  Variant            012F7904 

Note that this time, the blockType is $40, indicating a non-relocatable block. Again there is no size correction necessary. The Variant portion of the non-relocatable block contains the address of the heap: $012F7904.

Lets examine the zone header at this address in more detail:

dm 012F7904 Zone
 Displaying Zone at 012F7904
  012F7904  bkLim              0133D818 ->  
  012F7908  purgePtr           012F7938 ->  
  012F790C  hFstFree           012FA740 ->  
  012F7910  zcbFree            000080DC 
  012F7914  gzProc             0006F264 ->  
  012F7918  moreMast           0040 
  012F791A  flags              0000 
  012F7922  heapType           01 
  012F792C  purgeProc          0004503C ->  
  012F7930  sparePtr           4080ED12 ->  
  012F7934  allocPtr           01302140 ->  

Block32
 Displaying Block32Header at 012F78F8
  012F78F8  blockType          80 
  012F78F9  flagBits           80 
  012F78FA  unused             00 
  012F78FB  sizeCorrection     00 
  012F78FC  PhysSize           00055C0C 
  012F7900  Variant            01244508 

Hmm this heap zone is itself a locked relocatable block. MultiFinder (or in this case, System 7) allocates application heaps within its own heap as locked relocatable blocks. Where is the MultiFinder heap? A where command on the physical block location will let us know:

wh 012F78F8
 Address 012F78F8 is in the heap at 0010B5EC 
 It is FFFFFFF4 bytes into this heap block (in the block header):
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  •     012F7904
 00055C00+00R  0134FAF4 L

The MultiFinder heap is at address $0010B5EC (ignore that FFFFFFF4 stuff, MacsBug is reporting the negative blocksize offset from the start of our original heap). You can easily see the MultiFinder heap in the MacsBug hz command - all heaps contained within it are indented beneath it.

Conclusion

It is easy to see why some “invalid heap” errors can be so hard to detect. Suppose your program is writing data to a handle and mistakenly oversteps the handle size by one byte. If the block in question has padding bytes, whose value is meaningless, no symptoms will occur. However, if your handle was allocated as a multiple of four (and is larger than four bytes), this wayfaring byte will overwrite the blockType tag byte of the following block! Will you be alerted? Not necessarily: since just two bits of the tag are significant only a value of %11 in the top two bits of the byte will cause the next block to be invalid. Unfortunately, a value of %00 could “free” the next block, causing havoc later when a heap compaction moves other blocks into this “free” space.

We can use our knowledge of block structures to help in debugging. A locked block that is somehow being unlocked can be detected through use of a checksum. Knowing where the lock bit is stored, you can place a checksum on that byte (in MacsBug, use the step spy ‘ss’ command), causing the debugger to break on the offending line of code.

Future Tools

Last year Devon Hubbard and I wrote a tool called HeapQC that isolated many of the problems that can crop up in a Macintosh heap. This tool relied on very fast heap scrambling to catch “dangling pointer” problems, purge routines to find mismanagement of purgeable blocks, free memory invalidation to uncover “wild” pointers, and sophisticated heap checking that verified all of the linkages described in this article.

We grew disenchanted with the company that distributed the product (read: we were not paid) and decided to create a new company, Onyx Technology to pursue contract work. Before long, we began working on a stress testing tool designed to be order of magnitude more thorough and convenient than anything else on the market.

This new tool, QCPro, is being written from scratch to answer our own in-house needs for quality assurance on the contract jobs we undertake. The plan calls for MMU protection, trap discipline, leak detection, and variable frequency heap examinations (ranging from validating the heap after every instruction to validating it after every trap) as well as a host of more esoteric checks. As developers, we feel the Mac has needed more powerful error detection for some time.

Anyone who has ever chased down a memory bug for the better part of a day (week?) will find this tool invaluable. We are already using it to test itself. If you’d like more information or have suggestions, please don’t hesitate to drop me a line at AppleLink: D2238, America Online: B.Bell5, or CompuServe: 70550,137. We have gotten quite a few good ideas from people online and will try to incorporate as many of them as possible into the final product.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Microsoft Remote Desktop 8.0.19 - Connec...
With Microsoft Remote Desktop, you can connect to a remote PC and your work resources from almost anywhere. Experience the power of Windows with RemoteFX in a Remote Desktop client designed to help... Read more
OmniGraffle 6.3 - Create diagrams, flow...
OmniGraffle helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use Graffle to... Read more
PDFKey Pro 4.3.2 - Edit and print passwo...
PDFKey Pro can unlock PDF documents protected for printing and copying when you've forgotten your password. It can now also protect your PDF files with a password to prevent unauthorized access and/... Read more
Ableton Live 9.2.2 - Record music using...
Ableton Live lets you create and record music on your Mac. Use digital instruments, pre-recorded sounds, and sampled loops to arrange, produce, and perform your music like never before. Ableton Live... Read more
Macs Fan Control 1.3.1.0 - Monitor and c...
Macs Fan Control allows you to monitor and control almost any aspect of your computer's fans, with support for controlling fan speed, temperature sensors pane, menu-bar icon, and autostart with... Read more
NetShade 6.3.1 - Browse privately using...
NetShade is an anonymous proxy and VPN app+service for Mac. Unblock your Internet through NetShade's high-speed proxy and VPN servers spanning seven countries. NetShade masks your IP address as you... Read more
Dragon Dictate 4.0.7 - Premium voice-rec...
With Dragon Dictate speech recognition software, you can use your voice to create and edit text or interact with your favorite Mac applications. Far more than just speech-to-text, Dragon Dictate lets... Read more
Persecond 1.0.2 - Timelapse video made e...
Persecond is the easy, fun way to create a beautiful timelapse video. Import an image sequence from any camera, trim the length of your video, adjust the speed and playback direction, and you’re done... Read more
GIMP 2.8.14p2 - Powerful, free image edi...
GIMP is a multi-platform photo manipulation tool. GIMP is an acronym for GNU Image Manipulation Program. The GIMP is suitable for a variety of image manipulation tasks, including photo retouching,... Read more
Sandvox 2.10.2 - Easily build eye-catchi...
Sandvox is for Mac users who want to create a professional looking website quickly and easily. With Sandvox, you don't need to be a Web genius to build a stylish, feature-rich, standards-compliant... Read more

ReBoard: Revolutionary Keyboard (Utilit...
ReBoard: Revolutionary Keyboard 1.0 Device: iOS Universal Category: Utilities Price: $1.99, Version: 1.0 (iTunes) Description: Do everything within the keyboard without switching apps! If you are in WhatsApp, how do you schedule a... | Read more »
Tiny Empire (Games)
Tiny Empire 1.1.3 Device: iOS Universal Category: Games Price: $2.99, Version: 1.1.3 (iTunes) Description: Launch cannonballs and blow tiny orcs into thousands of pieces in this intuitive fantasy-themed puzzle shooter! Embark on an... | Read more »
Astropad Mini (Productivity)
Astropad Mini 1.0 Device: iOS iPhone Category: Productivity Price: $4.99, Version: 1.0 (iTunes) Description: *** 50% off introductory price! ​*** Get the high-end experience of a Wacom tablet at a fraction of the price with Astropad... | Read more »
Emo Chorus (Music)
Emo Chorus 1.0.0 Device: iOS Universal Category: Music Price: $1.99, Version: 1.0.0 (iTunes) Description: Realistic Choir simulator ranging from simple Chorus emulation to full ensemble Choir with 128 members. ### introductory offer... | Read more »
Forest Spirit (Games)
Forest Spirit 1.0.5 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0.5 (iTunes) Description: | Read more »
Ski Safari 2 (Games)
Ski Safari 2 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: The world's most fantastical, fun, family-friendly skiing game is back and better than ever! Play as Sven's sister Evana, share... | Read more »
Lara Croft GO (Games)
Lara Croft GO 1.0.47768 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.47768 (iTunes) Description: Lara Croft GO is a turn based puzzle-adventure set in a long-forgotten world. Explore the ruins of an ancient... | Read more »
Whispering Willows (Games)
Whispering Willows 1.23 Device: iOS Universal Category: Games Price: $4.99, Version: 1.23 (iTunes) Description: **LAUNCH SALE 50% OFF** - Whispering Willows is on sale for 50% off ($4.99) until September 9th. | Read more »
Calvino Noir (Games)
Calvino Noir 1.1 Device: iOS iPhone Category: Games Price: $3.99, Version: 1.1 (iTunes) Description: The film noir stealth game. Calvino Noir is the exploratory, sneaking adventure through the 1930s European criminal underworld.... | Read more »
Angel Sword (Games)
Angel Sword 1.0 Device: iOS Universal Category: Games Price: $6.99, Version: 1.0 (iTunes) Description: Prepare to adventure in the most epic full scale multiplayer 3D RPG for mobile! Experience amazing detailed graphics in full HD.... | Read more »

Price Scanner via MacPrices.net

Apple and Cisco Partner to Deliver Fast-Lane...
Apple and Cisco have announced a partnership to create a “fast lane” for iOS business users by optimizing Cisco networks for iOS devices and apps. The alliance integrates iPhone with Cisco enterprise... Read more
Apple offering refurbished 2015 13-inch Retin...
The Apple Store is offering Apple Certified Refurbished 2015 13″ Retina MacBook Pros for up to $270 (15%) off the cost of new models. An Apple one-year warranty is included with each model, and... Read more
Apple refurbished 2015 MacBook Airs available...
The Apple Store has Apple Certified Refurbished 2015 11″ and 13″ MacBook Airs (the latest models), available for up to $180 off the cost of new models. An Apple one-year warranty is included with... Read more
21-inch iMacs on sale for up to $120 off MSRP
B&H Photo has 21″ iMacs on sale for up to $120 off MSRP including free shipping plus NY sales tax only: - 21″ 1.4GHz iMac: $999.99 $100 off - 21″ 2.7GHz iMac: $1199.99 $100 off - 21″ 2.9GHz iMac... Read more
5K iMacs on sale for up to $150 off MSRP, fre...
B&H Photo has the 27″ 3.3GHz 5K iMac on sale for $1899.99 including free shipping plus NY tax only. Their price is $100 off MSRP. They have the 27″ 3.5GHz 5K iMac on sale for $2149.99 $2199.99, $... Read more
1.4GHz Mac mini, refurbished, available for $...
The Apple Store has Apple Certified Refurbished 1.4GHz Mac minis available for $419. Apple’s one-year warranty is included, and shipping is free. Their price is $80 off MSRP, and it’s the lowest... Read more
iPad Air 2 on sale for up to $100 off MSRP
Best Buy has iPad Air 2s on sale for up to $100 off MSRP on their online store for a limited time. Choose free shipping or free local store pickup (if available). Sale prices available for online... Read more
MacBook Airs on sale for $100 off MSRP
Best Buy has MacBook Airs on sale for $100 off MSRP on their online store. Choose free shipping or free local store pickup (if available). Sale prices for online orders only, in-store prices may vary... Read more
Big Grips Lift Handle For iPad Air and iPad A...
KEM Ventures, Inc. which pioneered the extra-large, super-protective iPad case market with the introduction of Big Grips Frame and Stand in 2011, is launching Big Grips Lift featuring a new super-... Read more
Samsung Launches Galaxy Tab S2, Its Most Powe...
Samsung Electronics America, Inc. has announced the U.S. release of the Galaxy Tab S2, its thinnest, lightest, ultra-fast tablet. Blending form and function, elegant design and multitasking power,... Read more

Jobs Board

*Apple* Evangelist - JAMF Software (United S...
The Apple Evangelist is responsible for building and cultivating strategic relationships with Apple 's small and mid-market business development field teams. This Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Desktop Analyst - KDS Staffing (Unit...
…field and consistent professional recruiting achievement. Job Description: Title: Apple Desktop AnalystPosition Type: Full-time PermanentLocation: White Plains, NYHot Read more
*Apple* Systems Engineer (Mclean, VA and NYC...
Title: Apple Systems Engineer (Mclean, VA and NYC) Location: United States-New York-New York-200 Park Ave (22005) Other Locations: United States-Virginia-Vienna-Towers Read more
*Apple* Systems Engineer (Mclean, VA and NYC...
…Assist in providing strategic direction and technical leadership within the Apple portfolio, including desktops, laptops, and printing environment. This person will Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.