TweetFollow Us on Twitter

Playing with Blocks
Volume Number:8
Issue Number:7
Column Tag:Debugging

Related Info: Memory Manager

Playing with Blocks

Memory manager structures - a software autopsy of your application's death!

By Brooks Bell, Bradenton, Florida

Note: Source code files accompanying article are located on MacTech CD-ROM or source code disks.

Let’s set the ‘Way Back’ machine for 1985. Programmers of the day complained heavily of the information that Apple was keeping to itself. Inside Macintosh just did not provide enough details.

The chapter on the Memory Manager, for example, was fuzzy on the meaning of certain fields in the Zone header. SparePtr, allocPtr even the ever-popular flags field were defined only as ‘used internally’. Even worse, fields such as cntRel, maxRel, and cntHandles were described as ‘not used’, when any debugger showed that they clearly contained valid data.

Of course, we have all by now learned that these things are hidden from programmers because they are fair game for change. What works for a 512K “Fat” Mac isn’t necessarily applicable to a Quadra with 64MB of RAM. We have grown accustomed to the trade off: Apple shields us from some OS details in exchange for blessed future hardware compatibility.

That is why it is something of a surprise to turn a few pages further in IM II and discover a very detailed description of the old 24 bit memory manager block structure. In retrospect, this knowledge led directly to the problems of 32 bit uncleanliness. Here is how to set the Master Pointer ‘lock’ bit and bypass the HLock trap overhead. Lo3Bytes is described - who needs StripAddress? Everything that you need to make a 32 bit dirty application is right here in black and white.

With the 32 bit memory manager and Inside Macintosh VI, having learned from their mistakes, Apple decided not to publish the Memory Manager block structures. Well written applications will use HGetState, HLock, etc There is no need to risk compatibility by accessing these structures directly.

However, when you are in the middle of debugging some code errors can crop up that cause an invalid heap. Because the Mac ROMs do not check parameters for errors, it is easy to pass a trap a bad value and have it write garbage all over the heap. It is also easy to move data into a disposed handle, store data beyond the bounds of an array, maybe even call BlockMove directly and blast data all over creation. Lacking sophisticated memory protection, the Macintosh heap is easily destroyed.

Perhaps the worst aspect of such damage is that fatal errors may not appear for some time. When they do show up in the form of the friendly ‘illegal instruction’, ‘Stack has collided with the heap’, ‘Bus Error’ or whatever the program may have crawled, bleeding, thousands of instructions away from the offending line of code.

It is at this point that the knowledge of memory manger structures can often be put to good use. If you are lucky enough to still have a low level debugger functioning, knowing the structure of the heap can provide vital information in “cracking the case”. Think of it as a software autopsy to determine the cause of your application’s death.

Healthy, Happy Heaps

At the highest level, the heap is made up of three basic components. There is one Zone Header, one Zone Trailer, and in between the two lie any number of blocks. The blocks occupy all the space in the heap and are divided into three categories: free, non-relocatable, and relocatable. This article will focus on the internal structure of these blocks.

Figure 1

With the recent addition of the new Inside Macintosh Memory volume Apple has decided to publish the 32 bit heap structures they left out of IM VI. Note that using these structures in a program guarantees incompatibility when Apple changes the memory manager. Precisely when that will happen is anyone’s guess, but with PowerPC looming on the horizon I’d imagine this information has a useful shelf life of less than two years.

Both 24 bit and 32 bit memory manager blocks have the same general structure. Blocks always begin with a Block Header, followed by any amount of logical data. A variable sized padding field may be tacked onto the end. The size of the entire structure is called the block’s physical size. Applications never deal with this size. The size of the variable length logical data field is called the logical size. It is this logical size that is returned when your application issues a GetHandleSize of GetPtrSize trap.

Figure 2

The blocks are arranged one after another in the heap. There is no empty space separating blocks: every byte in the heap between the zone header and zone trailer “belongs” to one and only one block. What an application thinks of as “free” memory is contained in “free” blocks.

Of the three fields in the Memory Manager Block, only the contents of the Block Header is of interest to us. The padding field exists to keep the blocks long word aligned on 68020, 68030 and 68040 class machines and to enforce a minimum block size of twelve bytes on all machines. The padding field’s contents are unused.

The logical data field’s contents are application specific - you can put whatever you like into Handles and Pointers that you allocate. In a free block, the logical data field will contain whatever garbage is remaining in memory at the time the block is created.

The structure of the block header depends on whether you are running the 24 or 32 bit memory manager. The 24 bit version is 8 bytes long, while the 32 bit version is 12. The 32 bit version has expanded several of the old fields and added one new field to accommodate larger memory models.

Figure 3

Defined in C, these two structures are as follows:

/* 1 */

typedef struct {
 long   blockType:2; // See defines below
 long   unused:2;
 long   sizeCorrection:4; 
 long   physSize:24;
 VariantData     v;
}Block24Header;

typedef struct {
 short  blockType:2;
 short  unused:6;
 short  flagBits:8;
 Byte   unused;
 Byte   sizeCorrection; 
 long   physSize;
 VariantData     v;
}Block32Head;

The first field is the block type. These two bits indicate whether a block is free, nonrelocatable, or relocatable (i.e., unused, a pointer, or a handle). In C, the defines for these values are:

/* 2 */

 #definefreeType 0x00
 #definenonRelocType 0x01
 #definerelocType0x02

The size correction field determines the size of the padding field in the block structure, while the physSize field contains the size of the entire block. To get the logical size of the block, take the physical size and subtract the sizeCorrection and the block header size (8 bytes for the 24 bit memory manager and 12 bytes for the 32 bit memory manager). In the 32 bit memory manager the physSize field has been widened to accommodate larger block sizes.

To allow for full 32 bit addresses, it was also necessary to change the location of the Master Pointer flags. In the 24 bit memory manager, these flags (lock, purge, and resource) occupied the high byte of each master pointer, while the lower 24 bits addressed a relocatable block. In the 32 bit memory manager, all 32 bits of the master pointer are significant in determining the address of a relocatable block. Routines such as HLock have been recoded to look for these flag in the flagBits field of the 32 bit block header. The bit usage within the flagBits field is exactly the same as it was for the high byte of the 24 bit Memory Manager’s master pointers: lock is the most significant bit, followed by purge and then resource. The other bits of the flagBits byte are reserved.

Variant Data

The variant data in either block header is a 32 bit value whose meaning depends on the block type:

/* 3 */

typedef union {
 // relocatable block: Offset from zone to master ptr
 long   relHand;
 // non-relocatable block: address of heap zone
 THz    itsZone;
 // free block
 long   unused;
}VariantData;

For a free block, the information is unused. For a non-relocatable block, this field contains the address of the heap zone that “owns” the block. For a relocatable block, the variant data contains an offset from the start of the heap to this block’s master pointer.

MacsBug Templates

In the examples that follow, I’ll be poking around with MacsBug in a heap while running the 32 bit memory manager. A MacsBug template and macro will come in handy. Open up the Debugger Prefs file (found in the System Folder) using ResEdit and create these mxwt and mxbm resources:

Figure 4

Once these are installed in the Debugger Prefs file, make sure that the Memory control panel is set to 32 bit addressing and reboot to let MacsBug load these resources.

Examining the Heap

After rebooting I enter MacsBug via the Programmer’s switch. The heap dump command will give us a view of the current heap (of course, your addresses will vary):

hd
 Displaying the Application heap at 012F7904
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N
•012F7A50 000026F4+00R  012F7A38 L CODE0003  0BC2
•012FA150 00000024+00R  012F7A0C L acur1964  0BC2
•012FA180 00000044+00R  012F7A08 L CURS1964  0BC2
etc 

The block at 012F7944 is the first block in the heap following the Zone header. Let’s use our new macro on the second block. First we have to display some memory to equate the MacsBug ‘.’ pseudo-register with location 012F7A50.

dm 12F7A50
 Displaying memory from 12f7a50
  012F7A50  0318 0001 4EFA 26BE  0000 0000 0000 6100  

This is the logical data that an application program would see. Now we can invoke our macro to display the preceding 12 bytes with our block header template:

Block32
 Displaying Block32Header at 012F7A44
  012F7A44  blockType          80 
  012F7A45  flagBits           A0 
  012F7A46  unused             00 
  012F7A47  sizeCorrection     00 
  012F7A48  PhysSize           00002700 
  012F7A4C  Variant            00000134 

Note that 012F7A44 is twelve bytes before the start of the logical data. The blockType of $80 is %1000 0000 in binary. Recall that the first two bits designate the block type and that a 0x02 value in this field signifies a relocatable block. Although the next six bits are reserved for future use, I have never seen anything in them other than zeros. As a human convenience, we can interpret our template blockType field as follows: $80 = relocatable, $40 = non-relocatable, $00 = relocatable. So the $80 value agrees with the Tag R designation in the MacsBug display.

The flagBits field contains A0. This is analogous to having a 24 bit Master Pointer’s high byte set to $A0. In other words, the lock bit and resource bits are set ($80+$20). Again, this agrees with the MacsBug display: the Lock column is set and the resource link (CODE 0003 0BC2) establishes this as a resource.

The sizeCorrection field is set to $00, so no size correction was necessary. MacsBug shows this as +00 next to the size field of 000026F4+00. Our physSize field shows 00002700. MacsBug displays the logical size as $26F4. Convert physical size to logical size by subtracting the block header size (a constant 12 decimal bytes) and the size correction (in this case, zero):

$2700- $C - $0 = $26F4

The next block in the heap should be found “physSize” bytes beyond the start of this blocks header:

$012F7A44 + $2700 = 012FA144
dm 12FA144 Block32Header
 Displaying Block32Header at 012FA144
  012FA144  blockType          80 
  012FA145  flagBits           A0 
  012FA146  unused             00 
  012FA147  sizeCorrection     00 
  012FA148  PhysSize           00000030 
  012FA14C  Variant            00000108 

That’s the next block, all right (the logical data starts at 012F7A50, which agrees with the MacsBug hd command output we received earlier).

Back to the block at $012F7A44. The Variant portion of the block header has a value of $00000134. This is the offset from the start of the heap to this block’s master pointer. From the MacsBug hd command we know we are looking at an application heap at 012F7904. Adding $00000134 gives:

$012F7904 + $00000134 = $012F7A38

Is this our master pointer? Examining the contents shows that it does indeed point back to the logical start of our block ($012F7A50):

dl 12F7A38
 Long at 012F7A38 = $012F7A50  #19888720  #19888720   '•/zP'

The master pointer itself can be anywhere within the logical portion of a non-relocatable block. To find the start of the block containing this Master Pointer, we’ll use the MacsBug where command:

wh 012F7A38
Address 012F7A38 is in the Application heap at 012F7904 
 It is 000000F4 bytes into this heap block:
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  Name
•012F7944 00000100+00N

We can use our macro to look at the master pointer block’s header:

Block32
 Displaying Block32Header at 012F7938
  012F7938  blockType          40 
  012F7939  flagBits           00 
  012F793A  unused             00 
  012F793B  sizeCorrection     00 
  012F793C  PhysSize           0000010C 
  012F7940  Variant            012F7904 

Note that this time, the blockType is $40, indicating a non-relocatable block. Again there is no size correction necessary. The Variant portion of the non-relocatable block contains the address of the heap: $012F7904.

Lets examine the zone header at this address in more detail:

dm 012F7904 Zone
 Displaying Zone at 012F7904
  012F7904  bkLim              0133D818 ->  
  012F7908  purgePtr           012F7938 ->  
  012F790C  hFstFree           012FA740 ->  
  012F7910  zcbFree            000080DC 
  012F7914  gzProc             0006F264 ->  
  012F7918  moreMast           0040 
  012F791A  flags              0000 
  012F7922  heapType           01 
  012F792C  purgeProc          0004503C ->  
  012F7930  sparePtr           4080ED12 ->  
  012F7934  allocPtr           01302140 ->  

Block32
 Displaying Block32Header at 012F78F8
  012F78F8  blockType          80 
  012F78F9  flagBits           80 
  012F78FA  unused             00 
  012F78FB  sizeCorrection     00 
  012F78FC  PhysSize           00055C0C 
  012F7900  Variant            01244508 

Hmm this heap zone is itself a locked relocatable block. MultiFinder (or in this case, System 7) allocates application heaps within its own heap as locked relocatable blocks. Where is the MultiFinder heap? A where command on the physical block location will let us know:

wh 012F78F8
 Address 012F78F8 is in the heap at 0010B5EC 
 It is FFFFFFF4 bytes into this heap block (in the block header):
 Start  Length Tag Mstr Ptr Lock Prg Type    ID    File  •     012F7904
 00055C00+00R  0134FAF4 L

The MultiFinder heap is at address $0010B5EC (ignore that FFFFFFF4 stuff, MacsBug is reporting the negative blocksize offset from the start of our original heap). You can easily see the MultiFinder heap in the MacsBug hz command - all heaps contained within it are indented beneath it.

Conclusion

It is easy to see why some “invalid heap” errors can be so hard to detect. Suppose your program is writing data to a handle and mistakenly oversteps the handle size by one byte. If the block in question has padding bytes, whose value is meaningless, no symptoms will occur. However, if your handle was allocated as a multiple of four (and is larger than four bytes), this wayfaring byte will overwrite the blockType tag byte of the following block! Will you be alerted? Not necessarily: since just two bits of the tag are significant only a value of %11 in the top two bits of the byte will cause the next block to be invalid. Unfortunately, a value of %00 could “free” the next block, causing havoc later when a heap compaction moves other blocks into this “free” space.

We can use our knowledge of block structures to help in debugging. A locked block that is somehow being unlocked can be detected through use of a checksum. Knowing where the lock bit is stored, you can place a checksum on that byte (in MacsBug, use the step spy ‘ss’ command), causing the debugger to break on the offending line of code.

Future Tools

Last year Devon Hubbard and I wrote a tool called HeapQC that isolated many of the problems that can crop up in a Macintosh heap. This tool relied on very fast heap scrambling to catch “dangling pointer” problems, purge routines to find mismanagement of purgeable blocks, free memory invalidation to uncover “wild” pointers, and sophisticated heap checking that verified all of the linkages described in this article.

We grew disenchanted with the company that distributed the product (read: we were not paid) and decided to create a new company, Onyx Technology to pursue contract work. Before long, we began working on a stress testing tool designed to be order of magnitude more thorough and convenient than anything else on the market.

This new tool, QCPro, is being written from scratch to answer our own in-house needs for quality assurance on the contract jobs we undertake. The plan calls for MMU protection, trap discipline, leak detection, and variable frequency heap examinations (ranging from validating the heap after every instruction to validating it after every trap) as well as a host of more esoteric checks. As developers, we feel the Mac has needed more powerful error detection for some time.

Anyone who has ever chased down a memory bug for the better part of a day (week?) will find this tool invaluable. We are already using it to test itself. If you’d like more information or have suggestions, please don’t hesitate to drop me a line at AppleLink: D2238, America Online: B.Bell5, or CompuServe: 70550,137. We have gotten quite a few good ideas from people online and will try to incorporate as many of them as possible into the final product.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

MacUpdate Desktop 6.1.3 - Search and ins...
MacUpdate Desktop 6 brings seamless 1-click app installs and version updates to your Mac. With a free MacUpdate account and MacUpdate Desktop 6, Mac users can now install almost any Mac app on... Read more
ExpanDrive 5.4.1 - Access cloud storage...
ExpanDrive builds cloud storage in every application, acts just like a USB drive plugged into your Mac. With ExpanDrive, you can securely access any remote file server directly from the Finder or... Read more
Espionage 3.6.6 - Simple, state-of-the-a...
Espionage offers state-of-the-art encryption and plausible deniability for your confidential data. Sometimes, encrypting your data isn't enough to protect it. That's why Espionage 3 goes beyond data... Read more
Pinegrow Web Designer 2.94 - Mockup and...
Pinegrow Web Designer is desktop app that lets you mockup and design webpages faster with multi-page editing, CSS and LESS styling, and smart components for Bootstrap, Foundation, Angular JS, and... Read more
1Password 6.3.3 - Powerful password mana...
1Password is a password manager that uniquely brings you both security and convenience. It is the only program that provides anti-phishing protection and goes beyond password management by adding Web... Read more
Sublime Text 3126 - Sophisticated text e...
Sublime Text is a sophisticated text editor for code, markup, and prose. You'll love the slick user interface, extraordinary features, and amazing performance. Features Goto Anything. Use Goto... Read more
ForkLift 3.0 Beta 2 - Powerful file mana...
ForkLift is a powerful file manager and ferociously fast FTP client clothed in a clean and versatile UI that offers the combination of absolute simplicity and raw power expected from a well-executed... Read more
OmniFocus 2.7.1 - GTD task manager with...
OmniFocus helps you manage your tasks the way that you want, freeing you to focus your attention on the things that matter to you most. Capturing tasks and ideas is always a keyboard shortcut away in... Read more
CleanApp 5.1.1 - Application deinstaller...
CleanApp is an application deinstaller and archiver.... Your hard drive gets fuller day by day, but do you know why? CleanApp 5 provides you with insights how to reclaim disk space. There are... Read more
Together 3.6.1 - Store and organize all...
Together helps you organize your Mac, giving you the ability to store, edit and preview your files in a single clean, uncluttered interface. Features Smart storage. With simple drag-and-drop... Read more

Concrete Jungle (Games)
Concrete Jungle 1.16 Device: iOS Universal Category: Games Price: $4.99, Version: 1.16 (iTunes) Description: A follow up to the puzzle hit 'MegaCity'! Concrete Jungle is a new take on the city building genre that swaps micro-... | Read more »
5 great apps for the budget traveller
Travelling abroad, or even within your home country, has never been easier thanks to our handy smartphone companions. There are hundreds of apps on the market that promise to make your world journeys hassle-free, but we've selected five of the... | Read more »
Zip—Zap (Games)
Zip—Zap 1.01 Device: iOS Universal Category: Games Price: $1.99, Version: 1.01 (iTunes) Description: Touch to contract.Release to let go.Bring the clumsy mechanical beings home. · · · over 100 levelsno adsno in-app-purchases Zip—... | Read more »
Paperback: The Game (Games)
Paperback: The Game 1.0 Device: iOS Universal Category: Games Price: $3.99, Version: 1.0 (iTunes) Description: You are an author trying to finish kitschy paperback novels. Complete Westerns, Science Fiction, Romance or even a Crime... | Read more »
How to Rule With a Firm Hand in My Majes...
My Majesty is a kingdom management sim not unlike August’s magisterial hit, Reigns. It’s essentially a reskin of developer Tigrido’s previous management sim, Dictator. As supreme ruler of the land, you must consult with a number of subjects to... | Read more »
Our 5 Favorite iMessage Sticker Packs
At long last, iMessage joins the ranks of messaging apps the likes of LINE and Whatsapp, adding an impressive collection of stickers. They’re a great way to add a little something extra to your daily conversations. [Read more] | Read more »
How to get past Vulture Island's tr...
Vulture Island is a colorful and quirky mish-mash of platforming and puzzles. It’s creative and fresh, but sometimes the game can throw a curveball at you, leaving you stuck as to how you should progress. These tips will help you explore smoothly... | Read more »
The new Clash of Kings is just for Weste...
If you’ve played the original Clash of Kings, you’ll probably recognise the city building, alliance forging and strategic battles in Clash of Kings: The West. What sets this version apart is that it’s tailor made for a Western audience and the... | Read more »
Frost - Survival card game (Games)
Frost - Survival card game 1.12.1 Device: iOS Universal Category: Games Price: $3.99, Version: 1.12.1 (iTunes) Description: *Warning: the game will work on iPhone 5C and above and iPad Pro / 4. Other devices are not supported* | Read more »
How to build and care for your team in D...
Before you hit the trail and become a dog sledding legend, there’s actually a fair bit of prep work to be done. In Dog Sled Saga, you’re not only racing, you’re also building and caring for a team of furry friends. There’s a lot to consider—... | Read more »

Price Scanner via MacPrices.net

Apple refurbished Mac minis available startin...
Apple has Certified Refurbished Mac minis available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: - 1.4GHz Mac mini: $419 $80 off MSRP - 2.6GHz Mac... Read more
13-inch 2.5GHz MacBook Pro available for $928...
Overstock has the 13″ 2.5GHz MacBook Pro available for $927.99 including free shipping. Their price is $171 off MSRP. Read more
Buying McLaren Would Give Apple Instant Car C...
Apple “iCar” rumors have waxed and waned over the years, piquing interest and speculation as to whether Apple is seriously interested in getting into the automotobile business, either in a joint... Read more
Aetna to Transform Members’ Consumer Health E...
Health care benefits company Aetna, which has an estimated 46.3 million clients, today announced a new initiative to revolutionize members consumer health experience by combining the power of iOS... Read more
USB-IF Announces USB Audio Device Class 3.0 S...
USB Implementers Forum (USB-IF), the support organization for the advancement and adoption of USB technology, today announced the USB Audio Device Class 3.0 specification to establish USB Audio over... Read more
Clearance 12-inch 1.2GHz Retina MacBooks, App...
Apple has Certified Refurbished 2015 12″ 1.2GHz Retina MacBooks available for $1189, or $410 off original MSRP. Apple will include a standard one-year warranty with each MacBook, and shipping is free... Read more
Logitech SmartDock and Skype For Business Com...
Logitech has announced Logitech SmartDock, an AV meeting room solution designed in collaboration with Microsoft. Logitech SmartDock works with Skype for Business and qualified devices, including... Read more
27-inch iMacs on sale for up to $220 off MSRP
B&H Photo has 27″ Apple iMacs on sale for up to $200 off MSRP including free shipping plus NY sales tax only: - 27″ 3.3GHz iMac 5K: $2099 $200 off MSRP - 27″ 3.2GHz/1TB Fusion iMac 5K: $1899.99 $... Read more
Apple Macs and iPads available for up to $300...
Purchase a new Mac or iPad using Apple’s Education Store and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free, and... Read more
Save up to $600 with Apple refurbished Mac Pr...
Apple has Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The following... Read more

Jobs Board

US- *Apple* Store Leader Program - Apple (Un...
…Summary Learn and grow as you explore the art of leadership at the Apple Store. You'll master our retail business inside and out through training, hands-on Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Restaurant Manager (Neighborhood Captain) - A...
…in every aspect of daily operation. WHY YOU'LL LIKE IT: You'll be the Big Apple . You'll solve problems. You'll get to show your ability to handle the stress and Read more
Sr. *Apple* Mac Engineer - Net2Source Inc....
…staffing, training and technology. We have following position open with our client. Sr. Apple Mac Engineer6+ Months CTH Start date : 19th Sept Travelling Job If Read more
*Apple* Retail - Multiple Positions-Norfolk,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.