Volume Number: 17 (2001)
Issue Number: 06
Column Tag: From the Factory Floor

The Road from Rhapsody

by Richard Atwell, ©2001 by Metrowerks, Inc., all rights reserved

Since the last installment of this column in February, Mac OS X 1.0 shipped, finally. Developers have gone so long waiting for a new operating system based on a “modern” core, the release was almost anticlimactic. It’s been a long road since NeXT and Apple merged back in December 1996 and Metrowerks has been following the saga from the beginning (See the Factory Floor article in MacTech Vol 13, No. 5 and subsequent articles).

In the earliest days, developers were only able to get their hands on a version of OpenStep for x86. Then Rhapsody appeared on the scene and continued with a few DR versions. Remember when Classic was called the BlueBox? Rhapsody morphed into Mac OS X Server then Mac OS X came into being in addition to the server version. After several Developer Preview (DP) releases of Mac OS X and many Carbon SDK releases, developers now have what they need to develop seriously for Mac OS X: a stable initial version with a rich new set of APIs.

At WWDC 2001, CodeWarrior for Mac OS, Version 7.0 Early Access was released. If you couldn’t come by our booth and pick up a CD, you can purchase a CD for a small shipping cost. See our website. The CD contains our initial support for Mac OS X, which I’ll cover in this article. But let’s catch up on some recent happenings first.

CodeWarrior for Mac OS, Version 6.0 was released last year in which the IDE was carbonized along with other components. Since then, there have been two updates release. There hasn’t been a Factory Floor article on Java for a long time so I’ll leave out those details and try to cover them in a future article.

In January we shipped a CodeWarrior Version 6.1 Update. Included was an IDE 4.1.0.3 that contained a single bug fix to correct a problem where custom key bindings were sometimes forgotten. The update also contained a newer MetroNub Plugin that fixed a problem where AltiVec registers wouldn’t display correctly the first time you debugged your target. Lastly, the patch addressed some MSL problems. See the release notes for details:

ftp://ftp.metrowerks.com/pub/updates/CWMacOS6/6_1_Update_Release_Notes.sit

In May Metrowerks shipped a CodeWarrior Version 6.2 Update. Although Version 6.0 was the last release to include 68K support, the update included a newer C/C++ compiler and linker to fix some lingering bugs. Our front end compiler received a number of fixes also and a new version of the PowerPC compiler was released. Among the highlights in the compiler were support for greater than 32-bit bitfields, new scheduling support, pooling of floating point constants and support for a new .axp file that will prevent symbols from being exported. There are too many other improvements to list here, so be sure to check out the extensive release notes. (The URL for this wasn’t available when this article was submitted.)

The biggest change in Mac OS X besides the rewriting of the Finder, is that PEF is no longer the preferred execution format for your code (despite the fact that this is what PEF stands for.) Mach–O is the new kid on the block, having moved from Redwood City to Cupertino back in 1997.

Mach–O is the native file format for storing executable code on Mac OS X that was developed for the defunct NeXT operating system. This is the equivalent of the ‘CODE’ resource for 68K programming, and the PEF container for PowerPC development since System 7.1.2. The ‘Mach’ part comes from Mach 3.0, the microkernel that is the foundation of Mac OS X. If you want to develop natively for Mach–O you need a linker that can produce that format.

Are there any compelling reasons to use Mach–O? Mac OS X still contains backward compatibility for Mac OS 9 in the form of support for CFM/PEF. If you carbonize your application appropriately using CodeWarrior Version 6.0 and use our PEF linker to write out your application, the application will still work on Mac OS X but it does so using a behind-the-scenes trick.

When you launch a CFM/PEF application on Mac OS X using the Finder, it launches an application called LaunchCFMApp. This is a native Mach–O application that lives within the Carbon.framework folder (more on this later). What it does is take a path to a CFM/PEF application as an argument and builds a mini-CFM world for your application to live in, effectively hosting it for the life of your program. This helps to bridge the divide between your application and the Mach–O universe but it’s hardly ideal. Your application will always launch slowly because LaunchCFMApp will have to launch first.

Why didn’t Apple combine CFM/PEF with the new kernel? Why are they requiring developers to “reformat” their applications in order to take ultimate advantage of the new OS? A similar question is why did they rewrite the venerable Finder? One reason probably has to do with the price of progress and the other development tools.

The NeXT OS came with the ubiquitous UNIX GCC compiler and GDB debugger, and modifying those tools would mean adding support that wasn’t already available in the open source community. What about MPW? They were command line-like and supported CFM/PEF, but they’ve been in maintenance mode for several years and they didn’t integrate with Project Builder and Interface Builder — Apple’s tools for rapid development using Cocoa. If you’ve been to Apple’s developer site over the last two years, you’ve probably noticed that they’ve emphasized the importance of using Cocoa for developers wishing to write new applications for Mac OS X (Cocoa used to be called Yellow Box before Carbon was introduced.) In my own personal opinion, it was their drive to provide developers with a way to write new applications quickly that drove them to the decision to switch to Mach–O. But it’s anyone guess.

So here we are, just as we were in the days right before the PowerPC introduction, with a developer need for a toolset that embraces new Apple technologies. Where does CodeWarrior fit in?

You’ve got an application that you wrote with CodeWarrior a while back and need to bring it over to Mac OS X. It does everything your users want it to do but you’ve just discovered all the new APIs that Mac OS X provides in the /System/Library/Frameworks folder at the root of your Mac OS X volume. These frameworks are essentially folders that combine the header files you need to compile your application with the libraries the application will need to call at runtime. This is in contrast to the modern Mac OS programming model: build you application against CFM stub libraries that we get from Apple and ship with CodeWarrior in the MacOS Support folder.

Apple has provided CFBundle and CFPlugin into the CoreFoundation.framework to help CFM applications communicate with Mach–O binaries. But for those who need to truly take advantage of all the new APIs that Mac OS X provides, the choice is to rebuild your application as Mach–O.

What CodeWarrior offers in this regard are updates to many of our components as well as a few new additions: a revised IDE and debugger, a new compiler and linker, an updated Powerplant and Constructor, and our MSL C++ Library. Let’s cover these each in depth. Hopefully not much will change since I’m writing this before we’ve completed everything for the WWDC Early Access CD, so consider this a true preview.

In CodeWarrior Version 6.0 we provided a carbonized IDE 4.1 as a first step towards moving our toolset to Mac OS X. Beyond tweaking the interface for Aqua we’ve been upgrading the project engine within the IDE to support frameworks. The first clue is a new fourth tab in your project window that appears when you are generating a Mach–O binary.

Figure 1. Project Window.

The Frameworks tab organizes the /System/Library/Frameworks that you need to link your application against. In Figure 1, Carbon refers to the Carbon.framework that exists on disk. Examine the framework and inside you’ll see a symbolic link called Current that points to another folder that contains the current framework version. This version may be labeled A, B, C, etc. Apple can update the framework in a newer version of Mac OS X and leave the old version there so as not to break binary compatibility with your application if the changes they have to make are radical. The final version of the IDE will allow you to browse the headers that the framework contains as well as choosing the particular framework version you want to use. You’ll add frameworks by using the Add Files… item from the Project Menu to modify your project.

My sample application is called “Application”. Let’s look at the target settings that you have control over.

Figure 2. Target Settings

Creating a Mach–O target starts with the selection of a new linker. The linker we’re using is a CodeWarrior plug-in version of Apple’s linker in much the same way that PPCAsm, our PowerPC assembler plug-in, is a repackaged version of the MPW tool with the same name. There is a LibImport Mach–O plug-in for importing native libraries into our project format and for now a separate compiler plug-in for Mach–O which we might combine with the MW C/C++ PPC compiler in the future.

What’s changed with the Access Paths that are usually setup for you by our stationery projects?

Figure 3. Access Paths

The first thing to notice is that the Interpret DOS and Unix Paths checkbox is set. This feature is necessary in order to support frameworks. Look at this simple application that puts up an alert then quits.

#include 

enum
{
kAlertID = 128
};

static void Initialize(void)
{
InitCursor();
}

int main(void)
{
Initialize();

NoteAlert(kAlertID, NULL);

return 0;
}

Everything looks typical except the ‘include’ at the top. Instead of including MacTypes.h and its friends you #include instead. If you were building for some BSD Unix system a path like #include would look normal; it would tell the compiler to look in the sys directory along the include path and find stat.h. On Mac OS X, because of its NeXT lineage, you won’t find a Carbon directory along the include path. Instead, the name before the / means look in the Carbon.framework directory along the include path.

Look at the system include paths in our panel. The first two, :usr:include and :usr:lib, are standard Unix include directories. In the first one you’ll find files like stdio.h; in the second library files like crt1.o, the pre-installed C runtime library. The approach we’ve taken to adapting our MSL C/C++ implementation to Mach–O has been to favor the built-in support for the C library and provide our C++ implementation on top, because not all C++ libs are created equal and porting your code will be easiest this way.

The third path points to our MacOS X support folder. This is a small folder compared to MacOS Support (even without the Universal folder) that contains our runtime sources and library along with a Mach–O version of MacHeaders, our precompiled header file.

The last two point to framework directories on your Mac OS X volume. At last count, these two folders contained over 13,000 files. As you can imagine when opening a project, our project engine goes for a bender trying to locate all the header and library files that your project is referencing. We’re working on a way to make the searching as efficient as possible due to this increase of header files.

We aren’t providing these files like we do with the Universal folder within MacOS Support on our Tools CD. If you are still going to run CodeWarrior from Mac OS 9, you’ll either need to have these files available from your Mac OS X installation volume or copy them near to CodeWarrior. Either way, to allow the access paths to work correctly you’ll need to set up a Source Tree as seen in Figures 3 and 4.

We introduced Source Trees in CodeWarrior Version 5.0. Source Trees and allow you to create a custom reference point for your access paths. We’ve provided Source Trees for System, Compiler and Project for as far back as I can remember, so any path you make relative to those will allow you to distribute a project to others that will build correctly in their similarly set-up CodeWarrior environment. For Mach–O development we’re requiring you to create a source tree called {OS X Volume}. When the latest IDE starts up for the first time, you’ll be prompted to create one and you can see the result when you open up the IDE Preferences.

Figure 4. Source Trees

My source tree is called OS X Volume and points to my Mac OS X volume. If I had copied the files to my Mac OS 9 Volume the source tree could simply point to the folder I created there to hold the support files. There are target level source trees in addition to the global settings that every project inherits. You may elect to use the target level setting for distributing your projects but it’s easiest to use a global setting.

Take a look at the PPC Mach–O target panel. This panel is enabled for your project by the selection of the Apple linker. The first thing to notice is that there is an option to flatten resources. This is the preferred way to create resource files for Mac OS X. You can use this to output a non-localized resource file, such as MyApp.rsrc or localized ones for each language system you plan to support (more below on resources and folder layout). The MacOS Merge linker not shown here can also flatten resources. This is handy because you will be able to create targets and sub-targets to build and manage localized applications.

So what kind of Mach–O binaries can we produce? Look at the new project types: Application Bundle, Executable, Library, Object and Shared Library (these names might change in the final release).

Figure 5. PPC Mach–O Target

Mach–O applications can load bundles to perform functions that CFM applications would load shared library plug-ins to do. Apple calls these “loadable bundles” because bundles are simply folders that collect things. Frameworks are bundles that contain headers and libraries. What visually distinguishes bundles from one another is the extension that is applied: .app for applications and their collective resources, .frameworks we’ve talked about already, .bundle for generic bundles or .whatever you might decide upon for a loadable bundle. Other variants include .debug for debug versions of applications and .profile to applications that emit profiling data.

Bundles use a directory layout to support multiple architectures. We’ve called these FAT for years on Mac OS. In NeXTStep and OpenStep they were called obese. If you want you can create a bundle that loads and executes a CFM app on Mac OS 9 and a Mach–O version on Mac OS X. The choice of binary is done by the OS and happens transparently to the user.

Bundles can also be localized and globalized easily because of the layout. At runtime the OS looks for localized resources and picks the resources for the active script system. Bundles are also flat, meaning there are no resource forks so they can easily reside on UFS, NFS and Win32 fileservers, whose file systems are flat by nature. Lastly, bundles can be moved around on disk as an atomic unit so items inside aren’t rearranged or lost by your users.

An executable is exactly what it describes: it corresponds to a traditional application with a filetype of ‘APPL’. You could use this to write a command line tool or write out an application within an .app bundle. A stationery project is an effective way to setup an .app bundle layout instead of having the linker emit these files the first time it builds. Here’s an example folder layout for an application package called MyApp.app:

Figure 6. Package Layout

Packages appeared in Mac OS 9 in preparation for Mac OS X. Originally the bundle bit for the enclosing folder was reused to indicate a package but since then that method has been augmented to use the structure in Figure 6. Everything lives within the Contents folder and within that folder there are several items of note. The first is the Info.plist file which is like the ‘vers’ resource from Mac OS.

/* Localized versions of Info.plist keys */

CFBundleName = “MyApp”;
CFBundleShortVersionString = “MyApp version 0.1”;
CFBundleGetInfoString = “MyApp version 0.1, Copyright 2001 by Metrowerks, Inc.”;

PkgInfo is the crucial file that decides whether or not the Finder will display the folder as a package. This is supported by Mac OS 9.1 in favor of the bundle bit. Its contents are even simpler. Be careful where you move the PkgInfo file because if it’s within a Contents folder it will turn the parent folder into a package. Mac OS 9 has a contextual menu option that lets you show/hide package contents.

APPL????

The last file, Info.plist, is required to describe the bundle in detail. It’s an XML file:

<?xml version=”1.0” encoding=”UTF-8”?>
<!DOCTYPE plist SYSTEM “file://localhost/System/Library/DTDs/PropertyList.dtd”>
<plist version=”0.9”>
<dict>
   <key>CFBundleDevelopmentRegion</key>
   <string>English</string>
   <key>CFBundleExecutable</key>
   <string>MyApp</string>
   <key>CFBundleInfoDictionaryVersion</key>
   <string>6.0</string>
   <key>CFBundlePackageType</key>
   <string>APPL</string>
   <key>CFBundleSignature</key>
   <string>????</string>
   <key>CFBundleVersion</key>
   <string>0.1</string>
</dict>
</plist>

Remember these are just sample file contents. See the extensive Apple document about these files using the URLs at the end of this article.

Back to the Mach–O Target Panel: the last three options are more easily described. Library is for generating a static library with the .a extension. Shared Library is for dynamic libraries with the .dylib extension. Typically dynamic libraries with this extension are designated so because they don’t live within a framework. Dynamic libraries are loaded by dyld, the Mach–O loader equivalent of CFM.

On Mac OS X, Mach–O dynamic libraries are lazily loaded. What this means is that symbols are resolved when functions in libraries are called instead of just when the application loads the library. The benefit of this is that code won’t get loaded if the execution path of the program results in those functions not getting called. The downside is that this slows down the running of applications a bit. You can pre-bind your application to avoid this using our linker options but, as always, there is a trade-off.

The last option is Object which is equivalent to the –r option you would pass to the command-line linker, ld. It’s not as useful as the others but you can learn more about it by typing “man ld” without the quotes from the Mac OS X Terminal.app.

The next new panel is a variation on the PPC Processor Panel that is used for generating PEF applications.

Figure 7. PPC CodeGen Mach-O

Notice that the TOC options are gone because we are dealing with Mach-O. The Common Variables and Implicit Templates checkbox are new. The first is a GCC–ism that prevents warnings from appearing because a common variable is multiply-defined in several header files. The front end of our compiler has never liked this but if you are trying to compile code that was previously built with GCC this option is useful. The second allows you to explicitly define which object files should have implementations of the template functions: another GCC–ism.

The last panel is for the Mach–O linker and there’s a host of new options. I’ll cover the ones that don’t also relate to the PEF linking.

Figure 8. PPC Mach-O Linker

Emit Full Path in Stab Debug Info determines whether full pathnames are stored in the symbolics. You would typically disable this option so you could build your target on one machine then give your project to a friend to debug on another machine. Another use would be when you need to move it from the folder where you built it to another folder in order to debug it.

What is stabs? For PEF debugging, we generate a separate .xSYM file that contains the symbolic information that allows our debugger to map your source code to the binary file that our linker produced. The Apple Mach-O linker generates stab symbolics the default way it is built. This is the standard symbolic file format that most Unix systems use. The GCC compiler generates it and the GDB debugger consumes it. Stab symbolics output is extremely verbose compared to xSYM. Unlike the separate file that is the case with xSYM, stabs is generated as a section of the output binary. To remove the stabs info from your debug application you can use the strip command-line tool but afterwards you won’t be able to debug it. If you build your debug application as Mach-O and all of a sudden it’s gigantic, blame the stabs format.

Emit Error on Multiple Symbols is an option to have the linker fail if multiple symbols are defined. The PEF linker generates a warning if you have symbols multiply-defined, but there is no option for failure.

Pre-bind External Symbols avoids the lazy binding that the Mach–O loader perform on your application. Do this to speed up your application. There’s probably a memory usage penalty for doing this but I’ve yet to completely experiment with it.

Use Objective-C Semantics is used for loading all the classes and categories from static libraries. Objective-C programs are really meant as dynamic libraries.

Current Version and Compatible Version tag dynamic libraries so you can check programmatically whether or not to load a particular library. Zero is the default value and it means no checking. If a number other than zero is placed there it has an X[.Y.[Z]]] format where X is a number between 0 and 65535 and Y and Z are both between 0 and 255 and optional.

Report Which Files Loaded produces output to the Error & Warnings window in the form of notes that tell you which object modules from which libraries were loaded in order to link your program.

/usr/lib/libSystem.B.dylib(k_rem_pio2.o)
/usr/lib/libSystem.B.dylib(cprocs.o)
/usr/lib/libSystem.B.dylib(isatty.o)
/usr/lib/libSystem.B.dylib(ungetc.o)

Report Why Files Loaded is slightly more verbose in that it also tells you which symbols caused the library to load.

/usr/lib/libSystem.B.dylib(k_rem_pio2.o) loaded to resolve symbol: ___kernel_rem_pio2
/usr/lib/libSystem.B.dylib(cprocs.o) loaded to resolve symbol: __cproc_fork_child
/usr/lib/libSystem.B.dylib(isatty.o) loaded to resolve symbol: _isatty
/usr/lib/libSystem.B.dylib(ungetc.o) loaded to resolve symbol: _ungetc

Treat Read-Only Relocs as Errors/Warnings/Suppressed for Mach–O is also a new option. To quote our compiler engineers, “it’s bad (for performance) to have relocations in a read-only data section, but it is supported. The user can select to generate an error and fail, generate a warning and continue, or do nothing and continue. Normally the compiler will not generate these Read-Only relocations, but there are some code generation situations that might cause this to happen.”

That’s it for the IDE and compilers. There’s quite a bit that’s new to consume but the stationery projects will help you along.

The hardest part of writing a debugger to support Mach–O is having to support a new symbolics format. If you’ve been to our website and seen the other products we sell, all of them have had one thing in common until recently: no support for stabs. We’ve been able to read DWARF, CodeView and of course xSYM for a number of years but we’ve never had a stabs reader to leverage for Mac OS X. Writing a stabs reader was a first step towards support for Mach–O debugging. The fruit of this labor is the StabSymbolics plugin that we’ll ship in the CodeWarrior Plugins:Debugger: folder along with the SymSymbolics plugin our debugger currently uses.

While we were doing this we had to provide a debugging solution for developers carbonizing their CFM/PEF applications for Mac OS X. At some point in the debugging process our debugger has to interact with the operating system to read/write memory, get register information and help to implement run control (step, run, stop). In addition, debuggers need to know when libraries load so debuggers can set breakpoints in them. To perform these tasks on Mac OS 9 we use our own code: the venerable MetroNub extension which was primarily written because Mac OS didn’t have robust enough debugger support built-in. Some debugging support appears in the PowerPC version of System 7 but at the time we needed a debugger that supported 68K and PowerPC debugging equally. See MacTech Vol 13, No. 2 for the whole story.

For Mac OS X, our debugger runs on top of GDB. If you’ve never used GDB before it uses a command line interface. If you’re used to CodeWarrior and not a hard core Unix person you may find it extremely difficult to use at first. When you use CodeWarrior to debug a PEF application, GDB is launching LaunchCFMApp behind the scenes and our debugger is reading the xSYM symbolics for your application and manipulating GDB which is providing the low level debugging. Sound complicated? All you see is the CodeWarrior interface as GDB is completely hidden behind the scenes.

Our initial support for Mac OS X debugging was in the form of a remote debugging solution: Mac OS 9 ran the IDE and Mac OS X ran the application being debugged. In the middle was an application called DebugNubController that passed messages back and forth between the two systems over the network using TCP/IP. Having two machines wasn’t ideal, so in CodeWarrior Version 6.0 we shipped an improved system that didn’t require two Macs, sometimes called local or single machine debugging that we’re all used to. Figuring out how to do this was tough up until a certain point because we had the same problem that other developers had: how to talk to Mach-O from a CFM application. We figured that out and now include a .bundle with CodeWarrior to let us interact with the Mach–O world ourselves.

Writing a compiler is no small task but at least you can debug it on Mac OS 9. Since we use CodeWarrior every day to write CodeWarrior, debugging the debugger on Mac OS X presented a classic bootstrapping problem and understandably we’ve been taking small steps debugging the debugger compared to the rest of the toolset. So where are we now?

We’ve been working with Apple to provide a satisfactory Mac OS X debugging solution. This has included revising GDB many times to allow us to debug CFM applications better, especially those that load shared libraries. We’ll continue to improve this over time. Now that we have a stabs reader, we’re working on the debugging of Mach–O applications. Check out our pre-release of this support when it becomes available.

Not much to say about MSL except that our C++ implementation can run on top of the built-in C library. You’ll also notice that in the next release there will be no 68K support. We made this decision based on your feedback and the fact that supporting 68K, PowerPC, Carbon and now Mach-O is simply unnecessary these days as Apple hasn’t shipped any 68k based hardware for a number of years now. The 68K tools work well, so just continue to use CodeWarrior Version 6.0 for any legacy development you need to continue.

PowerPlant is our popular application framework and Constructor is its companion UI design tool. For the early access release we updated Constructor to read and write flattened resource files. This is to allow you to create your bundles more easily.

PowerPlant has undergone a number of improvements: fixes for drawing origin problems on Aqua, support for building with Universal Interfaces 3.4 and the latest Carbon SDKs, improved Carbon printing support, better support for Aqua and the biggest news is that the Appearance Classes moved out of the _In�Progress folders.

That’s my article about our official support for Mach-O in a nutshell. Mac OS X is a new beginning for users and developers alike. Get these tools at WWDC or contact us directly afterward and send us your feedback a.s.a.p. so we can make sure that CodeWarrior for Mac OS, Version 7.0 is a great release.

If you’d like to get in touch with us about CodeWarrior issues, post to our newsgroup or email us directly. Visit our website at www.metrowerks.com to learn more about us.

• http://developer.apple.com/techpubs/macosx/macosx.html
• http://www.oreillynet.com/mac/
• http://www1.fatbrain.com/documentation/apple/

Newsgroup: comp.sys.mac.programmer.codewarrior
Technical Support: cw_support@metrowerks.com
Report Bugs: cw_bug@metrowerks.com
Suggestions: cw_suggestion@metrowerks.com

Richard Alexander David Atwell, aka ratwell, collaborates on CodeWarrior development tools at Metrowerks and takes time out to keep fellow CodeWarriors informed about the world of Metrowerks.