Mach2 - System 7 and 32 Bit Compatible

How to patch MACH2 to bring it up to date

By Jörg Langowski, MacTech Magazine Regular Contributing Author

Here’s news for the Macintosh Forth community. As you know, for quite some time there existed several commercial Forth development systems on the Macintosh, and over the years you have heard about all of them in this column - MacForth, the first one and probably the most widely distributed, Mach2 which generated 68000 code, and NEON, the object-oriented implementation. The only one of these three that is still commercially supported is MacForth, with an active user group and a forum on CompuServe. Let me start this column with the request to you, the readers, that we would really like to see examples of what you have done with MacForth and want to show to others.

MacForth is really the only Forth for the Macintosh that is backed by a companies’ technical support, and for a major development project, it is the wisest choice. On the other hand, you know from reading this column that there are public domain Forth products available on networks etc., and for those of you who just want to get their feet wet in Forth programming, get used to the programming style etc., some of them might be quite useful. Here I would mention in particular the two object-oriented Forths for the Mac: Yerk, the public-domain successor of Neon, and the Yerk (almost) lookalike Mops with its direct 68000 code generation. Then there is also Pocket Forth, which is small and fast and especially suitable for writing desk accessories for those of you who are still writing them under System 7.

Why all these preliminaries? To get you in the right mood for reading a number of letters that crossed my (Mac) desktop in the last few months. Mach2, the ‘other’ commercial Forth that you have been reading about here a lot, has lost its commercial basis. The company that made it, Palo Alto Shipping, is busy with other things now and does not support it anymore. But it seems that Mach2 may soon join the ranks of Yerk and others and be supported by users who don’t want to let go of this product, although it is not clear whether it will be really public domain. One Mach2 developer, Steve Wiley, has already made some modifications that removed many problems that Mach2 had with System 7; but recently someone else joined him who in his desperation had actually disassembled a great deal of Mach2’s code and done some of the work that Palo Alto Shipping should have been doing two years ago. Result: a version of Mach2 that is System 7 compatible, Apple Event aware and 32 bit compatible. Only virtual memory does not work yet, but they may get there. Read on.

[Of course, the wisest direction would probably be to change over to MacForth from Creative Solutions. Their phone number is 800-FORTH-OK. Since Mach2 is not officially public domain, you may eventually run into licensing issues as well as technical problems - Ed.]

1. Steve Wiley (excerpts)

Jörg,

Long time, no news. I am glad that MacTutor has been resurrected from the dead. [(Me too, thanks!] My intentions are to create a totally modern, updated MACH2 for the current systems. I must agree with you that there should be room for a simple, quick and easy programming system for the Mac that doesn't require years of work and study to produce usable results. I would like to keep this philosophy alive with MACH2. people who like using FORTH won't be forced to use other programming systems to remain compatible (except for MacForth, of course). Let me know what you think about this and I will keep you updated on my progress.

- Steve Wiley

2. John Fleming

Dear Jörg:

I have been a user of Mach1 and Mach2 Forth since 1986. And have read your column in MacTutor nearly as long. Since the creators no longer seem to be supporting it, and can’t be reached, I have taken the liberty to update it for System 7. I have done extensive decompilation and patching to make it System 7 compatible, 32-bit clean, and Apple Event aware. I know it works, since I use it every day on my Mac IIci, in System 7, 32-bit mode.

I have also rewritten the trap compiler to allow me to update it as new routines are defined. I have all the System 7 and Communications Toolbox interface routines installed in my version of Mach2.

I also installed some extras, such as the standard DA and XDEF glue words, C-style #ifdef conditional compilation, code-optimized record definition and compilation, and “modules” for the name space so I can load and delete temporary constant definition files.

I started with Mach2 2.14, since that is the latest version I could get a hold of. I understand there is a version 2.24, but I couldn't find anyone who had one who could give me a copy.

Future updates I am considering are:

a) Install an optimizer similar to the one used in MOPS.

b) Make the Editor multi-window

c) Support true code/name loadable modules

To do this first requires decompiling and understanding much of the MACH2 code, and then figuring out how to patch it.

If you think there might be interest in the Forth community, I could distribute the update, along with the various source code files that I used during the modification. Unfortunately, Mach2 is currently not competitive with MOPS, (which I have and use), so I don't think many will want it. But for the die-hard Mach2 users, some might appreciate this update.

Separately, do you know any way to get in touch with the PASC folks? (NO!) All this work I have done would be certainly easier if I could get the source code from them.

Thanks.

Enthusiastic response from JL. Immediate answer:

3. John Fleming again.

Hi Joerg:

It is going to take me a week to prepare a package. I would like to make sure I have my documentation in order on how to use my new words, the trap compiler syntax, and how to use High-Level and Apple Events. I will have to do some form of cpt/hqx to make sure it reaches you. They don't let us have unlimited access to Telnet/FTP over Internet here at Motorola, too much proprietary stuff in our computers. So we have to do it over E-Mail.

I don't mind you announcing this version. I can’t really yet guarantee that every trap definition is 100% perfect, and so on, because I only reverse-engineered what I had to, and haven't yet done a proper software quality inspection. I also used the interface files in Think C 3.0 to determine the exact calling sequence for the System 7 calls. I don’t have any official Apple interface definitions to work from. I just finished this update and haven’t done any System 7 programming except the Apple Event stuff for the Mach2 startup code patch. (Segment 1).

By the way, Apple really messed things up in System 7 in regards to trap routine selector handling. Toolbox and OS routines, depending on the manager, put either word or longword selectors in either D0 or on the stack. And the CTB is completely non-standard. It caused me quite a lot of head-scratching to figure out how to do this. But I think I have created a more general trap compiler than existed in Mach2. If the exact calls are defined improperly, it is easy to modify my source file and re-compile the TEXT 2 resource. And you can easily add new trap definitions as they are defined by Apple. I understand that PASC had included a trap compiler in v2.24. I hope you find my implementation better.

My intention is to make available all my patch files and unique work, so that everyone can see what is going on and fix what I didn’t get quite right. I feel uncomfortable right now about releasing everything I learned about Mach2 from decompiling the whole thing, including all the commented Mach2 source files, until I get a release from the MACH2 creators.

I sent an email to Steve Wiley. Hopefully he got the source code, and we can get these fixes integrated into the source, and release a true update.

I just thought about one other thing. Mach2 has a basic problem in that it uses TRAP 0 to switch between tasks. That is incompatible with A/UX. I never considered fixing that because it was buried too deeply in the MACH2 structure. With the actual source code, we could fix that, install an interface to make A/UX calls, and then MACH2 could be used in an A/UX environment. Wouldn’t that be slick!

I’ll be talking to you soon.

John Fleming flemingj@mot.sat.com

4. Next letter from John Fleming, somewhat later.

Dear Steve and Joerg:

Here is a complete list of what I have done.

A) I have not tested it for Virtual Memory. If what you say is so, my version won’t work. But I know how and where the LoadSeg patch is installed, and we could redefine around that, since I have reverse-engineered the exact startup sequence. I don't think it would be too difficult, based on my experience.

B) I patched the first 5 Kbytes of Segment 11 to completely rewrite the trap compiler. I also wrote a set of words that create the TEXT 2 trap definition resource from a text source file. All routines in Inside Mac Vol 1-6 are installed, along with the Communications Toolbox. I also had to patch the trap compiler stubs located at the bottom of Segment 11. I have not performed a complete quality inspection of the trap definitions, so there may be hidden mistakes.

C) I fixed a bug in the Debugger that involved trying to get a locked pointer to a “Monaco” FONT ID=9. This was incompatible with System 7, TrueType, and NFNT’s. The debugger now installs correctly.

D) I looked through every code segment to find all instances of pointer masking with $00FFFFFF, and replaced them with _StripAddress calls. I created a global variable set at Mach2 startup, per the Apple TN, that contains the _StripAddress mask. I will eventually go back and replace all the _StripAddress calls with an AND.L of this mask.

E) I installed a High Level Event Handler Mechanism, and an Apple Event handler, in the IOTASK (segment 2).

F) I rewrote the startup code, along with associated code in the compiler segment (ID=10), and the Editor Segment (ID=17) to recognize the existence of Apple Events and use them for startup.

G) I rewrote the IOTASK to recognize the existence of WaitNextEvent and use it.

H) I created a set of Record definition words that compile optimized code sequences for records. A record is defined as

 :RECORDsilly
 var1 word
 var2 longword
 var3 SizeOf( otherRecDef )
 var4 word
 ;RECORD

(where a record can be one of VarRec, *VarRec, CodeRec, *CodeRec, local variable or address of a local variable)

with a FORTH sequence of

 " ... silly mysilly VarRec... 
 \ defines and allocates a Variable space record

 " : try ... var2 .OF. mysilly @ ..." 
 compiles to "MOVE.L $-nnn(A5),-(A6)"
 " : try ... var2 .OF. ^ mylvar @ ..." 
 compiles to "MOVE.L $-nnn(A2),-(A6)"

and other equally compact forms for the other record definition types.

Nested records are not supported, such as

 " ... subvar1 .OF. var3 .OF. mysilly ..."

I then rewrote the common Macintosh Interface files to conform to this syntax.

I) I created a set of C-style conditional compilation words, #ifdef .. #else ... #endif and #ifndef, #define .

J) I defined all the TASK variables (why PASC didn't do this is a mystery)

K) I installed the standard stand-alone code module words

 DA: ... ;DA  XDEF: ... ;XDEF

also an Apple Event Handler mechanism that allows your handler to execute with the registers set up for your task:

 AE: my.handler ... ;AE

(thanks to Pocket Forth for the idea)

L) You can now compile ASYNC trap calls:

 "... CALL HOpen,ASYNC ..."

and other trap modifier words. All Trap routines with routine selectors, both stack and register based, are supported. This is completely transparent.

M) I created a word “SysEnvirons” that either calls SysEnvirons if it exists, or fills in the SysEnvRec manually if it doesn't. The SysEnvRec is now a global variable filled in at startup.

O) I created a set of MODULE words that allows redirection of the Name Space to an allocated handle, then allows switching back. This allows a long set of “nn CONSTANT myC” definitions, which occupy only the name space, to exist temporarily, then be deleted, without taking up permanent Name space.

P) The amount of _MoreMasters calls at startup is configurable (thanks to MacForth for the idea)

Q) I reverse-engineered the Dictionary header, and discovered the locations of many of the MACH2 system globals.

R) I decompiled and have extensively commented all of the Mach2 Code segments.

5. Some comments from Steve Wiley about the problem of running Mach2 Forth with virtual memory. Gives some interesting insights in the functioning of virtual memory.

I did an extensive trace of the problem with VM and reached a different conclusion regarding the problem. I don't think that the trap patching is a problem because under system 7 each application keeps a private copy of the dispatch table. My tracing showed that the patch worked correctly under system 7. The problem appears to be that VM itself works by patching the _LoadSeg trap. If you have TMON, you can notice this by setting the trap intercept to _LoadSeg. Under normal memory, _LoadSeg is called rarely, mostly when loading an application. However, when VM is running, _LoadSeg is called constantly and the actual address of the routine changes several different times. So apparently, the problem is that the Mach2 trap switches the stacks, and VM switches the trap from underneath Mach2, resulting in a crash when the local stack is not restored after _LoadSeg returns. The fact is, the _LoadSeg patch is one of the greatest flaws in Mach2 and should be discarded. For example, if a dialog box is executed and _LoadSeg is triggered, the program will always crash because of the stack switching (analogous to the use of CALL instead of (CALL) from within a dialog event loop). At the very least the _LoadSeg patch should be optional as it originally was.

I originally tried to remove the _LoadSeg patch, but was unsuccessful for very odd reasons. It is trivial to simply not execute the patch, but Mach2 would then not run on any machine or system. I could never figure out why. The startup code would execute, but the crash would occur when the program attempted to jump to the main I/O task. A simple inspection suggested that the pointers in the jump table were now wrong, resulting in a jump into the middle of the I/O code. Just a guess. If you would like information on the location and workings of the _LoadSeg trap and how I attempted to patch it, I will be glad to provide specifics.

I don’t know whether you have tried to run Mach2 on the Quadra, but there are problems there also. The difficulty appear to be that Mach2 uses "self-modifying" code in a few places, particularly in the startup code. I have noticed a few examples of this during my own decompilation efforts. The trouble is that the modified code remains in the data cache whereas the instructions are fetched from main memory, resulting in a trip into the great unknown. The solutions are: 1) remove the self-modifying code (the best approach and the main reason I wanted the source code), 2) flush the data cache before using the modified code, 3) turn off the cache during program startup. The third choice was the basis of my patch since it was so simple. I just turn the cache off during startup and turn them back on when I have finished.

6. Reply from John Fleming:

Hi Steve:

I work at Motorola Satellite Communications, in Chandler, Arizona, on the IRIDIUM, program. It is a constellation of satellites that supports world-wide cellular personal communications from hand-held cellular telephones. FORTH for me is just my preference for a personal hack and problem solving engine. Just like every large project (and anymore, any large project is necessarily a large software project), the real software is written by a marching army of programmers who write in C and C++. I need the instant feedback of FORTH when I write code to simulate and model things.

Yes Mach2 is self-modifying during startup. During my decompilation of the entire startup process, I noticed two things:

a) `the MACH2 A5 value is written directly into the ioCompletion routines for the BLOCKs. During an ioCompletion routine, the following code sequence takes place:

MOVE.L A5,-(A7)
 \ save current A5
MOVE.L #$00123124,A5  
 \ $00123124 is written in at startup

b) The whole STATUS/WAKE/SLEEP task switching mechanism is self-modifying code. The startup process builds the I/O and OPERATOR task tables, then jumps directly to

 JMP $-2(A4)

to start the round-robin.

MacTutor about two issues back discussed how to organize code to prevent ’040 cache problems for self-modifying code. I will have to take a look at it. The MACH2 task-switching mechanism needs eventual modification, because it is incompatible with A/UX, which also uses (I think) TRAP #0 as the UNIX call interface. So Mach2 won’t run under Unix.

I never tested my version on a Quadra. I am quite sure, from what you tell me, that it won't run. I was only trying to get my version to work, not to make it right. If you really can get the code, then that would be great. Especially since, the Dialogs, Editor, and Debugger are black holes (but were written in Mach2 FORTH). In fact, all the code segments above 10 were written in FORTH, and the startup code (segment 1) was also. But I haven't done detailed inspection of these segments to determine if there are other sleepers in there than what I have found.

I can replace the use of IAZNotify in my package if I hurry.

I have documented all the code for the _LoadSeg patch. It is all in Segment 4, and is called at startup by the hidden word COLD.

I don't have any ideas on how to remove/replace the LoadSeg code. The problem still remains that the Mac OS occasionally uses most of the 8Kbyte stack. MOPS has a demo file that implements task switching. Maybe I will look at that for ideas.

When BYE is executed, one of the first things it does is de-patch LoadSeg. I figured that out.

Oh, one more thing. The IOTASK and the Editor both used the system global CurActivate during activate events to determine if a Desk accessory is being activated. This allow the correct handling of the menus. In 32-bit mode, CurActivate is used for something else, and is usually an invalid address. I fixed that in my version.

Talk to you soon.

John Fleming

7. (From Steve Wiley)

John,

Your modifications look good. I am particularly impressed with your new record structures. I will also try to put together a package for you that has my modifications in it. Particularly, I have put in a small modification in the I/O task that allows “floating palettes” to exist. These are normal Mach2 tasks that always float above other windows. However, they cannot have a menubar since the window they float above has control of that. By changing the window type IDs and looking for “floating” type, you can redirect specific events from it to the window that “owns” the floating palettes (such as redirecting menu tasks). If you would like, I can send you a copy of an “image-processing” type program written in Mach2 that shows how this works. Other basic functions (in line with your “record” definitions) is a complete set of string handling utilities. One way the floating palettes and the string-handling functions can work in conjunction is a utility I wrote that floats an output window above a multiple-task program. I can print out variables, stack parameters, strings and the such from anywhere in the program and they all appear in the floating window (which also scrolls up and down). This is very useful for debugging complex code on the fly. I will also send the code that shows how to get Mach2 to work with the Quadras. I will try to see whether this works in your modified environment this weekend.

Thanks again for your generous gift. I will be in touch.

Steven Wiley

8. John Fleming:

Hi Steve: Thanks for your comments. I’m not sure at all why it [the text editor - JL] crashed on your IIfx. While I decompiled the editor, I only reverse-engineered two little pieces. I will have to take a look at what I have done. I included the Segment 17 patch code for you to take a look at. The only other thing I did was to manually patch the pointer masking with $00FFFFFF that was going on in Segment 16 and 17, and call StripAddress instead. If you can tell me what you did with your Mach2 in Segment 17 and 16, I might be able to get this one fixed also.

I would like to see your string and task handling code, the Quadra fixes, and the image-processing program. In previous jobs I have had to do image-processing work, so I am always interested in what is out there.

I need to send you one other file. At home I have a patch to the Debugger segment, CODE 14. The debugger needed to be updated for System 7 and TrueType fonts, because it was going out there looking for a Monaco Size 9 FONT (which no longer exists), so I had to do some manipulations with the FMInput and FMOutput records. I also had to include a Monaco FOND and NFNT resource in the application resource fork to make it work correctly.

Thanks for the kind words on the record words. They can really be nice because it makes the code more readable. But it can be a little cumbersome typing in:

ioResult .OF. ^ my.ioPB @

as compared to

^ my.ioPB ioResult + @ 

The NEON-workalike program MOPS has a nice optimizer that creates code from the second example as compact as my record syntax. That way one can have both standard FORTH syntax and compact subroutine-threaded code.

Have fun. This sure would be easier with real source code to work from.

John Fleming

[end of letters]

The letters that you’ve read speak for themselves - this is really an impressive story about ‘user-supported software’.

Most impressive are the new features that John has built into his modified Mach2. This makes that long-forgotten Forth package an up-to-date development system again. You now have full Apple Event support, and since Mach2 is a multi-tasking Forth, you can install separate high-level Event handlers for each task. Also, the trap compiler is in a state where you can accommodate all new system calls that Apple will think up in the future. From the Forth side, I especially liked the support for records and modules. The new record definitions (see above) are finally giving you most of the support for record structures that I had try do define in some early Forth columns. Modules are useful for loading a set of definitions (e.g. constants) that you need temporarily during compilation but which would take up too much name space if they are kept in the name area all the time. You’ll find more about this in the documentation that is on the source code disk.

For those of you who want the modified Mach2, I include in the source code disk the current release of John Fleming’s Mach2 patches. Everything is in there except the running version of Mach2, which we cannot release for obvious reasons. But if you have a copy of Mach2.14, all the documentation and all the instructions how to patch it are in there. The result will be 32-bit clean and high-level event aware. John and Steve are still working on getting Mach2 virtual memory-compatible; let’s hope that we’ll see some succcess soon.

Next month, look forward to reading something about some Fortran goodies again; the new debugging tools from Language Systems just arrived in the mail.