Protocols for an IAC Driver, Part 1

By Frank Alviani, Contributing Editor, Paul Snively, Contributing Editor

How to Get Your Applications Talking to Each Other Again

[Frank and Paul win our “Program of the Month” Award for this fantastic presentation on an elegant way to do something Apple said can’t be done. As contributing editors, both Frank and Paul have consistently shared their talent and knowledge with the rest of the developer community and we extend our thanks to them for their willingness to do so. This was the ultimate hack at MacHack this year, in my opinion. -Ed]

The meeting had just broken up, Paul and I were at the local beer-and-beef joint, and we were feeling rather annoyed. We had been told that Apple wasn’t going to be implementing any sort of method for really using MultiFinder to the max until at least System 7. No form of inter-application communications until at least then, they said.

“Can’t really be all that difficult,” I said over a cookie milk shake. “For Pete’s sake, you could use a device driver to get the needed ‘alien’ memory needed for communicating! Drivers normally have to grab some working memory when they open anyway, so it’s no big deal.”

“Of course it’s not that hard! You and I could probably do a complete job--with sample applications to show Dave Smith that we’re not circling in a low orbit-- in a few evenings,” Paul rumbled back over his cookie shake (they make awfully good shakes there...). “In fact, we oughta do just that! Show Apple all the talent isn’t inside the city limits of Cupertino...”

Thus was born the SAWS Inter-Application-Communications Protocol (detailed in the accompanying description of the protocol). Our purpose in writing it was to show that communicating between programs under MultiFinder isn’t really that difficult in principle (a proof by demonstration) and have some fun in the process. In the next few months we will be publishing the device driver, its installation INIT, and a variety of samples. Various other members of the Chicago-area Mac Gang are joining in on the fun by writing and submitting simple applications such as graphing programs, text editors, etc. that communicate between each other using the IAC protocols.

While the protocol is copyrighted,

anybody may use it freely.

We hope that encouraging its widespread use will help spread the concept of mix-and-match software in the community and let people discover how nice really personalized, “custom-blended” software is. Widespread experimentation in the freeware/shareware arena may also help us all find out what interface approaches work and what approaches don’t.

Assumptions: I made the following assumptions about my users in designing the protocol:

1) Programs must explicitly be written to use this protocol. It is NOT written to be retro-fitted onto existing applications. Many developers grumble about MultiFinder and the numerous kludges it contains in order to be compatible with existing programs; the fact is, however, that MF is a triumph of programming precisely because it succeeds in that compatibility with programs written when it didn’t exist. I am not convinced that a backwards-compatible approach is even possible.

2) They are using several programs together on a single machine, rather than across a network. This simplifies communications greatly.

3) Documents will tend to be used together on the same machine, most often the machine where they were created. This reduces the probability of “collisions” between documents created on different machines but being processed together. We can thus use a simpler method of identifying documents so that they can “recognize” each other.

4) Enough information must be stored with documents to allow links to be reestablished as automatically as possible when stored documents are reopened.

5) The user interface should seem to be a natural extension of the existing interface, needing minimal learning and not intruding into the normal operation of the programs using it. Larry Atkin at Odesta suggested presenting it as an extended clipboard, hence our “magic clipboard” metaphor.

Basic Concepts: Given 2 programs running together under MultiFinder, they are linked if changing a specified chunk of data in document ‘A’ (the source) results in the changed data being used in document ‘B’ (the target) without any explicit intervention by the user. We say that the target document is dependent on the source document. This idea is central to our entire design.

Notice that while we assumed 2 programs running under MultiFinder, we spoke of documents being dependent on each other. This is due to the ability of most programs to manage multiple documents; the programs carry out the communication, but the documents are related.

There are two mechanisms needed for programs to be able to communicate automatically under MultiFinder (or any multitasking system, for that matter):

1) A method for a “ target” program to determine that a chunk of data it is interested in has changed, so that it can retrieve that data.

2) A method of transmitting chunks of data between programs.

Our protocol addresses both of those problems in a simple fashion. Let me emphasize that there are many ways of addressing them--the designer’s choice of methods depends on the needs of her/his users. Apple’s final approach will probably be quite different from ours (but who knows)?

Keeping track of changed data: This is the feature of the IAC protocol that allows data to flow between programs without user intervention. Since there are 3 parties involved with the process of getting data between programs (source, driver, and target), there are 3 places where knowledge of the relationship between programs can be kept. Having the IAC driver keep track of which target documents are interested in a given chunk of data (called an “extent” in the protocol description) is the simplest solution:

1) Source programs simply publish the latest version of an extent when it has changed. They have no knowledge of who’s interested, thus keeping their lives simpler.

2) A target program can retrieve data from a number of source programs with calls to just the IAC driver, rather than having to have some way of communicating with each source program explicitly.

Since an extent of changed data must be stored by the IAC driver anyway, we have added a feature to deal with the possibility of data being updated before all target documents have retrieved it. This is called an “edition,” and starts with 1 when data is first written to the driver. If a new edition of an extent is written to the driver before all target documents have retrieved an existing version, the latest edition is linked at the end of a chain. Target documents, by convention, remember the last edition read and simply request the latest edition higher than that.

When a document makes a data extent available to other programs for the first time, it is issued a document ID by the IAC driver, which serves as part of an “address” in communicating later. A properly written IAC application will store the document ID for each extent it is a target of so that (in a later session) should that document be opened the links can be automatically re-established.

Communicating between programs: The IAC driver serves in some respects as a set of highly intelligent clipboards, or mailboxes. A source program actually writes a chunk of changed data to the driver, using normal write calls to the file system, where it is stored until all the target programs that are interested in that chunk have read it, using normal read calls to the file system. While temporarily storing the data in the driver does take space, our only alternative would be to manipulate the data in different application heaps. This would require too much knowledge about the unpublished guts of MultiFinder to be either stable or safe.

A major implication of using normal file system calls for communicating is that target programs must actively poll the IAC driver in order to determine if the data they need has changed. This can be done both in the foreground and the background. However, if the target application genuinely needs to ignore the outside world for a while, there is no penalty since succeeding editions of each data extent are maintained by the driver.

Since our driver has functions significantly different from those of a standard driver, it needs a different set of parameters. These are transmitted in the I/O buffer that is used in read & write calls; since control calls to drivers don’t use a buffer, control calls aren’t used.

Interfacing to the IAC driver: While the attached protocol description gives the details of how the driver works and how to work with it, there is no need for everybody to re-invent the interface. Therefore, in the sample test program that will accompany the actual driver, a set of interfaces in C is provided as a package that can be used “off the shelf.” The interfaces deal with the messy details of setting up parameter blocks, doing the calls, and checking operation statuses. Since most of the intelligence resides in the driver, they are straightforward.

Without further ado, here is the definition of the protocol and its associated routines:

A Protocol for Inter-Application Communications Under MultiFinder
-- Version 1.2 --
© by Frank Alviani and Paul Snively, June 1988
Permission granted to MacTutor for Publication
All Rights Reserved

This document is intended to define how programs may be written to support “hot links” between several programs executing en suite under MultiFinder, using a device driver to provide the communications facilities required. It is divided into the following sections:

1) Definition of the concepts implemented. In particular, “hot links” and “data dependency” are defined.

2) Definition of the facilities made available by the device driver, and how they are used by the “IAC-friendly” program.

3) Operation of the driver and its internal data structures.

4) Suggestions for user-interface implications of this protocol, and how they might be implemented.

5) A list of the data formats supported in this version.

It is our hope that these protocols will be adapted freely and further improve the power and sheer fun of the Macintosh interface.

Section 1--Definitions and Assumptions

• An “extent” is the amount of data involved in a “data dependency”. The source program must maintain the start and end of the extent in order to be able to detect when it has changed, so a change notice can be posted to all interested programs.

• A “data dependency” exists between a “source” document and a “target” document when a change in an extent manipulated by the source program requires the data manipulated by the target program to change to correspond. In this protocol, the source program uses the device driver to make the changed data available to all interested target programs, and to post notification that the change has occurred. Target programs, in this protocol, are required to periodically poll the device driver to determine the latest state of the “foreign” data they are interested in.

It is vital to note that the basic objects dealt with are links between documents. This is required because most well-designed programs can manipulate several documents simultaneously. However, we often refer to source and target programs, since those are the machinery behind those documents; this is done when we are dealing with actually reading and writing the changing data.

Data dependencies involve several components. The complete set of data required must include:

1) The data itself, referred to as an extent.. The source program must maintain some way to remember its range - a block of cells in a spreadsheet, or a selection in a text document - so that it can be communicated after a change has occurred.

2) When a dependency is established by a source program, the IAC driver must have a way of identifying it later. This identification is in 2 parts:

A) It issues a document ID. This is the clock time in seconds, assigned by the IAC driver when a dependency is first registered. We work under the assumption that it is impossible to complete all the actions required to establish and complete a dependency in less than 1 second and register a second dependency. Just in case someone is using Tempo, the driver can maintain a record of the last document ID issued and if the next one would be redundant, add 1 to maintain uniqueness.

B) It issues a “hatcheck” for the extent - an index of the extents registered so far for that document. Extents are always identified by the combination of document ID and hatchecks, by both source and target programs. Hatchecks are unique and associated with a specific extent and document.

3) The identity of all target documents. A data source can be linked to up to n other documents. As a part of the updating process it is necessary to have some way for target programs to indicate that they’ve gotten the newest version of the data. The “slot_ID” is used for this purpose; it is an index into the IAC driver’s internal table of registered documents.

4) An “edition” indicator for the data. When a piece of data is linked to a target document, it is given level 0; the 1st change results in level 1, etc. When several links exist between a source and a target document, the edition associated with each extent allows the target program to identify and “grab” just the data units it requires to be “up to date.” When an existing document is opened all existing extents are assigned edition 0 to keep the bookkeeping manageable.

5) A format indicator. This serves the same purpose as it does for the clipboard. If a target program is unable to interpret any of the formats provided by the source program the dependency cannot be established and the user must be informed.

• A “hot link” is the user-interface for these data dependencies. It is the visual means of:

a) Specifying the range of source data to be echoed from the source program to the target program.

b) Indicating the location in the target document where the source data is to be included and shown.

• “External link definition” is the information stored in both the source and target documents that allows the data dependency to be re-established after they have been closed and re-opened and are being manipulated together.

Assumptions: The design laid out in this document is based on the following assumptions:

1) This is aimed primarily at individual users, or people working in small groups; this provides a simpler environment to plan for. In particular, this allows the use of fairly simple external link information.

2) Manual intervention by the user at any point should be absolutely minimal and transparent.

3) There is no need for “system administration” operations, and therefore are no provisions for them.

4) Programs need to be written to be “IAC” friendly; this is not a facility designed to be retrofitted to existing works.

Section 2 - Device Driver Facilities and their Usage

A) The device driver will be opened by an INIT31 resource at boot-up. It will remain open at all times. It will be named “.IAC.” User programs gain access to it by opening it (which is harmless) in order to get an ioRefNum for further access. While we are less than thrilled at forcing the user programs to open a driver with a specific and fixed name (it seems to be a rather “brittle” approach) there doesn’t seem to be a better way.

B) Since the normal device driver calls do not have provisions for extensions to the standard parameter blocks, the necessary additional information will be passed at the beginning of the buffers that are part of the driver calls. That is, the buffers will consist of an “IAC parameter block” followed by one or more data blocks.

C) The basic operations supported by the driver are the following:

1) Establish Dependency. This is a 2-part process that involves both the source and the target programs. After the user has determined the data of interest, the source program tells the driver to “add this extent to the dependencies table,” at which time it becomes the “available dependency.” After the user has determined where the source extent is to be linked into the target document, the target program tells the driver “I am interested in the available dependency.” Note that a benefit of the above approach is that several target programs can each indicate they are interested in the same source extent without having to re-establish it each time. A variation of this occurs when an open target document determines that it can re-establish an existing link with a source program, in which case the target program will specify the document and extent it is interested in.

2) Remove Dependency. This can be done by either the source or the target program. Target programs normally sever their dependencies when closing a document. If all target programs sever a dependency, the source program is notified so it can avoid the overhead of posting future changes; this notification occurs in the ioResult field of the next “UpdateData” or “Check status” calls.

3) Identify a dependency as the “available dependency.” This is implemented to simplify the process of linking more than one target to a single source extent.

4) Census. This can be called to get a list of all registered extents, so that programs can determine if external links have changed. It would normally be called if the latest “check status” call showed that the number of registered documents had changed since the previous status call.

5) Check status. This call is normally made by all programs during their “null event” processing, in order to determine what can be done. If several links are being maintained, this can save the effort of making a “ReadData” call for each extent. The user program should be very careful to make any needed calls in response to the IAC status before using any of the event-manager calls that could result in your program being swapped out; this will ensure the consistency of the environment.

6) Update data. The source program “writes” an extent to the IAC driver (identifying the extent, of course). The driver will assign it the next edition for that extent. It will be placed into a queue if any previous editions for that extent remain “unvisited” by any interested target programs.

7) Read Data. This is of the form “I am interested in extent ‘x.’ If it is more recent than edition ‘y’ please give me the new data.” If the indicated extent is not newer 0 bytes of data will be returned.

D) It is important to note that the source program must copy the extent to the IAC driver, and the target program must copy it from the driver. This approach was chosen deliberately, in spite of the seeming overhead, for the following reasons:

1) It avoids the potentially severe memory-management problems associated with multiple heap-zones. Frankly, we don’t know all the possible evil ramifications of trying to resize a relocatable block in another zone (for example) and don’t care to find out the hard way.

2) The target document is guaranteed to have data available at all times, even when it is re-opened without the source document. Having a graph vanish from a text document because you forgot to open the spreadsheet the graphing program was dependent upon would be justly perceived as a Bad Thing.

E) The driver will not permit dependencies to be defined in the following situations:

1) In both directions between 1 extent in each of 2 documents. This is to avoid an infinite-loop of updates.

2) Within the same document.

External Link Information: This is crucial in allowing documents to re-establish links after they have been closed and are being reopened, without manual intervention. The basic principal is that programs must retain identifying information with documents that allows a “target” document to recognize its “source” documents; this information is given to the IAC driver when extents are first registered.

While the document_ID/hatCheck combination is not perfect (primarily in a larger environment where several documents could be initially registered simultaneously to within 1 second), it seems workable as the external link information and has the virtue of simplicity. While “collisions” are possible, we assume related documents would be exchanged together and therefore processed together, maintaining the original context.

The syntax of the IAC driver calls: These will be described in terms of the “standard” calls made to the device driver and the fields in the parameter block that starts the I/O buffer. All calls return a value in the ioResult field of the standard parameter block, which should be checked by the user program. Since the control message to I/O drivers makes no provisions for buffers, we need to use just the read/write calls.

In the parameter blocks below, links are identified by both “slot_ID” (for the program) and “hatCheck” (for the document/link). While the use of 2 fields may seem slightly redundant, it seems easiest to work this way, since there are occasions when we need identification of just the program.

1) AddDependency: via FSWrite()

 block: short  func = 1;  /* IN */
 long docID;/* IN/OUT */
 short  slot_ID; /* OUT */
 short  hatCheck;/* IN/OUT */
 short  edition; /* OUT */

docID is needed to identify the source document. If docID is zero, the source program has never registered this document with the IAC driver before; the assigned docID will be returned. docID should be stored with the document. As it identifies when the dependency is first registered (in seconds), it is extremely likely to be unique. docID should be 0 only for the first extent being registered for a document; all succeeding extents must have a valid docID value in order to be associated with the correct document.

slot_ID identifies the document during the course of the current session. It will be assigned by the IAC driver if 0 (zero) on input. If it is known (because the document is the target for 1 or more dependencies) it should be filled in to keep the internal tables of the IAC driver consistent.

hatCheck is the opaque identifier assigned to this extent. If this field is 0 (zero) on input, a value will be assigned by the IAC driver. When a valid docID and hatCheck are provided the driver assumes an existing extent is being registered by the source program. It must be used in all future communications regarding it, by all programs interested in it, both source and target(s).

edition will be returned as 0 (zero) in this version.

It will be necessary to use the “WriteData” call after the dependency is started to actually post the data to the driver.

2) CompleteDependency: via FSWrite()

 block: short  func = 2;  /* IN */
 long docID;/* IN/OUT */
 short  slot_ID; /* IN/OUT */
 short  hatCheck;/* IN/OUT */

docID is used to identify the source document. If it is 0 (zero) on input the IAC driver will assume the “available dependency” (this is a “defaulted” call); the hatCheck field will be ignored in that case. If it is non-zero both docID and hatCheck must match a registered extent.

slot_ID is used by the IAC driver to know what documents are targets for a dependency in order to maintain the “interest mask” (described below). The source document for the dependency, however, does not need to know and is never informed of the targets. As explained above, it should be filled in if known; if 0 (zero) the value will be assigned by the driver and must be provided in all future calls .

hatCheck is returned to identify the extent involved in the dependency (it is identical to the identifier issued by AddDependency to the source program), and must be used in all future communications with the driver.

The “ReadData” call must be made to actually retrieve the data for the newly established link.

3) RemoveDependency: via FSWrite()

 block: short  func = 3;  /* IN */
 long docID;/* IN */
 short  slot_ID; /* IN */
 short  hatCheck;/* IN */

docID and hatCheck are used to identify the extent and source document involved.

slot_ID is used to identify who is terminating their interest in the dependency.

If the source program is involved, the data for all editions of the extent are immediately purged. The edition will be set to -1 so that all interested target programs will be informed the next time they execute a “Status” or “ReadData” call. As target programs attempt to retrieve the latest version of the data they will be informed that the source has terminated the dependency, and the target’s bit will be turned off in the interest_mask in the extent entry. When no interested targets remain, the extent entry is purged. If, instead, all target programs terminate their dependency on an extent, the source program is informed so that it can stop tracking changes in that extent.

4) AvailableDependency: via FSWrite()

 block: short  func = 4;  /* IN */
 long docID;/* IN */
 short  hatCheck;/* IN */

This sets the indicated extent as the “available dependency” that will be the source to future defaulted “CompleteDependency” calls.

5) Status: via FSRead()

 block: short  func = 5;  /* IN */
 short  slot_ID; /* IN */
 short  vers_ID; /* OUT - driver version */
 short  doc_count; /* OUT - # active */
 short  extent_count;/* OUT - relevant */

slot_ID is passed in to identify the document that is making the inquiry.

vers_ID is always returned. It is an integer with an implicit scaling of 100; i.e. version 1.0 is returned as 100, version 1.01 as 101, etc.

doc_count is the number of documents currently registered.

extent_count is the number of extents relevant to the document.

6) Census: via FSRead()

 block: short  func = 6;  /* IN */
 short  extent_count;/* OUT */
 /* for each extent: */
 long docID;/* OUT */
 short  hatCheck;/* OUT */

extent_count is the total number of entries in the census.

each census entry consists of the current values for the docID and hatCheck fields.

7) WriteData: via FSWrite()

 block: short  func = 7;  /* IN */
 long docID;/* IN */
 short  hatCheck;/* IN */
 short  edition; /* OUT */
 short  format_count;/* IN - # of formats */
 handle the_data;/* IN - handle to data */
 /* In the block whose handle is passed, the following 
 * is repeated once per format supported by the user 
 * program
 */
 short  format_code; /* format of this data */
 long data_size; /* size of this data */
    the actual data   

docID and hatCheck identifies the extent to be updated.

edition is returned for the use of the user program. It is optional whether it is retained with the document, but may be useful somehow.

format_count is the number of copies of the data that are being written in this updating of the extent. It can vary from update to update. For each copy of the data included in the block whose handle is passed in the buffer you must precede the data with:

format_code, as shown in the list later in this document.

data_size must be rounded up to an even number of bytes.

Now comes the actual data.

8) ReadData: via FSRead()

 block: short  func = 8;  /* IN */
 long docID;/* IN */
 short  slot_ID; /* IN */
 short  hatCheck;/* IN */
 short  edition; /* IN-OUT */
 short  format_pref[3]; /* IN */
 /* this is an array of preferred formats */
 short  format_code; /* OUT */
 handle the_data;/* IN */

docID and hatCheck specifies the extent and source document.

slot_ID specifies the target document that is interested in it.

edition is set by the user to the minimum change-level he is interested in (normally 1 more than the last edition it accessed); on output it contains the actual edition returned. The driver can (and probably will) skip any intervening editions, and mark them as “read.” The user program should remember this value, but it need not be preserved when the program quits, since all extents start at level 0 when an existing document is opened.

format_pref is an array that specifies up to 3 data formats the user program can accept, in order of desirability. A null entry ends the list if there are less than 3 entries.

format_code will specify the actual format returned.

the_data is a handle to a block in which the data will be placed. It can be allocated to 0 bytes since the driver will resize the block as needed.

Section 3 - Operation of the IAC driver

• The driver will maintain a list of all currently-registered documents. This is a dynamic list. A fixed array of n (numbered 0 to n-1) entries is maintained. When a document terminates, it’s slot is made available - the array is not compacted. The slot index is also the bit index in the “who’s interested” word maintained with each extent.

When an extent is registered or linked to an additional document, the bits for each interested target document are turned on in the “interest mask”. A copy of this is issued to each change-level when it is accepted by the driver. As a target program copies that change-level, its bit is turned off in the mask; when the mask hits 0, all interested parties have accessed it and it is discarded.

• The entry for an extent (not an edition!) is structured as follows:

handle  next_extent; /* link */
long    document_ID;
short hatCheck;
long    master_interest_mask;
handle  head_of_edition_list;

• The entry for an edition is structured as follows:

handle  link_to_next_CL;
long    interest_mask;
short   number_formats_stored;
struct  { /* 1 entry/format */
 short  format_code;
 handle data_handle;
};

• The process of determining the dependencies that are of interest to a given document (in response to a Check_status call) is quite simple. The slot_ID of the inquirer specifies the bit to check in the interest mask for each active extent/edition, so it involves merely scanning every active extent/edition and remembering which ones have that bit set.

Section 4 - User Interface Suggestions

1) The most natural metaphor for the user is that of the “magic clipboard” that handles all the details of cutting and pasting for him/her automatically. In line with this metaphor, the establish-dependency item on the Edit Menu could well be called “Hot Copy,” with the complete-dependency item called “Hot Paste.” (Thanks to Larry Atkin for the suggested names.)

2) There should be a mechanism for identifying (displaying) active dependencies in a document. This can be implemented by some alternative highlighting of the extent, for example.

3) Clicking (double-clicking?) on a displayed dependency should cause the program (if it is the source program) to send a message to the IAC driver identifying that extent as the “available dependency” so it can be simply linked to a target document. This approach avoids problems associated with trying to link the same source extent to several documents (such as accidentally changing the range of data involved for different links). It may be desirable to have some slightly different highlighting (such as a darker stipple) in order to identify the “available dependency.”

4) It is usually (but not always) desirable to post an update for an extent only when the changes are done and the data will be stable for a reasonable amount of time. In a text selection, for example, this means that you shouldn’t post a data change for every character typed, but wait until the insertion point is placed outside the data range. There are times when every single change--no matter how small--should be posted immediately, so this is a somewhat elastic guideline.

5) In data structures that can be expanded & contracted (such as a text selection) the definition of the extent should be modified as needed as changes are made. In a text selection, for example, if some characters were inserted into the middle the original starting and ending characters would still be the start and end.

Section 5 - Data Formats Supported

1) Plain text.

2) Formatted text (a TE scrap record with style information).

3) PICT. Probably should include PICT2.

4) Spreadsheet range.

This is what we have designed. It will probably take some time for these concepts to soak in, which is just as well, since the driver (as of this writing) is still in the process of being implemented. Next month we will publish the commented MPW assembler source code of the driver, along with support stuff (like Rez source for its resources, an accompanying article, etc.) and our first sample application that supports the driver. Hopefully this has whetted your appetite somewhat, and, like us you can’t wait to see a spreadsheet, graph program, and word processor, all running under MultiFinder, and to make a change to the spreadsheet, which then causes a change in the graph, which then causes a change in the report in the word processor. Until then, try to absorb this hodge-podge of information that Frank has put together and I have tried (ahem) to clarify and verify in terms of accuracy. Happy hacking!