Newton Communication Programming

The basics

By Kent Sandvik

About the author

Kent Sandvik is working at PIE Developer Technical Support, Apple, as a Technical Lead. This means less programming and more meeting attendance. You can try to reach him on Internet, CompuServe or AppleLink, but he may be in Nepal for a few weeks on his sabbatical

Introduction

This article is an introduction to communication application development on the Newton platform. Some of you maybe have read the Communication chapter in the Newton Programmer’s Guide, and also looked at the various samples and the Q&A material (posted on AppleLink and Internet). This article will complement the existing material, and the intent is to be more practical and provide an introduction into writing communication applications. The end result is (hopefully) that you could implement communication aware applications on the Newton platform within a short time frame and with no headache.

Newton provides a high level interface to the built-in communication facilities (serial port, modem, Infrared and ADSP) by the way of routing. When the end user selects a specific action from the action button, the system will place the contents into a specific outbox, and later send and receive information from and to the inbox and outbox. In this case we are talking about store-and-forward communication support.

In this article we will dive further down into the guts of Newton communication programming. In other words we want to implement point-to-point connections with other Newtons or external services, and write applications that talk with various application level protocols. Examples of such cases could be various ADSP servers, on-line services, mail systems, relational databases and other interesting entities on the future information highway.

In this article we will mainly discuss the basics of the communication system, what various buzzwords mean, how to implement basic communication applications using the serial port and a serial cable and how to write protocol implementations.

Communications, Routing

Testing

In many cases we do testing of communication applications using a Newton, an external service, modems, cables, if possible also using line analyzers for debugging purposes (but these are fairly expensive). If you want to configure a fairly cheap environment, I would suggest to purchase a Macintosh terminal program that is Communication Toolbox aware. This means that you could test out various protocols and services using existing CTB tools (modem, MNP5, ADSP, plain serial ).

Oh yes, what about the cable? We are dealing with so called null modem cables. In the case of a Macintosh connection, use M0197, part number 590-0552-A. [I simply used the cable that came with a StyleWriter - Ed stb]

Endpoints

So, what are endpoints? The new X/OPEN standard by the name of XTI specifies a new communication entity called an endpoint. The idea with the endpoint is that it encapsulates the behavior and attributes associated with a particular connection. Think about it as a black box, from the outside you have a specified interface that works for any kind of endpoint. However the endpoint itself implements various protocols so two ends could talk with each other, either an endpoint talks with another similar endpoint, or a with a service that talks the same protocol lingo.

Endpoints

All this means that it’s fairly easy for an application developer to understand the endpoint APIs - they all look the same with some minor variations.

In practice you create an endpoint, initiate it, specify the options needed for various communication protocols (speed, transfer rates and so on), send a Connect message to the endpoint, read and write data from and to the endpoint, and finally disconnect and dispose the endpoint. The endpoints also have more functions that we won’t talk about in this introduction article.

So, how is this done in the Newton programming world? Here’s an example of an endpoint frame definition:

/* 1 */
protoSerialEndpoint := {
// basic endpoint proto
 __proto: protoEndpoint,

// options (array stored inside the configOptions slot)
 configOptions: [
 // basic serial port
 { label: kCMSAsyncSerial, 
 type: 'service,
 opCode: opSetRequired },
 
 // basic serial port options
        { label: kCMOSerialIOParms, 
 type: 'option,
 opCode: opSetNegotiate,
 data: { bps: k9600bps, dataBits: k8DataBits,
 stopBits: k1StopBits, parity: kNoParity } },
 // flow control (XON/XOFF)
        { label: kCMOInputFlowControlParms, 
 type: 'option, 
 opCode: opSetNegotiate,
        data: { xonChar: unicodeDC1, 
  xoffChar: unicodeDC3,
  useSoftFlowControl: true, 
         useHardFlowControl: nil 
 } 
 },
   ],
});

First we specify that we want to inherit the basic protoEndpoint behavior by stating in the proto slot that we want to proto from the protoEndpoint (this contains the basic behavior for all endpoints).

After this we want to spice the endpoint by defining what options we want to make this a serial endpoint. We want to specify the speed, parity bit settings, stop bits and flow control.

All options are stored in an internal slot called configOptions. The option settings are valid before we create the actual instance of the endpoint specified. This means that the settings should be set before or during the Instantiate method call. After that the endpoint will happily continue with the earlier defined settings, and we need to do some magic to re-configure the endpoint.

Looking more closely at the complicated option setting frame, we notice that we have label slots, type slots, opCode slots and dataSlots. The label slot is the universal option identifier, so we know that for instance we are dealing with an async serial option setting. The type slot will contain the type of option, are we dealing with service settings, or option settings? Service settings identify the actual communication service. In this case we want an asynchronous serial service, so we specify the constant kCMSAsyncSerial. You could use this slot for specifying IR services, or ADSP services, or any other comms services.

Opcode slots identify how the service should deal with the actual option request. We could for instance honor the request (opSetRequired) but if the option is unbelievable or not available (let’s say a speed of 3 billion bps) the option setting will fail. If we specify opSetNegotiate the service might substitute the 3 billion bps to 14.2kbps as a more reasonable value.

The data slot is used for further specifications of the options, such as actual speed rates, XON/XOFF characters and so on.

If we dissect the kCMOSerialIOParms option, we would see that:

/* 2 */
 // basic serial port options
        { label: kCMOSerialIOParms, 
 type: 'option,
 opCode: opSetNegotiate,
        data: {  bps: k9600bps, 
 dataBits: k8DataBits,
 stopBits: k1StopBits,
 parity: kNoParity 
 } 
 },

We are indeed dealing with a service option ('option), and that we want to have 9600bps, 8 data bits, 1 stop bit, and no parity (the data slot constant values). We also want to negotiate and the service could substitute some of the values to something else if it’s not available. If this happens we need to use the GetOptions method to later see how the endpoint is really configured.

State Machines

Another important concept to understand when writing Newton communication-aware applications is the notion of state machines. Reading my Dragon Book, state machines or state transition diagrams are used for defining actions that take place when a certain state has been triggered.

Here’s a simplified example showing a state diagram of when I’m logging into a UNIX system, and I’m trying to see if I could finger a user named sandvik on the apple.com system.

From a pure textual point of view, here’s the session:

4.3 BSD UNIX (apple.com)

login: sandvik
Password:
Last login: Sat Mar 19 12:32:59 from 17.205.4.47
4.3 BSD UNIX #44: Mon Feb 28 15:36:01 PST 1994

>>> Apple.com will be down on Saturday, 3/26/94 from 
>>> noon until 9pm for a full monthly backup.
Sat Mar 19 12:37:51 PST 1994

apple.com: finger sandvik
Login name: sandvik
In real life: Kent Sandvik
Office: M/S 75-3T, 974-8355
Directory: /u/ind2/sandvik              Shell: /bin/csh
On since Mar 19 12:37:46 on ttyq4 from 17.205.4.47
Project: DTS hacker/slacker
Plan: To reach inner Dolpho in Northern Nepal in less than three weeks, 
late April 1994.
apple.com: logout

If we now conceptualize this as a set of transaction diagrams, this would look like:

So what we need is a definition on what we will trigger upon, what’s the next state, what is the starting state, and what should we do as part of the state.

All this we will define in something we call an input specification. It maps the state information into a frame with slots that will keep track of the specifications. When the specifications are true, the input spec will trigger, code will run, you will do something inside the specification - like moving data or sending information - and then you will let the communication sub-system work along until the next specification is triggered. This is passive programming; the system will trigger your specification frames when needed.

Input Specifications

The input specification is defined inside a frame. In this frame we have slots that could define what we are triggering upon. Examples.

byteCount: we could trigger after we have received N amount of bytes. This is OK for very static protocols where we know that we will indeed receive for instance 200 bytes and that’s the block we want to handle.

endCharacter: This is the last character in a string return block. If we want to terminate and trigger based on a text dump ending with carriage return or new line, we could specify this here.

inputForm: Here we define what we are chasing after, strings or raw bytes, integers or even Newton frames. Note that when we are chasing string input, by default we are indeed dealing with Unicode strings, not ASCII-8 bit strings. This means in practice that we really need to use the built-in Newton string functions that understand Unicode character values.

discardAfter: The system is actually reading in all the information into a temporary buffer, and we could specify the size of this buffer so the system could flush the temporary data out if it has not triggered on any of the input.

Now, we also want to get some work done. This is done by specifying a frame called inputScript with code that will trigger when the state is active. As we are down in the guts of an endpoint, usually not attached to the application base view inheritance system, we have to think twice when we want to call methods outside out small endpoint inheritance world. This is the reason we set the _parent slot in the example later. This way we got the parent inheritance chain attached to our endpoint, or input specification.

Here’s finally an example of a simple input specification; in this case we are triggering based on the login: string from the UNIX system:

/* 3 */
LoginState : {
 inputForm: 'string, // we are dealing with strings
 endCharacter: unicodeCR,
 discardAfter: 200,// 200 bytes, new scan
 InputScript: func(endpoint, string)
 begin
 if(StrPos(s,"ogin:" 0)) then
 // if we really catched [Ll]ogin:  
 begin
 endpoint:Output("sandvik", nil);
 endpoint:SetInputSpec
 (endpoint.waitForPassword);
 end;
 end
},

OK, more interesting functions here. As we could not specify the terminating string - just the character - we have to double-check the incoming string information inside the inputScript. The InputScript will get a reference to the actual endpoint as well as the whole string that we parsed in (based on an ending carriage return that would trigger this input specification). So we just check the string, if it indeed contains an ending "ogin:" string, then we send out our login name as a string using the Output method, and set the next input specification. The next specification will trigger based on a carriage return, and in that InputScript we will chase for any strings that have the magical word "Password:" included.

More About State Machines

As you have seen we are working with Command Control Language specifications (CCL). The Newton communication API was explicitly designed for such purposes. This because we want developers to write smart interface application interfaces to existing and new services. If you want to write communication code that deals with the lower ends of the OSI suite of protocols (transport, session, link layer and the rest) you need to have access to the forthcoming communication device driver development kit.

In practical work, you either try to learn the protocol for the service, or then you maybe are even designing one. In the provided sample I have specified a proprietary protocol named Llama Application Modularity Association Protocol, or short-named LLAMA-P. What it does is to wait for ACK from the other end, and if this happens, it will respond to various client-generated requests based on a set of string commands where all of them surprisingly end with a ?character. See the example source dump for more details.

So where do we write these input specifications? Usually they end up inside the frame where the actual endpoint is defined. In our case we defined the whole endpoint including the input specs inside the NTK Project Data file. This means that you could just copy the text and included it into your own comms projects. The current NTK only provides browser capabilities for frames that are associated with a view element. This means that you can’t just now create communication frames using NTK browsers, unless you want to include the endpoint as part of a view, of course.

Asynchronous Versus Synchronous Communication

This discussion leads to the argument why this is the case with the Newton platform? Why using state machines and input specifications? Well, we are really dealing with battery driven devices. It does not make sense to poll the line for incoming data, and let the CPU spin around and around, and around, eating up the batteries. We strongly suggest to use the model of fully asynchronous programming, defining input specs for triggering, and meanwhile the system might sleep or otherwise conserve the energy resources. In other words polling for a change is not recommended at all, and any of the few synchronous APIs we provided in the first release of the Newton ROMs are already undocumented in the current documentation suite.

Instantiate, Connect, Disconnect

So now we have our endpoint fully configured. We need to create an instance of this template (remember, using NTK we specify just a template), and we need to create a real endpoint for the actual work. In our viewSetupDoneScript we have the following lines of code:

/* 4 */
  viewSetupDoneScript:
      func()
      begin
 // Create an endpoint for our use, protoSerialEndpoint is defined inside 
the
 // Project Data file.
      
        ep := {_proto: protoSerialEndpoint, _parent: self};
        // We added the parent slot for parent inheritance  inside the 
endpoint frame.
      end,

After this we want to initialize the endpoint just before use (to preserve memory use, initialization will build the actual underlying endpoint), and do a connect. We will add the following code to a button, so when the button is triggered we will issue the code:

/* 5 */
// Make the initial connection and start listening/sending.
func()
begin
 local anErr;

// Initialize the endpoint.
 anErr := ep:Instantiate(ep, nil); 
 if anErr then
 begin
 :Notify(kNotifyAlert,EnsureInternal("SerialEx"), 
 EnsureInternal("Sorry, serial port in use!"));
 return;
 end;

// Do a Connect.
 anErr := ep:Connect(nil,nil);
 if anErr then
 begin
 :Notify(kNotifyAlert,EnsureInternal("SerialEx"),
 EnsureInternal(
 "Couldn't connect, check cable and other end!"));
 return;
 end;

// Set the Next Input Specification.
   ep:SetInputSpec(ep.waitForACK);
end,

We initialize the endpoint, check for any non-nil return values, if so we have problems. Similarly we do a connect. In this case when we are dealing with a serial connection, we don’t need to worry about address information that is passed to Connect, so we just specify nil. It is important to check for any return values from these calls, if they return non-nil, the connection is not properly constructed.

After this we specify the first input specification, and after this the underlying communication system will trigger whenever the next designated specification is true.

When we want to disconnect and disable the endpoint, we use the following cryptic code (that will be explained):

/* 6 */
 ep:FlushInput ();
 ep:FlushOutput ();
   endpoint.nextInputSpec := nil;
 endpoint:SetInputSpec(nil);
 ep:Abort();
 AddDelayedAction(func() 
   // let the endpoint clean up from the abort before disconnecting and 
disposing
 begin
   ep:Disconnect();
 ep:Dispose();
      end,
      [], 1000);

We want to flush out any pending output or input, after this we will terminate the input specification chain, first by stating that we don’t have a next input specification, then that we neither have a current one!

After this we will issue the Abort message that will abort any possible pending I/O in progress.

Now, we are dealing with an environment where the endpoint’s state frames are actually triggered from a work queue. We can’t assume that the endpoint is in a known state because we are currently operating in a specific part of the NewtonScript code thread, and the operations (flush, abort) might happen at a later stage when the queued endpoint jobs have finished. This all means that we have to continue after a certain time with disconnection and disposing of the endpoint memory. We use a heuristic time delay of 1 second that seems to work fine in most cases. By writing a delayed action code block we will disconnect the connection and dispose the endpoint memory.

Sending And Receiving Data

See the source code listing concerning these operations. We are using the Output call for sending string information, and we receive strings when an input specification is triggered. Some of you might wonder why we don’t use the Listen method? Well, it’s used, but for specific purposes, for instance if you want to wait for incoming connection requests and accept these. Currently this is only doable with the modem and IR endpoints.

Some of the brave ones might also try to use the Input method that will force the current input specification to terminate, and get the contents of the incoming buffer. You could use Input if you don’t know what is really going on, and you want to look at the incoming data.

Performance

Here’s lesson number one concerning Newton communication programming:

Don’t spend too much time inside the InputScript.

The other end (server) might not like this, it could time out. Or then you could encounter buffer overflow problems because you spend your time drawing nice pictures based on the incoming data, and meanwhile the PowerPC server is spitting out more data over the 19.2kps serial line, and XON/XOFF is not enough.

So when you receive data, put it aside, move it to a soup for instance, and later when the system has time, for instance inside a viewIdleScript, or even better, after the comms session spend time handling the data. If you move data from your incoming buffer to a soup you are guaranteed that the underlying heaps including the NewtonScript heap will not run out of space.

Lesson number two is:

You don’t need to use inheritance for calling other methods

This sounds like heresy for OOP programmers, but in some cases using the inheritance chains of NewtonScript could cause delays in calling various methods. Remember that you could easily set up references by simple assignment operations in the Newton Toolbox, so you could just assign a slot in your endpoint frame with an interesting other frame that has methods you will need to call from inside your InputScripts. This could be done when the actual endpoint is created during runtime.

Serial Port

The current MP and MP110 systems have the same dear Zilog serial chip that most of us Macintosh programmers have learned to love. In other words you have the same functionality and limitations. As in the case of the Macintosh world, don’t assume too much about the underlying hardware. Actually, the more abstractions you build into your application, the easier it is for you to substitute the serial endpoint with a wireless endpoint, for instance.

Debugging

Well, we have the inspector window open for checking out various print statements and for introspection of the application. And we also need the serial port for the serial application? Yes, this is not fun, as many Newton programmers have noticed. The current solutions are based on printing into an array in the application base view, or a soup, or otherwise store state information somewhere else, maybe even implement a simple text view that shows the current state and possible error messages.

Even if we had access to the Inspector window, it is nearly impossible to print to the inspector while an input specification construct is running, this because the interrupts from handling the Inspector window would kill the concurrent communication session.

In our case we are using a simple protoStaticText view where we display state information, this way using a multitude of print statements we know roughly where we are in the code if something fails.

Listing Of Comms Sample

/* 7 */
// © Copyright 1994 Apple Computer, Inc, All Rights Reserved

// Basic constants that we may or may not need
constant kAppName  := '|SerialEx:PIEDTS|;
constant kAppSymbol:= '|SerialEx:PIEDTS|;


// This is our protoEndpoint specification. Note we are using DefConst 
so the symbol is
// known as a constant when we compile the proto, and later use it during 
runtime.

DefConst('protoLlamaPEndpoint, {
// basic endpoint proto
 _proto: protoEndpoint,

// Options (array stored inside the configOptions slot.
 configOptions: [
 // basic serial port
 { label: kCMSAsyncSerial, 
 type: 'service, 
 opCode: opSetRequired 
 },

 // basic serial port options
 { label: kCMOSerialIOParms, 
 type: 'option, 
 opCode: opSetNegotiate,
        data: {  bps: k9600bps, 
 dataBits: k8DataBits, 
 stopBits: k1StopBits, 
   parity: kNoParity 
 } 
 },

 // flow control (XON/XOFF)
 { label: kCMOInputFlowControlParms, 
 type: 'option, 
 opCode: opSetNegotiate,
       data: { xonChar: unicodeDC1, 
 xoffChar: unicodeDC3, 
 useSoftFlowControl: true, 
        useHardFlowControl: nil 
 } 
 },
   ],

// Our exception handler (asynch exceptions from the endpoint).
 exceptionHandler: func(exception)
 begin
 print("Got the exception from Abort, "
 & "ignoring it for the time being");
 end,

// Our state machines:
// WaitForAck, main dispatcher, waits for the first ACK from the other 
side.
   waitForACK:
   {
 InputForm: 'string,
 endCharacter: $?,          // ACK? expected
   discardAfter: 200,// scan buffer size
       
 InputScript: func(endpoint, s)
   begin
   if (StrPos(s, "ACK?", 0)) then  
 // send response (help instructions in this case)
        begin
         endpoint:Output("Send any of the following commands:" 
 & unicodeCR
 & unicodeLF, nil);  
 endpoint:Output("PLAY! - will play a tone on the Newton"
 & unicodeCR
 & unicodeLF, nil);
 endpoint:Output("Add more comments if more functions...."
   & unicodeCR
 & unicodeLF, nil);
         endpoint:FlushOutput();
         endpoint:SetInputSpec(endpoint.waitForFUNCTION);
 // the main dispatch loop
        end
   end,
   },


 // This is the generic dispatcher state, send something ending with 
! and
 // the Newton will serve.
 waitForFUNCTION:
 {
 InputForm: 'string,
 endCharacter: $!, // expects a '!’ as part of the command
   discardAfter: 200,// scan buffer size

      InputScript: func(endpoint, s)
      begin
        if(StrPos(s, "PLAY!", 0)) then // play function
            begin
            PlaySound(ROM_funbeep);
               endpoint:Output(unicodeCR, nil);
               endpoint:Output(unicodeLF, nil);
               endpoint:FlushOutput();
            end;
       end;
 }, 
}
);


// -- End Project Data --



// -- File baseView.t --
baseView :=
   {title: "Serial Example 1.0",
    viewBounds: {left: 0, top: 2, right: 236, bottom: 334},
    viewFormat: 83951953,
    
    viewSetupFormScript:
      func()
      begin
 // Make sure the screen is adjusted to MP and MP110.
        constant kMaxAppWidth := 240 ; // original MP width
        constant kMaxAppHeight:= 336 ; // original MP height
      
        local b := GetAppParams() ;
      
        // Make view no bigger than the original MP.
        viewBounds := RelBounds(b.appAreaLeft, b.appAreaTop,
        MIN(b.appAreaWidth, kMaxAppWidth),
        MIN(b.appAreaHeight, kMaxAppHeight));
      end,
    ep: nil,
    DisposeEndpointScript:
 // Dispose our endpoint gracefully
      func()
      begin
        :DisplayInfo("About to flush...");
        ep:FlushInput ();
        ep:FlushOutput ();
      
        :DisplayInfo("Flush OK, now zap input specs");
 ep.nextInputSpec := nil;
        ep:SetInputSpec(nil);
      
        :DisplayInfo("InputSpec handling OK, now Abort");
        ep:Abort();
        :DisplayInfo("Abort OK, now delayed action later...");
      
        AddDelayedAction(func() begin
        // let the endpoint clean up from the abort before disconnecting 
and disposing
        ep:Disconnect();
        ep:Dispose();
        ep := nil;
            end,
            [], 1000);
      end,
    viewQuitScript:
      func()
      begin
        if ep <> nil then
        :DisposeEndpointScript();  // get rid of any active endpoints
      end,
    displayInfo:
      func(string)
      begin
        SetValue(infoView, 'text, string);
      end,
    _proto: protoApp,
    debug: "baseView"
   };

ConnectButton := /* child of baseView */
   {text: "Connect",
    buttonClickScript:
 // Make the initial connection and start listening/sending.
      func()
      begin
        local anErr := nil;
      
        if commState = 'Disconnected then    // create a new connection
        :ConnectScript();
      
        else if commState = 'Connected then  
 // we want to disconnect the connection
        begin
        :DisconnectScript();
        end;
      end,
    viewBounds: {left: 66, top: 218, right: 158, bottom: 238},
    commState:
 // this keeps track of our current communication state
      'Disconnected,
    ConnectScript:
      func()
      begin
 // Create an endpoint for our use, protoLlamaPEndpoint is 
        // defined inside the Project Data file.
        ep := {_proto: protoLlamaPEndpoint, _parent: self};
      
        // We added the parent slot for parent inheritance inside
        // the endpoint frame.
      
        if ep = nil then
        :Notify(kNotifyAlert, EnsureInternal("SerialEx"),
        EnsureInternal("Couldn’t create an endpoint in RAM"));
        else
        :DisplayInfo("Created an endpoint in RAM");
      
        :DisplayInfo("About to Instantiate");
        // Initialize the endpoint.
        anErr := ep:Instantiate(ep, nil);    
      
        if anErr then
        begin
        :Notify(kNotifyAlert, EnsureInternal("SerialEx"), 
        EnsureInternal("Sorry, serial port in use!"));
        return;
        end;
        else
        :DisplayInfo("Initialized the endpoint.");
      
 // Do a Connect.
        anErr := ep:Connect(nil,nil);
      
        if anErr then
        begin
        :Notify(kNotifyAlert, EnsureInternal("SerialEx"),
        EnsureInternal("Couldn’t connect, check cable and other end!"));
        return;
        end;
        else
        :DisplayInfo("Connected to the other side.");
        
 // Set the Next Input Specification.
        ep:SetInputSpec(ep.waitForACK);
      
        // We are using one single button, so change the text to disconnect
        SetValue(self, 'text, "Disconnect");
        commState := 'Connected;
      end,
    DisconnectScript:
      func()
      begin
        :DisposeEndpointScript();
      
        // We are using one single button, so change the text to Connect
        SetValue(self, 'text, "Connect");
        commState := 'Disconnected;
      end,
    _proto: protoTextButton,
    debug: "ConnectButton"
   };



infoView := /* child of baseView */
   {text: "Information View",
    viewBounds: {left: 16, top: 24, right: 224, bottom: 176},
    viewJustify: 0,
    viewFormat: 67109200,
    _proto: protoStaticText,
    debug: "infoView"
   };
// View infoView is declared to baseView



// -- Beginning of section for non used Layout files --

// End of output

Other Sources

• Newton Programmer's Guide 1.0, Communication Chapter

• Q&A Communications, available from AppleLink or Internet (ftp.apple.com)

• PIE DTS Communication samples, from AppleLink or Internet

Future Articles

We hope to follow-up with articles in which we will look more closely into details concerning modem connections, how to configure modems and how to use non-Apple Newton modems, how to scan single byte or character streams, how to create ADSP clients and how to make use of the IR protocol.