Scripting the Web with Frontier

Customizing your web server
with UserLand Frontier and the CGI Framework

By Mason Hale, hale@onr.com

Out of the box, web servers like Starnine’s WebStar don’t offer much in the way of user interaction. Adding interactive elements like clickable maps and fill-out forms requires CGI (Common Gateway Interface) applications running on the server. When Macintosh webmasters want to add some interactivity to their web sites, they usually consider two sources for the tools they need to customize their web servers: commercial and shareware CGIs like NetCloak and MapServe, or custom applications they write themselves using AppleScript.

Commercial packages are often PowerPC-native and multi-threaded to handle the multiple concurrent requests a busy web site is likely to generate. On the other hand, AppleScript offers the ability to create custom CGIs quickly and easily, but the resulting applications are not threaded and are much slower than CGIs written in a low-level language like C. Webmasters have often been forced to choose between the poor performance of AppleScript and the steep learning curve of C.

An excellent alternative that few have considered until recently is UserLand Frontier. Frontier was the first system-level scripting language for the Macintosh when it was released in January 1992. Though it earned respect from prepress houses and network managers, its hefty $495 suggested retail price and the free availability of AppleScript caused it to be overlooked by the Macintosh scripting mainstream.

The price/performance scales tipped strongly in Frontier’s favor this past May when Frontier co-author Dave Winer released a free Internet-savvy version of the software under the code-name Aretha. Aretha, also known as Frontier 4.0b1, is identical to the previous shipping version with the addition of some Internet-specific scripts and a change to the version resource. The software and on-line documentation can be downloaded from Dave Winer’s Aretha website (given at the end of this article).

The CGI Framework is a collection of scripts I’ve written that add a suite of CGI development tools to Frontier. These scripts are free and can be downloaded from the Frontier CGI Scripting site at http://www.webedge.com/frontier/.

In this article I am going to show you how to use Frontier and my CGI Framework to enhance your web server. I’m going to assume you’ve never used Frontier, but are familiar with what CGIs are and how they work.

A New Frontier

I discovered Frontier by accident. At the first WebEdge developer’s conference held last April in Austin, I won the hacking competition with a web-based miniature golf game written entirely in AppleScript. My hack used the mouse click coordinates to calculate the angle and power of the putt, then scripted Yves Piquet’s Clip2Gif utility to draw the individual frames of the ball moving around the hole. Those frames were displayed as an animation using Netscape’s then new server-push capabilities. It was an impressive piece of work, but it also highlighted the inherent weaknesses of using AppleScript for CGI development.

I didn’t have a practical way of storing ball coordinates after the putt. As a result, each player only got one putt, and if they missed, they had to start from the beginning. You got a hole-in-one or nothing. Since it wasn’t multi-threaded, it became unbearably slow and unreliable as more people tried to play it at the same time.

After the WebEdge conference I started looking for alternatives to AppleScript. By coincidence, I searched the HotWired website for the word “Macintosh” and found, among other things, Dave Winer’s essay titled “Platform is a Chinese Household”. It is a powerful essay that equates Apple’s developer relations to a lousy lover. I was very impressed, so I started rummaging around Dave Winer’s web space looking for other interesting reading. What I eventually found blew me away even more. It was an article Dave had written for MacTech magazine titled “A Nerdy Guide to Frontier” (MT 9.8). I couldn’t believe what I was reading; it was like a laundry list of all the features I had been looking for in a CGI development environment: fully multi-threaded, integrated Object Database, built-in debugger, full verb set, completely scriptable. Here was what appeared to be the ideal CGI development environment and I had never heard anything about it.

I sent an e-mail to Dave Winer asking why I had never heard about Frontier. His reply was essentially “we gave up”. I wasn’t easily discouraged. I keep peppering him with messages, urging him to consider the potential in the Macintosh web developer community. I must not have been the only one putting the Internet bug in Dave’s ear, because the following month Dave released Aretha.

Soon after the release of Aretha, I volunteered to take over development of the CGI Framework.

A Brief Introduction to UserTalk

In this section, I’ll give a brief overview of the UserTalk scripting language and the Object Database. This is not designed to be a comprehensive guide to scripting in Frontier, but I hope to give you enough information to get started. If you are already familiar with Frontier, you can skip ahead to the next section.

Creating a script in Frontier is sometimes frustrating for new users. When you launch Frontier you are presented with Frontier’s message window. If you click the flag on the right, it drops down to reveal four buttons: Menu Bar, Object DB, Quick Script and Tech Support. Click the Object DB button to open the root table of the Object Database. A new Table menu should appear in the menubar. To create a new script, choose New Script from the Table menu. A new script will be created in the currently open table.

Figure 1. Frontier’s Message Window

Frontier’s Script Editor uses a collapsing outline structure to organize scripts. Double-clicking the wedges before each line expands and collapses sections of the outline. The script editor also features built-in debugging tools that allow you to set breakpoints and step through the script line by line. When debugging, you can look up and change the values of variables by selecting the variable name in the script and clicking the Lookup button. Command-double-clicking the variable name has the same effect.

Figure 2. Frontier’s Script Editor and the New CGI Template

Frontier’s Script Editor is itself scriptable, making it possible to automate the editing of scripts. The New CGI command added to the Table menu by the CGI Framework is an example of automated scripting. The command creates a script in the webServerScripts table, and adds code that is common to most CGI scripts - including a commented list of available parameters.

An essential tool for anyone writing scripts in Frontier is DocServer. DocServer is an online reference to the UserTalk scripting language. Control-double-clicking on a verb name in Frontier will launch DocServer and look up the command in the DocServer dictionary. Control-option-double-clicking a verb will copy the syntax for the verb directly into the open window. See below for URLs to the DocServer software and website.

Figure 3: The root table of the Object Database

Tables, tables everywhere

To understand Frontier, you must understand the Object Database, Frontier’s built-in storage system. The Object Database supports over 20 built-in data types, ranging from Booleans, characters and numbers to scripts, outlines and word-processing text. Frontier also has a catch-all “binary” type that can contain any data type.

The UserTalk scripting language is tightly integrated with the Object Database. Verbs used in the scripting language actually exist as objects in the Object Database. This integration makes the scripting language incredibly extensible. Any script in the Object Database can be called by another script in the same way you would call a standard UserTalk verb. Every script becomes a new command in the scripting language. This design encourages modular scripts and reusable code.

Local variables are stored in a temporary space within the Object Database as well. When a script declares a variable x, an object named x is created in the temporary space. Every script occurrence gets its own variable space, so multi-threaded scripts don’t stomp on each other’s data. Variables can be any data type, including scripts, tables, outlines, word processing text, or even menubars.

Objects are stored in a hierarchy of tables and sub-tables, similar to folders and sub-folders in the Macintosh file system. Frontier uses a dot notation to describe the table hierarchy. For example, the address of the name object stored in the user table would be user.name. The address of the address object in the same table would be user.address. The readme object stored in the utilities sub-table of the webServer table would be addressed as webServer.utilities.readme.

UserTalk has a with statement to help you work with names of deeply nested objects. For example, the following script works with the values in the webServer.preferences table:

with suites.webserver.preferences
    « refer to suites.webserver.preferences.framework
 if framework   "3.0.1"
 dialog.notify ("This script requires version 3.0.1")
 return (false)
    « ditto for server and verboseLogging
 server = "WebSTAR 1.2 via Frontier"
 verboseLogging = true
 edit (@errorPage)

The CGI Framework

The CGI Framework is a set of tools and a table structure designed to make Frontier a powerful and easy-to-use CGI development environment. The two main parts of the CGI Framework are the suites.webServer table and the AppleEvent handler that processes events received from WebSTAR.

The webServer suite adds commands that make writing CGIs easier. The webServer.httpHeader command creates standard HTTP protocol headers that are needed by almost every CGI script. The webServer.errorMessage logs errors generated by CGI scripts and returns a user-defined error page. The webServer suite also defines tables to take advantage of WebSTAR’s custom action feature and to store macros that can be embedded in text objects or files.

The AppleEvent handler routes the parameters received to the appropriate script. It does this by creating a suffix mapping in WebStar that routes all requests ending in .fcgi to Frontier. Frontier uses the filename that was requested to determine which script or object to return to WebStar. Frontier ignores everything before the last / and chops off the .fcgi suffix to determine the script name. For example, if the requested URL was http://www.webedge.com/frontier/samples.tellTime.fcgi, Frontier would look for an object named samples.tellTime in the webServerScripts table.

The CGI Framework is restricted to serving only objects located in the webServerScripts table hierarchy. Not only scripts, but also string, word-processing text, and binary objects can be served from the webServerScripts table. Any macros embedded in string or text objects are automatically processed when these objects are served from the Object Database.

The AppleEvent handler wraps any CGI scripts in an error handler that traps any errors generated by the CGI. Any errors encountered during the CGI script execution are automatically logged and the error message is reported to the client.

With minor modification, AppleScript CGIs can be served from the Object Database, allowing existing AppleScript CGIs to take advantage of Frontier’s multi-threading capabilities and providing an easy migration path for webmasters who use AppleScript, but are interested in trying out Frontier. Step-by-step instructions are available on the Frontier CGI Scripting site.

The AppleEvent handler detects the scripting language used by the CGI script and formats the CGI parameters accordingly. In the case of UserTalk language CGIs, the parameters received from WebSTAR are parsed into a table and the address of the parameter table is passed as a single parameter to the CGI script.

A Sample CGI: The Pizza Processor

To illustrate how the CGI Framework works, I’ll use an example application that processes pizza orders taken from a web page. This example receives the form data, writes it to a tab-delimited text file and returns a confirmation message to the client. The form we are using has five fields: Name, Address, Phone, Size and Toppings. I’m not going the explain how to create forms using HTML, but the following line from the HTML coding is important to the running of this script:

<FORM ACTION="pizzaProcessor.fcgi" METHOD=POST>

This line tells the browser to send the encoded form data to pizzaProcessor.fcgi using the POST method. Based on the .fcgi extension, WebStar will send Frontier an Apple event containing the form data and other CGI-related information. The CGI Framework receives the Apple event and automatically decodes the form data. The field values are parsed into a sub-table named argTable in the parameter table along with other CGI values. The CGI Framework looks for a script named pizzaProcessor in the webServerScripts table. It runs the script, passing it the address of the parameter table as a parameter.

The POST method is important because the CGI Framework will only decode and parse the form data if it is sent using the POST method. It is possible to use the GET method, but you would have to decode the form data on your own.

Figure 4: Pizza Order Form

When the data gets to the pizzaProcessor script, the argTable should contain six values: Name, Address, Phone, Size, Toppings and Submit. All values will be strings except argTable.toppings which will be either a string or a list. Because it is a checkbox, multiple values can be selected. If more than one item is selected, argTable.toppings will be coerced into a list to accommodate the multiple values.

The UserTalk Code

The script we’ll use to process the information is named pizzaProcessor and is stored in the webServerScripts table. Here is a first look at the pizzaProcessor script:

pizzaProcessor : Version 1

on pizzaProcessor (params) 
 local 
 html = webServer.httpHeader ()
 dataFile = file.getSystemDisk () + "Pizza Orders"
 i
 entry
 on add (s) 
 html = html + s + cr
 with params^ 
 if defined (argTable) 
 entry = clock.now ()
    « Create record
 for i = 1 to sizeOf (argTable) 
 entry = entry + tab + string (argTable[i])
    « Create data file
 if not file.exists (dataFile) 
 file.new (dataFile)
 file.setType (dataFile, 'TEXT')
    « Write data to file
 file.writeLine (dataFile, entry)
    « Build return page
 add ("<HTML><HEAD>")
 add ("<TITLE>Thank You!</TITLE>")
 add ("</HEAD><BODY>")
 add ("<H1>Thank You!</H1>")
 add ("</BODY></HTML>")
 else 
 html = webServer.errorMessage ("No Data", params)
 return (html)

Step By Step

Let me explain the code line by line.

on pizzaProcessor (params)

The first line defines the handler that is triggered when the script is run. It receives one parameter, params, which is the address of a table containing the CGI parameters sent from the web server.

 local 
 html = webServer.httpHeader ()
 dataFile = file.getSystemDisk () + "Pizza Orders"
 i
 entry

This is where we declare the local variables for the script. The first variable, html, is a string containing the page of HTML-formatted text to be returned to the client. Its initial value is a standard HTTP header created by the webServer.httpHeader command. The header is necessary for the browser to display the returned data properly.

The full path of the text file used to store the pizza order data is kept in the dataFile variable. It is set to use a file named Pizza Orders at the root level of the startup disk. A counter variable i is created for use in a loop later in the script. The entry variable will hold the individual records to be written to the data file.

 on add (s) 
 html = html + s + cr

These two lines create a subroutine to ease adding text to the html variable. With this routine text can be appended to the html variable using add ("New Text") instead of html = html + "New Text".

 with params^ 

This with statement allows the values in the parameter table to be called by name. Since params is a variable containing the address to the table of parameters, the ^ symbol is used to expand the variable to the contents of the table it points to.

if defined (argTable)

This line checks for the existence of a sub-table named argTable within the parameter table. The argTable is created automatically by the AppleEvent handler if form data is received using the POST method. If the argTable is defined, it is safe to assume data from the form was received.

 entry = clock.now ()
    « Create record
 for i = 1 to sizeOf (argTable) 
 entry = entry + tab + string (argTable[i])

These lines create the record of the pizza order. The first line, entry = clock.now (), sets the entry variable to a string containing the current date and time. The last two lines loop through the argTable adding the value of each item and a tab separator to the record. Each item is coerced to a string to correctly deal with multiple selection items in the form, like checkboxes, which appear as a list in the argTable.

 if not file.exists (dataFile) 
 file.new (dataFile)
 file.setType (dataFile, 'TEXT')

These lines create a text file at the path specified by dataFile if one doesn’t already exist.

 file.writeLine (dataFile, entry)

This line writes the record to the data file using the file.writeLine command. The file.writeLine command automatically opens and closes the file and adds a carriage return at the end of the line.

 add ("<HTML><HEAD>")
 add ("<TITLE>Thank You!</TITLE>")
 add ("</HEAD><BODY>")
 add ("<H1>Thank You!</H1>")
 add ("</BODY></HTML>")

These five lines create the HTML page that will be returned to the client. They each use the add subroutine defined earlier to append text to the html variable. A more refined CGI would return more information, such as a confirmation with the total price of the order.

 else 
 html = webServer.errorMessage ("No Data", params)

This else statement is part of the if defined (argTable) statement above. It is called if the argTable is not defined, meaning no form data was received by the CGI. The second line sets the html variable to the result of the webServer.errorMessage command. The webServer.errorMessage logs the error and returns a user-defined HTML-formatted error page. Using the webServer.errorMessage command to generate errors is encouraged, but not required. It logs error messages to a single location and creates a consistent look by using the same error page for multiple CGIs. The first parameter is the text of the error message, the second parameter is the address the parameter table. The webServer.errorMessage generates its own HTTP header, so the result completely overwrites the html variable rather than being appended to it.

return (html)

This last line returns the contents of the html variable, either a confirmation page or an error message, to WebStar, who returns it to the client. The return command ends script execution.

Making It Better

The pizzaProcessor script is a simple example of a common CGI application. However, it does have a few problems that need correction.

First, the order of the fields in the data file has not been defined. If a field is left blank, some browsers will return a nil value for the field, while other browsers will discard empty fields entirely. It is important to maintain a consistent field order in the data file so that it can be imported into a database.

Another issue that wasn’t addressed is defining fields that are required for processing. We can’t deliver a pizza if we don’t know at least the name, address and phone number of the customer and the size of pizza being ordered.

Lastly, since this script will be run in a multi-threaded environment, it is possible that two instances of the same script will attempt to write to the data file at precisely the same time. We need to use semaphores to lock the file so that only one script can write to it at a time.

An improved version of the script looks like this:

pizzaProcessor: Version 2

on pizzaProcessor (params)
 local
 html = webServer.httpHeader ()
 dataFile = file.getSystemDisk () + "Pizza Orders"
 i
 entry
 order = {"Name","Address", "Phone", "Size", "Toppings"}
 req = {"Name", "Address", "Phone", "Size"}
 on add (s) 
 html = html + s + cr
 with params^ 
 if defined (argTable) 
 entry = clock.now ()
    « Create record
 for i = 1 to sizeOf (order)
 if (defined (argTable.[order[i]])
 and (argTable.[order[i]]   nil) 
 entry = entry + tab + string (argTable.[order[i]])
 else 
 if req contains (order[i]) 
 return (webserver.errorMessage (order[i] 
 + " is required.", params))
 entry = entry + tab
    « Create data file
 if not file.exists (dataFile) 
 file.new (dataFile)
 file.setType (dataFile, 'TEXT')
    « Write data to file
 semaphores.lock (dataFile, 3600)
 try
 file.writeLine (dataFile, entry)
 semaphores.unlock (dataFile)
 else
 semaphores.unlock (dataFile)
 scriptError (tryError)
    « Build return page
 add ("<HTML><HEAD>")
 add ("<TITLE>Thank You!</TITLE>")
 add ("</HEAD><BODY>")
 add ("<H1>Thank You!</H1>")
 add ("</BODY></HTML>")
 else 
 html = webServer.errorMessage ("No Data", params)
 return (html)

Explanation of Changes

 order = {"Name","Address", "Phone", "Size", "Toppings"}
 req = {"Name", "Address", "Phone", "Size"}

These two lines define two new variables. Order is a list of field names in the order they should appear in the data file. Req is a list of field names that must contain data for the order to be processed.

 for i = 1 to sizeOf (order)
 if (defined (argTable.[order[i]]) 
 and (argTable.[order[i]]   nil) 
 entry = entry + tab + string (argTable.[order[i]])
 else 
 if req contains (order[i]) 
 return (webserver.errorMessage (order[i] + " is required.", params))
 entry = entry + tab

This is a new loop to create the record of the pizza order. Unlike the loop in the first version, this version loops through the items in the order list, checking that each value is defined and contains data before adding it to the record. If the field is empty the if req contains (order[i]) line checks for the field name in the list of required fields and returns an error if a required field is empty.

Semaphores

 semaphores.lock (dataFile, 3600)
 try
 file.writeLine (dataFile, entry)
 semaphores.unlock (dataFile)
 else
 semaphores.unlock (dataFile)
 scriptError (tryError)

The above lines use the semaphores suite to lock and unlock the data file so that only one script will attempt to write to the file at a time. Using semaphores is very important when writing to a shared resource in multi-threaded situations.

Semaphores are like that little “occupied” sign on airplane restroom doors. When one person goes into the restroom they lock the door, which puts up the “occupied” sign. The next person comes to use the restroom, sees the sign and waits for first person to exit before opening the door.

If the first person forgets to lock the door, the “occupied” sign doesn’t light up and the next person is likely to walk in on him while he is in the restroom.

Semaphores are very similar. They manage which scripts have access to a shared resource at any given moment. Semaphores are needed when a script writes data to a shared resource, like a file or an object in the Object Database. Semaphores are not needed when reading data or when writing to local variables.

Frontier has a built-in semaphore suite to handle locking and unlocking access to shared data. The semaphores.lock command actually does two things. It waits for the resource if it is locked, and it locks the resource as soon as it becomes available.

Jumping back to the airplane restroom analogy, you wouldn’t seat someone in the restroom for the whole flight. Not just because it would be uncomfortable for the passenger but also because no one else would be able to use the restroom for the whole flight. The same thing applies to semaphores and scripts. A semaphore creates a bottleneck through which each script must pass one at a time. Don’t lock a resource for the entire running of the script. Lock the resource, write to it, then immediately unlock it.

It is always a good idea to put any semaphores.unlock commands in an try error handler. Like this:

semaphores.lock (filePath, 3600)
try
 file.writeLine (filePath, data)
 semaphores.unlock (filePath)
else
 semaphores.unlock (filePath)

This way if an error occurs during the file.writeLine operation, the lock on the resource is still released. Restarting Frontier will reset any lingering locked semaphores.

CGI Tips & Tricks

I hope the pizzaProcessor example was clear enough to get you started writing your own CGIs. When you do dive into writing your own CGI, here are some things to keep in mind when you are writing your own scripts:

Debugging Tips

• Turn on verbose logging in the webServer.preferences table. With the verboseLogging preference set to true the values of every parameter will be written to the error log.

• Use the samples.tellParams script to test how different parameters are being received by Frontier.

• Try testing your scripts with different browsers. Many browsers function very differently in identical situations.

• Always declare local variables in your scripts.

• Check the CGI Scripting site periodically for updates to the CGI Framework.

• Join the frontier-talk mailing list by sending a message to majordomo@webedge.com with the phrase “subscribe frontier-talk” in the body of the message.

Performance tips

• Always keep the WebStar application in the foreground.

• Limit the number of Apple events you send to other applications. Each Apple event exchange requires approximately 1/4 second.

• Cache reused data in the Object Database, especially data read from external files.

• Use server-push to return partial results immediately to the client. This works with any browser, not just Netscape.

Useful URL’s

Aretha http://www.hotwired.com/staff/userland/aretha/

DocServer website http://www.hotwired.com/staff/userland/docserver/

DocServer
http://www.hotwired.com/staff/userland/aretha/636builtinverbs_228.html

ClayBasket http://www.hotwired.com/userland/clay/

CGI Scripting http://www.webedge.com/frontier/

CGI Scripting BBS http://www.webedge.com/frontier/bbs.welcome.fcgi

Frontier-talk list mailto:majordomo@webedge.com

in the body of the message put: subscribe frontier-talk

Platform is Chinese Household
http://www.hotwired.com/staff/userland/platformischinesehouseh_28.html

WebEdge http://www.webedge.com/