|Column Tag:||From The Trenches
True Life Story
Developing a Client-Server system
By Malcolm H. Teas, Rye, New Hampshire
Note: Source code files accompanying article are located on MacTech CD-ROM or source code disks.
About the author
Malcolm Teas has been programming the Macintosh for five years in C with MPW and Think C. Hes active with object programming in TCL because he has a short attention span and likes to write applications quickly. His most recent shareware is Apple Π, a Π calculation program available on America Online. He lives, works, and consults from his house on the seacoast of New Hampshire. Seacoast Software, 556 Long John Road, Rye, NH 03870-2213, firstname.lastname@example.org or email@example.com
In the spirit of all those TV shows with real-life themes, and temporarily short of real article ideas, I proposed an article on the true-to-life story of how I recently helped develop a client-server system. They probably accepted this article because I used the words Case History in my proposed title, which sounds like Im knowledgeable. Of course I might have just caught them on an good day.
In any case, the plot for this story takes two programmers (including myself) and pits them against the clock and programming bugs to implement a system that the customer needed for a legal deadline. Our customer decided that, while itd be alright if I wrote about this, they didnt want their name and business included, therefore, theyll remain our customer to preserve their competitive secret. Their business, of course, should never be associated with air conditioning equipment. You never heard it from me.
What the customer needed
Our customer must, by law, track the intelligent use of their product by their large customers. To do this, there are customer representatives that visit their customers, survey them, then file papers on the surveys. For various reasons these surveys may be audited periodically, so, unlike the rest of us, they must actually be able to find the papers theyve filed over the last several years.
Our customer had been doing this for some time, not completely successfully, with paper, filing cabinets, and lots of clerks. The Macintoshes were used to generate paper and keep the clerks busy. Although the Macs were connected in a large network, they werent taken advantage of for this sort of purpose. Pages were printed on a laser printer, photocopied, then one copy was filed locally and the other was US-mailed to an archiving office.
The problems with this system were substantial. To start with, it was slow. Papers often became misfiled, were on someones desk for audit work and so unavailable to others, or got lost in the mail. The US Mail costs were starting to get expensive. It was difficult to analyze the reports in another order from the original filing order. In short, the existing system was unwieldy.
Our Client-Server design
The management that brought us in wanted to use the Macintosh more fully. The customer was using MS-Mail and file servers throughout the company successfully and was interested in making better use of the Macs and their network with a network-enabled application or netware as they called it. They saw this project as a way to move toward that.
We came up with a client-server system that uses a 4D database as the server application, and a client application written in Think C. Documents are archived by dragging their icons onto the client application icon. The client app starts up, allows the user to specify the archiving criteria, sends the document to the server, and quits.
Since far fewer people need to retrieve documents than need to archive them, we used simple file sharing on the server to allow people to retrieve documents. The client front-end application allows a wider range of less technically sophisticated people to archive documents correctly. A user doesnt need file sharing privileges to archive a document. Later, in release two, we changed this because the number of users was expected to grow substantially.
The client application has one window with popups on the left side to specify the state, division, and year that the document should be filed under. On the right side of the window are two scrolling lists: the customers name and number; and the customers location. The customers name list contents is determined by the criteria on the left side. The contents of the customers location list is determined by the left side criteria and the selected customer.
There are fields that show the currently selected customer and location. Some checkboxes allow new customers names, numbers, and locations to be added. This addition takes place when the document is actually filed. The document is filed when the Archive button in the lower right of the window is clicked.
These are the essential elements of the client application. It has no menu bar. The application is basically a one-shot. The window (in the first version) is actually a dialog.
Whats done where
We found that the main design issue was how to divide the functionality between the client and server applications. We had to decide this first; the system design was too unwieldy otherwise. Once determined, the messages between the two were defined and each piece became modularized.
Our server is a simple one that receives, stores, and retrieves data for the client. A client-server architecture is good for sharing common data with more than one user. The server handles the common part and the client creates the interface to it. We needed the server to store the documents and the criteria (customer information, state, division, etc.) used to index them.
User interaction and interface, on the other hand, is best handled on the client. After all, a program with one user is faster than a program with multiple users. This, ultimately, is the reason behind moving from mainframe-centered systems to client-server systems. Although speed isnt always an issue in the user interface (after all, humans can take a long time - up to several hundred milliseconds - to recognize that something changed on the screen, much less understand it), we can use the processing time to format and display the data in useful ways for the user. The client application also tries to condition the data sent to the server so the server doesnt have to handle as many error conditions. This is no excuse for the server programmer to forget to program error detection and recovery. Were trying to save server execution time, not make the server less robust.
Different kinds of messages
Once the customer and the other filing criteria are specified, we need to send the document to the server, and the server must file it. This interchange of messages while the user is waiting must be handled quickly, but the document sending doesnt need to be handled interactively. It could be processed by the server up to minutes later. Because of this difference between the two types of messages we needed to handle, we used two messaging methods.
Custom AppleEvents became the interactive message protocol and programmatic MicroSoft Mail became the non-interactive message protocol. This was convenient since we could easily enclose documents in the MS-Mail message to send them to the server. In the second release, we were to extend these messages to do document and report retrieval from the server.
We built custom AppleEvents to: establish communication with the server, get the customer number list, get the list of customer locations for a customer number, and tell the server to add a new customer number or location. We used the MS-Mail message to send the document to the server. The server could, at its discretion, defer processing of this document if AppleEvent messages were coming in. We gave interactivity a higher priority.
Note that a true client-server system has no real knowledge of a session. A session is the sort of communications protocol that happens when you log onto AppleLink or America Online for example. Youre connected until you log off and, more importantly, whatever happens depends on whats happened before. But a client-server system exchanges complete messages. Whatever the server does with a message is completely determined by the contents of that message from the client. Theres no explicit state memory of what the clients done before as there is with a terminal session.
Only one of our messages came close to abusing the pure client-server architecture. The AppleEvent message to establish communication from the client to the server was used to trade version numbers between the two. After all, we wanted to plan for future versions with different messages; this allowed us to detect a client and server with different versions trying to talk to each other. Other messages assumed that this was already established.
Real Life - designs great, but how do we do it?
We didnt have much time to design and build this. We were given two months, and of that time we were to design, build, and have a user test of the system before it went into production. This meant that we needed to restrict the design to only the absolutely necessary elements and to build those as quickly as possible.
This constraint help us decide to use 4D (version 2.2.3) for the server and Think C for the client. 4D uses a higher level language. While it proved easier to build database code with, the user interface you can build with it isnt as flexible. The interface you can build with 4D doesnt follow the Apple Human Interface Guidelines very closely. In addition, like a lot of specialized higher-level languages, its good at what its designed for, but of limited use in a more general application. Fortunately, the server application didnt have or need much of a user interface. It needed to work with data and communicate. The former was what 4D was designed for, the latter we added with externals (extensions in code resources) from third parties. The server uses 4D externals written by third parties to get and send messages with the client application. (We were not using the 4D Server from ACI US, just standard 4D with externals.)
The client application needed to be written in a more flexible language than 4D. It turned out to be the more complex application of the two; user interface code often is since it needs to deal with a wider range of possibilities. We chose Think C (version 5) for this since its fast compile/link/build cycle would help us meet our time goal.
Another factor in the logistics are the people. The person writing the server application is quite experienced in 4D and less familiar with C. As the author of the client, Im quite the opposite. We picked development environments that played to our strengths. This was very important in such a short-cycle development project. Partly due to our experience, we both had code samples and snippets that we reused in our respective development environments. The reused code was already written and tested, so it also sped up the process.
One of the problems of client-server development is that to develop either part, it helps to have the other part already running. After all, an unstable messaging interface is a key component that can slow development. We solved this by having the server application development lag the client. I initially developed the client with a virtual server. The clients messaging routines checked whether a global variable was set. If it was, the routines faked the expected response of the server. If not, they talked to the real server. As our messaging interface was largely defined beforehand, we knew what to expect.
Later, as the server application was developed, we could test it against the already running client application. Although there were errors in both sides, the bulk of the client was already written and running. This left us free to concentrate on the messaging and server development.
Adding bells and whistles
Security was a feature that we, as developers, were interested in. The users werent concerned with this, actually, we had to talk to them quite a while to convince them to use passwords. We didnt want them to accidentally lose something and come back to us saying why didnt you think of that?. One of the things we get paid for is to think of these things ahead of time. We also added keywords in the messages that the server checks for. If these keywords arent in a message, the server ignores the message. Its a little harder to spoof the system this way.
Once wed made the decision to use System 7s file sharing as the method of retrieving documents, our major security features were already implemented. The AppleEvents use the same security as the file sharing. Since we require that the user use the same MS-Mail ID as the file sharing user name and that the user already be logged on to MS-Mail, the security there is already taken care of too. While the user name being the same in the file sharing and the mail system may seem onerous, its not really as our customer already has this requirement to simplify their system management.
One last required feature was an autosearch of the customer number list to do auto-completion. The client application has two lists on the right side of its window that show the customer and their location. Above each of them are fields that indicate the current customer and location selected from the list. If the user types in one of these fields, the client program searches the list for the closest match. If it finds only one match, it fills out the rest of what you wouldve typed. If it finds more than one match, it moves the list to display the first matched item (the list is kept in sorted order).
Client-server systems can vary on several grounds: Computation/request, size (in bytes) of the request, amount of expected requests per unit of time, and number of clients serviced by the server. These, naturally, interact. If the server has a lot of work to do for each request (or the average request), then the amount of requests it can process is lower. Luckily for us, the parameters of this system were quite nice. A low request rate, low computation overhead per request, and small (comparatively) number of users. This let us run the server (for the first version) on an SE/30.
The second version is being rolled out to the whole USA. Were anticipating a rather larger number of users. However, weve got information from the first release to allow us to better estimate the load. One parameter for us thats important is the disk space used. We expect that to be quite high. The initial release helped us to estimate that better.
We decided that there are two ways of estimating these parameters, either the peak method or the average method. Each is better for different parameters. For example, you wouldnt use an average method for the disk space needed, youd need the peak estimate there - and a generous one too. However, if the server couldnt respond as quickly as it should, that wouldnt be terrible. So, the average method could be used to estimate the needed CPU capacity of the server.
Putting version one into production
The system went together rather quickly. Less than two months after starting development, a couple of us drove to the customers pilot office to bring the client application to the first users and train then. We promptly ran into a culture clash.
There was no problem getting the users trained. However, as predicted, they didnt want to use passwords. Wed designed the system so that most anyone could use it - including the representatives that visited the customers. We didnt know, though, that the office culture was such that the representatives didnt actually touch the keyboard. They dictated the reports, clerical staff took the taped dictation, created the documents, and filed them. Now, using the client application to file the documents, we had a very few heavy users instead of a larger number of occasional users.
In any case, the customers loved the system. One month later we received a letter praising the system and its on-time, under-budget development that met all their needs.
The things that helped us make this a quick project were: its clear, focused project definition, our ruthless approach to feature creep, and our ability to reuse existing source code. Without the focused project definition, we wouldve gotten lost in message definition problems, and issues like what feature goes where debates. When new features came up to be discussed, our approach was usually negative. Now, it isnt fun to be a killjoy, but if your goal is to get the thing out the door, then youve got to have a ruthlessly pragmatic approach: will this feature add enough benefit to compensate for the time delay? Bear in mind that estimates of development time and benefits may not be accurate either; you have to factor in risk adjustments, too.
Reusing existing code is something that should be done more often. Its like walking in seven-league boots. Imagine that youre a carpenter. Suddenly youre told that because youd built one bookshelf, youd never have to build another. You could sell that same one over and over. Why, youd be overjoyed! But many developers neglect to scavenge a project when theyve finished for reusable source code pieces. Perhaps thats the difference between just a programmer and a real software engineer.
So, why a second version?
If the customer liked it so much, why do another version? Like many complex systems, its hard to know exactly whats needed before hand. Also, some features we dropped out earlier when wed gotten too ruthless on feature-creep needed to go back in. The customer wanted several things: to get lots of its U.S. offices using this, drag-n-drop of multiple documents, better reports, and most significantly, they wanted to track the activities of the representatives.
Sealing the system
To use this system in all of their offices, wed need automatic document retrieval. After all, permitting file sharing access to a few people is one thing. A larger group is quite another, especially with the need to archive the files automatically. We could improve security and simplify the systems management with an automatic document retrieval feature. Document deletion would still be manual however. Since this is an archival system, we didnt want to make that part easy. We decided that, in large part, the system would be sealed against easy file-sharing access.
The hands-on users wanted some changes made to the interface for ease of use. We hadnt anticipated the pattern of use which led them to want to drag-n-drop multiple files. They also wanted some bigger fields so they could see longer document names. These and other, similar features would make the system far more usable. While these were little features, they were essential to the day-to-day users.
If we got graphics, use graphics!
The activity tracking was the least well-defined feature. After some work, we ended up with a flowchart of the activities that a representative goes through to inspect a customer. Some of this flowchart was defined by the legal guidelines, some by the companys guidelines. After some faltering attempts at a user interface for this, we put the flowchart horizontally in a window that scrolled from left to right.
This seems to be working out well. It shows the information in the way that the representatives and other users think of it, and clarifies the relationships between the items in the flowchart. Each item has a box with a checkbox as its title, a date field for the deadline date, and another for the actual date for finishing the item. When the item is done, the checkbox is checked. The deadline date is calculated automatically, and the actual date is filled in by the user. When the deadline date is getting near, the boxs edge changes to red and becomes bold. This gives the user time to do something before the deadline arrives.
The users also wanted to generate reports on all of their customers at once. The reports, document retrieval, and activity information all seemed to fit together and didnt seem to fit in the existing client application, so we designed a new client for the same server that handled these new functions.
Going to Objects
Technically more important, we moved from C to the subset of C++ used in Think Class Libraries (TCL). This meant that the original client (the Archiver) needed rewriting too due to its changes. Its structure was complex enough that additions were quite difficult and there were a lot of internal dependencies. Rewriting in in TCL would permit us to redesign these out. This would also allow us to re-use much of the same code in the new client (called the Monitor). We chose TCL because it was part of the development environment we were already using. While MacApp has good points - better control over segmentation for example - the overhead of MPW was too great.
In mapping out the hierarchy of objects for the Archiver and Monitor, I used a simplified form of the Booch method of design. I could simplify the method since I was the only one doing client application development. The view hierarchy was fairly obvious in design while the internal data hierarchy was less so. Often, in a client application, theres a mirroring between the view objects, those that make the visible elements of the interface, and the internal objects that maintain the data for the application. Another method is to build the view in the way that suits the interface, and the internals in the way that the data is best built.
Unfortunately, this wasnt as clear to me then as it is now. After close to ten years of building programs in C, I was able to design easily on the fly. Not that I was ever that lazy, but I did tend to keep my designs rather informal. This is less easy in OOP design. Its more difficult to go back and reshape existing objects. After all, the point is to encapsulate the information they need. This information includes the design information for that object and class. If you have to go back, youve forgotten too much already.
Im not saying that you cant go back and modify, but your systems architecture and structure is more important in OOP than in procedural programming. Extra time up front on OOP design isnt wasted. I believe its essential. The lack of good OOP design tools is also a factor; better tools would make this process easier, but they arent a cure-all, either. The mindset for OOP design and programming is different than that for procedural programming.
If you were building a car and you didnt have standardized parts, you could custom-craft the necessary parts as you went. As long as you know how to build each part as you come to it, and as long as you know overall what you want to build, custom-crafting parts isnt a problem. This is analogous to procedural programming - it takes longer, but theres no concern with standardized parts. However, to produce a number of similar cars, youd want standard parts. To use them, you need a more detailed design so you can know what to use when. Its a trade-off between design and greater flexibility. This isnt to say that OOP is bad; quite the opposite. The greater flexibility with custom programming isnt usually needed. I strongly prefer the OOP approach.
Part of the OOP design problem is figuring out just what an object is. Using the TCL helped in this respect. Objects were already defined. I could usually sub-class something to specialize its operation to what I needed. This reduced the issue of deciding what operations and data to encapsulate in a object. This issue of deciding what an object is can be quite important. After all, an object is the software representation of a design concept. Do it right and the design is written in code easily, do it wrong and the development is difficult and schedule-busting.
We re-wrote the Archiver and developed the Monitor (a more complex application) in a little over three months. As I write this, were just past user test. We made some bug fixes and small changes, and are now ready to implement across the nations. This development was significantly faster than the prior version, even though I felt I couldve done the design better.
I re-wrote the Archiver first. Some code I ported from the prior version, but most of it was new. Generally, the code I ported were algorithms that I made into methods. I couldve also simply called regular C code from the methods. That approach would be good for a collection of interrelated C routines. The approach I used was better because the original routines were largely concerned with user interface and not operations. The TCL takes care of the user interface features either by itself or by you sub-classing existing objects.
Wed specifically designed the Monitor to be similar in user interface to the Archiver. This allowed me to reuse many of the Archivers objects in the Monitor, so that sped up development significantly.
What would I do again?
The step-wise approach to client-server development with the fake server layer in the client was clearly something Id repeat. Also, 4D makes a good server for this kind of architecture. However, if the traffic to the server were significantly higher, wed have to reconsider this.
Id definitely repeat the OOP development. The lucky opportunity to do much the same thing in C and in TCL was useful in that in gave me a clear comparison. I prefer the TCL, that way I can concentrate on writing the interesting code, not the same stuff over and over.