TweetFollow Us on Twitter

Jun 99 Factory Floor

Volume Number: 15 (1999)
Issue Number: 6
Column Tag: From The Factory Floor

A Conversation with Chuck Shotton

by Chuck Shotton and Dave Mark, ©1999 by Metrowerks, Inc., all rights reserved.

This month's Factory Floor interview is with Chuck Shotton, known by many as the father of Macintosh web servers. Chuck was the author of MacHTTP, the first Macintosh web server, which later became WebStar. Chuck took time out from his latest groundbreaking progamming project to give us his side of the story.

At various times, Chuck Shotton has toiled in the salt mines of defense contracting, under the yoke of academia, and in the trenches of private enterprise. His latest exercise in humility is as CEO of BIAP Systems, Inc. where he, along with a cast of thousands, alternates between groveling before mighty VCs and flogging lowly C compilers to do his bidding in a quest to create the Next Killer App (tm) for the Internet. Coming soon to a CRT near you!

Dave: How did you get started with Mac Programming?

Chuck: In one of my previous lives, I worked for a defense contractor in the Washington, DC area. One of the few bright spots in that otherwise Dilbert-esque landscape came along in late 1984 when we got our first Macs to play with. Turns out that most of the guys in my department were old Apple ][ fans and a little arm-twisting resulted in some screaming 8 mHz, dual floppy 128k Macs. We were supposed to port a large chunk of code from a Vax 11/780 to these Macs (which actually had more RAM than the Vax!) for a process management system used by the Navy. At that time, "Inside Mac" wasn't really available and the total extent of the Mac programming documentation available to me consisted of back issues of MacTutor magazine and the Consulair C Compiler's instruction manual. So basically, I learned Mac programming at taxpayer expense on a contract for the Navy by sitting around all day reading old magazines and disassembler output.

My first real dose of uninhibited (read that "unclassified") Mac programming didn't come until 1986 when I started working on a contract in my spare time for Simutronics, which developed most of the popular on-line games for the GEnie on-line service. I built the graphical front end for Mac players of the "Orb Wars" game. I think that was the one and only time I ever got to work on game software for money and it's been down hill ever since.

Dave: What can you tell me about the early days of MacHTTP?

Chuck: MacHTTP was one of those life altering events that you don't really know whether it was good or bad until a long time afterwards (turns out it was good - mostly). In late 1992, rumors of this World Wide Web thing were floating around in the scientific community. Most people involved in academic information systems at the time (I was working at the U. of Texas in Houston then) were drooling over Gopher, Archie, and other text-based interfaces to existing Internet services. I decided to give the Web a try and began corresponding with the guys at CERN in Switzerland. Because almost everything was Unix-based at the time, I wanted to work on a Mac Web browser. Tim Berners-Lee wrote back that CERN had a great Mac browser in the works (anyone remember the CERN web browser?) and maybe I should look into writing a server for the Mac instead.

So MacHTTP sprang into being in early '93 after an all night coding session. In its original incarnation, it handled exactly one connection at a time, had this Byzantine event loop that split its time between babysitting the MacTCP driver and looking for the user to type command-Q, and served anything on your hard disk. I still had to test it with a Unix browser until NCSA released the first Mac version of Mosaic, but the Mac OS was now the proud host of the second Web server on the planet after the Unix one created by the guys at CERN.

Another 6 months of tweaking, fiddling, tuning, and very little interaction with sunlight resulted in the first shareware version of MacHTTP. By the middle of '94, MacHTTP had become the proverbial self-eating watermelon. Developing, testing, supporting, and selling a software product by yourself can get to be a herculean task. Even with my wife and I working at it (she was a C coding geek as well, though a reformed Ada zealot now), it was too much to do as a part time effort. We ended up flipping the baby, the bathwater, and my day job over the fence to StarNine Technologies in April of '95 and WebSTAR was born.

Dave: What lessons did you learn in bringing MacHTTP to market?

Chuck: Take cash!. If you can grow a product into something that gets on peoples' radar scopes, someone will offer to make your life easier by taking it off your hands. Take cash! Seriously though, I think the biggest lessons learned had to do with understanding how big companies (like Apple and Quarterdeck) do business and what technology-driven individuals have to do to survive in that commercial environment. Being able to build a community of users and developers around a product and stay with it from start to finish is a luxury that few people get to experience. Finding good people to work with (the DTS guys at Apple, StarNine's management team, etc.) made my life a lot easier.

Dave: As MacHTTP evolved, Apple was spinning their own Open Transport TCP/IP technology. How did you deal with this?

Chuck: Open Transport was probably one of the most painful O/S transitions Apple has ever inflicted on its developer community. Just at the point that MacTCP was becoming well-understood in the developer community, widely distributed, and pretty much universally supported, along comes OT. As with most examples of not-invented-here syndrome, OT was based on the mostly unsupported STREAMS interface instead of the nearly universally supported Berkeley Sockets interface. So not only did developers have to rewrite all their code to use the new APIs, but they had to learn something from scratch that most of them had never seen before and didn't really work for the first 3 releases.

OT was definitely a case where it didn't pay to be an early adopter. In fact, it was unusable for server applications until its third release due to memory leaks, odd timing errors with large file transfers, and premature connection terminations. We managed to accomodate it pretty easily once it was debugged because WebSTAR was built to our own TCP/IP APIs and not the lower level O/S specific APIs for MacTCP, OT, or BSD. That's not to say we had our own TCP/IP stack, just that we chose to wrap the low level stuff in some easier to use code. That philosophy was used throughout much of WebSTAR and it made it a lot easier to accomodate changes in the O/S than other code that was written to the native O/S APIs.

Dave: What did you have to do to create a real high performance server on the Mac OS?

Chuck: Most well-written servers turn out to be I/O bound applications. They spend most of their time shoving data from point A to point B and very little time actually doing anything to the data itself. For all the rocks that get thrown at it, the Mac O/S can still outperform servers running on other platforms like Linux if the code is well-written. One of the biggest fallacies is that the Mac's tasking model is bad (not preemptive) and the file system is slow. I won't argue the latter point, but the former is totally bogus.

The Thread Manager is actually a server developer's best friend if you use it right. Operating systems like Unix (prior to standard implementations of lightweight threads) have to run parallel threads of execution as separate processes. That means heavy-duty context switches and a very crude control over task priorities and how control is relinquished. That generally equates to poor performance in a multi-threaded application with lots of context switches (like a Web server). On the Mac, the Thread Manager allows for really quick context switches that are completely under the control of the application. If you know it's time to handle a new incoming connection, your code can yield directly to the code responsible. Or if you know you're waiting on a remote client to disconnect, you can pass control to another thread that is in the middle of a massive file transfer. Being your own scheduler is a good thing if you have a well-understood process that you are implementing. Relying on a generic O/S-level scheduler results in pretty lame performance compared to a highly-tuned, application-specific scheduler built on a threading environment like the Thread Manager.

There were two other major areas of optimization that helped tune MacHTTP and WebSTAR to the Mac O/S. The first was data caching. Since it's costly to find a file in the Mac O/S (either to read data from it or to find file system attributes like the creation date), caching this information for future use provided one of the biggest performance increases. In-memory databases are all the rage now and they are basically fancy caching mechanisms much like the ones these Web servers used. If you can come up with a good algorithm for managing cache hits, you can end up with a large majority of your data requests coming out of memory instead of ratting through B-Trees on disk trying to locate Finder info.

Accomodating the idiosyncracies of the Mac's networking code provided the final big performance boost. Simple things like double-buffering your data output so that one write operation is always awaiting the completion of its predecessor ensures that the network connection is always fully utilized. Reading data into buffers larger than the default sizes makes sure that you don't waste a lot of CPU cycles processing small amounts of data lots of times. Too many context switches between threads can occur if you don't have the right balance between I/O time and calculation/processing time. Spend more time reading or writing data since that takes a lot more time to perform than the calculations the server has to perform on the data. A lot of this stuff is just common sense, but it's amazing how many developers don't bother to tune their code for best performance after they have the basic functionality up and running.

Dave: How does an app developer deal with the rich mix of connection types and speeds (e.g., slow dial-ups to high bandwidth direct connect)?

Chuck: That's a big challenge for the next generation of Internet apps. The current crop of Web browsers and servers typify the one-size-fits-all approach to network software development. Application performance is metered by the user's patience level and is in direct proportion to the available network bandwith. This results in adequate performance if you have a large network pipe and painfully slow performance if you're sucking data through a 28.8 soda straw.

Ideally, the data and application performance should scale with the performance of the network. A user with a 28.8 dial-up connection should have application performance just as perky as someone using an ADSL connection. The way to accomodate this is to write smarter software that can select data appropriate for the connection speed and adjust dynamically to changing CPU and network loads. Intelligent caching, back-side information sharing on the LAN, and distilled content (e.g., discarding banners, eye candy, and other non-informational content) can get just as much useful data through a dial-up line as currently flows over a T1, from a user's perspective.

We're working on a new generation of desktop network applications (super secret codename, "Gossip", for the terminally curious) that takes a shot at implementing some of these concepts. We have to very carefully monitor the user's available bandwidth, their usage patterns, the types of data they are interested in, and adjust the clients' interaction with the net accordingly. It's hard to write smart software. I think that's the next big hurdle for developers trying to develop apps for the Internet. All the niches for dumb products are filled with stuff from Microsoft, Sun, AOL, and other NASDAQ luminaries. If you want a piece of this market now, your software better do something smart for a change.

Dave: What advantage do I have as a Mac Develper dealing with the web and the internet?

Chuck: The Mac's singular advantage over every other platform centers around AppleEvents and scripting. And unfortunately, it's one of its most underutilized and underadvertised features. Apple birthed a phenomenal technology with AppleEvents. I've advocated a cross-platform version of AppleEvents since the day after I read that chapter in Inside Mac. But until then, the ease with which complicated Web-based applications can be put together on the Mac is unrivaled. Sure, you can muck around with active server pages, glom together some Perl scripts, or hack out some custom CGIs. But nothing available on any other O/S rivals the ease with which you can hook two distributed applications together via AppleEvents. I wish Apple would revisit this technology and consider a cross-platform version that runs over TCP/IP. Until then, Mac developers can smugly take advantage of a feature that doesn't really exist on any other platform.

Dave: What are you focused on these days?

Chuck: I'm back at the helm of BIAP Systems, Inc. We just finished relocating the company's main offices from Houston, TX to Leesburg, VA and are busily at work on the Next Big Thing (tm). Much of last year was spent raising money for this exercise and we're finally getting to roll up our sleeves and sling some code again. We're primarily focused on software that creates communities. That's an outgrowth of some of the personal server concepts we kicked around with MacHTTP and WebSTAR as well as the chat server development we did last year at BIAP. It's all cross-platform stuff this time and we're torturing ourselves with the prospect of getting the same chunks of code running on the Mac, Windows, and Linux. If you know anybody that wants to help, give us a holler.
 
AAPL
$94.72
Apple Inc.
+0.78
MSFT
$44.83
Microsoft Corpora
-0.01
GOOG
$594.74
Google Inc.
+5.27

MacTech Search:
Community Search:

Software Updates via MacUpdate

Airmail 1.4 - Powerful, minimal email cl...
Airmail is a powerful, minimal mail client.It was designed to retain the same experience with a single or multiple accounts and provide a quick, modern and easy-to-use user experience. Airmail... Read more
Macs Fan Control 1.1.12 - Monitor and co...
Macs Fan Control allows you to monitor and control almost any aspect of your computer's fans, with support for controlling fan speed, temperature sensors pane, menu-bar icon, and autostart with... Read more
A Better Finder Rename 9.37 - File, phot...
A Better Finder Rename is the most complete renaming solution available on the market today. That's why, since 1996, tens of thousands of hobbyists, professionals and businesses depend on A Better... Read more
MacBook Air EFI Firmware Update 2.9 - Fo...
MacBook Air EFI Firmware Update is recommended for MacBook Air (Mid 2011) models. This update addresses an issue where systems may take longer to wake from sleep than expected and fixes a rare issue... Read more
FileZilla 3.9.0.1 - Fast and reliable FT...
FileZilla (ported from Windows) is a fast and reliable FTP client and server with lots of useful features and an intuitive interface.Version 3.9.0.1: MSW: Fix installation issue with locked DLLs... Read more
OS X Yosemite 10.10 DP4 - Developer Prev...
Note: This is a Developer Preview. You must be a registered Apple Mac Developer to download this update. OS X Yosemite is Apple's newest operating system for Mac. An elegant design that feels... Read more
FinderPop 2.5.6 - Classic Mac utility, n...
FinderPop is a Universal preference pane that extends OS X's contextual menus using a FinderPop Items folder much as the Apple Menu Items folder used to do for the Apple menu. It has other features... Read more
SpiderOak 5.1.7 - Secure cloud backup, s...
SpiderOak is a multi-platform secure online backup, storage, access, and sharing solution engineered for the consumer and small businesses. You must first sign up to use SpiderOak. Running natively... Read more
Espionage 3.6 - Simple, state of the art...
Espionage offers state-of-the-art encryption and plausible deniability for your confidential data. Sometimes, encrypting your data isn't enough to protect it. That's why Espionage 3 goes beyond data... Read more
calibre 1.45.0 - Complete e-library mana...
Calibre is a complete e-book library manager. Organize your collection, convert your books to multiple formats, and sync with all of your devices. Let Calibre be your multi-tasking digital... Read more

Latest Forum Discussions

See All

Celebrate Summer With a Cat in the Hat L...
Celebrate Summer With a Cat in the Hat Learning Library Sale Posted by Ellis Spice on July 22nd, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
MyTaskList Review
MyTaskList Review By Jennifer Allen on July 22nd, 2014 Our Rating: :: EFFECTIVE IF PLAINUniversal App - Designed for iPhone and iPad It’s not the most stylish of task management apps, but MyTaskList has all the features you could... | Read more »
FlyCraft Herbie: Crazy Machines Review
FlyCraft Herbie: Crazy Machines Review By Jennifer Allen on July 22nd, 2014 Our Rating: :: TRICKY FLYINGUniversal App - Designed for iPhone and iPad A tough game of careful thrusting and navigation, FlyCraft Herbie: Crazy Machines... | Read more »
MTN Review
MTN Review By Jessica Fisher on July 22nd, 2014 Our Rating: :: ADORABLE, SERENE, AND AMUSINGUniversal App - Designed for iPhone and iPad MTN is an adorable, talking pet mountain that is less game and more zen garden.   | Read more »
Fly High with Ninja UP! Now Available o...
Fly High with Ninja UP! Now Available on the App Store Posted by Jessica Fisher on July 22nd, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Bio Inc. Review
Bio Inc. Review By Nadia Oxford on July 22nd, 2014 Our Rating: :: SICKENING - IN A COMPELLING WAYUniversal App - Designed for iPhone and iPad Bio Inc is about orchestrating the medical destruction of a single person. If that doesn’... | Read more »
HELMUT Review
HELMUT Review By Andrew Fisher on July 21st, 2014 Our Rating: :: TRUNDLE SIMULATOR 2014Universal App - Designed for iPhone and iPad HELMUT is a fun, fleeting time-sink that offers a momentary distraction and nothing else.   | Read more »
Walkr Review
Walkr Review By Jennifer Allen on July 21st, 2014 Our Rating: :: ORIGINAL WALKINGiPhone App - Designed for the iPhone, compatible with the iPad Walking is a bit more exciting thanks to this planet building/discovering sim reliant... | Read more »
Zombie Commando Review
Zombie Commando Review By Jennifer Allen on July 21st, 2014 Our Rating: :: MINDLESS SLAUGHTERUniversal App - Designed for iPhone and iPad Briefly fun but ultimately forgettable, Zombie Commando will scratch an itch then be... | Read more »
Swords & Poker Adventures Review
Swords & Poker Adventures Review By Jennifer Allen on July 21st, 2014 Our Rating: :: SOULLESS POKER PLAYUniversal App - Designed for iPhone and iPad Swords & Poker Adventures is a mishmash of Poker and RPGing, but it lacks... | Read more »

Price Scanner via MacPrices.net

WaterField Designs Unveils Cycling Ride Pouch...
High end computer case and bag maker WaterField Designs of San Francisco now enters the cycling market with the introduction of the Cycling Ride Pouch – an upscale toolkit with a scratch-free iPhone... Read more
Kingston Digital Ships Large Capacity Near 1T...
Kingston Digital, Inc., the Flash memory affiliate of Kingston Technology Company, Inc.,has announced its latest addition to the SSDNow V300 series, the V310. The Kingston SSDNow V310 solid-state... Read more
15-inch 2.0GHz MacBook Pro Retina on sale for...
B&H Photo has the 15″ 2.0GHz Retina MacBook Pro on sale for $1829 including free shipping plus NY sales tax only. Their price is $170 off MSRP. B&H will also include free copies of Parallels... Read more
Apple restocks refurbished Mac minis for up t...
The Apple Store has restocked Apple Certified Refurbished Mac minis for up to $150 off the cost of new models. Apple’s one-year warranty is included with each mini, and shipping is free: - 2.5GHz Mac... Read more
Twelve South HiRise For MacBook – Height-Adju...
If you use your MacBook as a workhorse desktop substitute, as many of us do, a laptop stand combined with an external keyboard and pointing device are pretty much obligatory if you want to avoid... Read more
Why The Mac Was Not Included In The Apple/IBM...
TUAW’s Yoni Heisler cites Fredrick Paul of Network World whoi blogged last week that the Mac’s conspicuous absence from Apple and IBM’s landmark partnership agreement represents a huge squandered... Read more
Save $100 on 13-inch Retina MacBook Pros, plu...
Adorama has 13″ Retina MacBook Pros on sale for $100 off MSRP. Shipping is free, and Adorama charges sales tax in NY & NJ only: - 13″ 2.4GHz/128GB MacBook Pro with Retina Display: $1199 - 13″ 2.... Read more
Blurr it 2.3 for iOS – Quickly Blurs Selected...
Hyderabad, India based TouchLabs has announced a new update of Blurr it 2.3, their photography app for iOS users. Blurr it allows you to blur part of the image to hide potentially sensitive or... Read more
MacBook Airs on sale for $100 off MSRP, start...
Best Buy has the new 2014 MacBook Airs on sale for up to $100 off MSRP on their online store. Choose free home shipping or free local store pickup (if available). Prices valid for online orders only... Read more
Amazon Announces Kindle Unlimited: Unlimited...
Amazon.com has introduced Kindle Unlimited — a new subscription service which allows customers to freely read as much as they want from over 600,000 Kindle books, and listen as much as they want to... Read more

Jobs Board

Sr Software Lead Engineer, *Apple* Online S...
Sr Software Lead Engineer, Apple Online Store Publishing Systems Keywords: Company: Apple Job Code: E3PCAK8MgYYkw Location (City or ZIP): Santa Clara Status: Full Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Sr *Apple* Engineer - IT - Requisition #: -...
For more information about TIAA-CREF, visit our website . The Apple Engineer will provide engineering and third-level incident support for 300- 500 MacOS desktop/laptop Read more
*Apple* Systems Administrator - DISH (United...
…satellite service provider, and Dish is currently looking for an experienced Apple /Mac Systems Administrator. Apple systems administrator will be responsible for Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.