Volume Number: 15 (1999)
Issue Number: 10
Column Tag: KoolTools

KoolTools

Introducing DCon

by Matt Slot, Ambrosia Software

DCon, the"Debugging Console," brings the joy of printf() based debugging to the MacOS. Yes, that's right, I said joy. Sometimes you just need to examine a few key variables from a given function while your application is running. Under UNIX or DOS you would simply print those values to the console, but it hasn't been so easy for Mac programmers. Until now.

There are alot of sophisticated debugging tools on the market today, so why would anyone use 1980's technology to debug a modern application? Probably because it's still effective for finding some of the most elusive bugs. Wrap a troublesome piece of code with markers, and you can tell if the application reaches one or both successfully. If you do this several times in a row (known as "binary partitioning"), you can narrow down memory corruption or crashing bugs to a given function or block of code.

Listing 1: 
Binary Partitioning
Test whether this function executes properly by adding calls to dprintf() before and after the body of the function. At the end, return the result code indicating whether the function succeeded or why it failed.

	void SampleFunction()
		{
		OSErr		error = noErr;
		
		dprintf("Start of SampleFunction()\n");
		
		error = PossibleCrashingFunction();
		if (error == noErr)
			error = AnotherCrashingFunction();
		
		dprintf("End of SampleFunction() : %hd\n", error);
	
		}

Add the header and libraries to your application, then call dprintf() to record any sort of data to the DCon console window, a log file, or both. As with printf(), you use a string to format any combination of integers and text for output. There is even a dprintmem() call to display arbitrary chunks of memory.

The DCon console window is created by running the application, or installing the DConNub extension. The latter method creates a system-wide floating window that is always in front, so the user can switch applications and still watch debug output scroll by. This window can also be hidden and shown with just a hotkey.

The DCon callbacks are interrupt safe and very fast. Since they are drawn to an always floating window, it's easy to see the output even if the target application locks up the machine shortly after.

You might wonder "Why not just use printf()?", but that requires you link against those large ANSI libraries and it's not interrupt-safe. The latter makes it unusable for those of who write device drivers, network callbacks, or even system patches.

Others might suggest DebugStr() instead, but it's not as easy to format variable strings. It also has one other drawback: dropping into Macsbug on a modern PowerMac is very slow. It must take over a whole monitor, change the depth, and swap the display - consuming a good portion of a second with each change. This overhead may interfere with operations like sound or networking that are timing sensitive.

The last, and coolest, feature of DCon is that dprintf() is safe to call even if the DCon application or extension is not installed. It simply does nothing. You can build a single debug application that you can use internally and ship to your customers without worrying whether they have installed the debugger. If they find a problem, they can install the DCon application and send you back a debug log.

I've successfully used DCon to find elusive bugs in my network and sound code, bugs which are timing dependent or only surface after hours of extended use. I now instrument all my code with dprintf(), reporting Toolbox error codes and asserting on invalid conditions. I've found DCon an invaluable addition to my debugging arsenal, because it really does kill bugs dead.

When he's not fighting the forces of evil, Matt Slot writes games and utility software for the ultimate in Macintosh shareware, Ambrosia Software. If that isn't enough, he spends quality time with his wife feeling the baby kick and shopping for adorable outfits. He welcomes your feedback and suggestions for baby names at fprefect@ambrosiasw.com. You can also visit his homepage at http://www.ambrosiasw.com/~fprefect/.

Sophisticated Circuits Rebound! 2.0

by Kelly Konechny

Sometimes the simplest ideas are the best. An idea like the one that Sophisticated Circuits had in Rebound!, a small device that detects a crash and then restarts the computer, is simple but extremely valuable. This simple, yet powerful idea has been built upon in Rebound! 2.0, which adds many new features that are bound to impress any webmaster or systems administrator.

Rebound! offers the certainty that your most important machines, web servers, file servers, or mail servers will run smoothly providing consistent service to your users. It maintains service by monitoring your computer invisibly, and detecting crashes when they happen. When a crash is detected the computer is restarted, and service is restored, all in minutes.

Installing Rebound!

Installing Rebound! is a snap. Rebound! consists of two parts, a small ADB device, and a control panel. The ADB device, which resembles a dongle, plugs into the ADB port on the back of your Macintosh allowing the keyboard to be plugged into it. And because of the introduction of USB with the iMacs and the blue and white G3's there is also a USB alternative to Rebound! called Kick-off, that is due to be released soon. The software installer for Rebound! places a small control panel in your system folder that provides you options for customizing Rebound!'s actions.

Configuring Rebound!

Once you have finished installing Rebound! it begins monitoring your system for crashes. Rebound! will work quietly in the background, and you may never know that it's there unless your system crashes. Rebound!'s default configuration will work with most systems but you may customize the settings in the Rebound! control panel.

The 'Enable Crash Detection' button is checked which allows Rebound! to monitor your systems activity, and attempt a restart if it detects a crash. Rebound! works by updating an internal counter periodically in the Rebound! software which then communicates with the Rebound! hardware. Any time in which the counter is stopped, Rebound! sends the 'command-control-power on' keystroke to restart the system.

When setting the time on the first sliding bar, 'Restart if system doesn't respond', consider that if this is a server unit it could encounter complex tasks which take time and Rebound! could interpret it as a system crash. The default setting of five minutes is suitable, but if you know that the server is weighed down with users or heavy tasks, you should set this number higher.

'Time allowed for system restart' simply tells the Rebound! hardware how long to wait before starting to send events again to the software. If the system crashes again during the start up process, Rebound! will attempt to restart it again based on how many time you set it to try to restart on the 'Number of restart attempts' sliding bar. This setting should typically remain at one attempt, if your system requires more than one attempt to restart during the start up process, then there many be other problems to address.

Rebound! can also monitor individual applications on your system. If you check off the 'Enable Application Crash Detection' box Rebound! will present you with options that enable the two types of crashes, unexpected quits, and application timers. You can then choose how to respond to the crash by selecting either relaunch application, or restarting system. Rebound! also displays a list of the applications running along with their application timers.

• Appleshare IP 6.2 – Apple Computer
• Communigate – Stalker Software
• LetterRip Pro – Fog City Software
• Newstand – Imagina Inc.
• PageSentry – Maxum Development Inc.
• Quid Pro Quo – Social Engineering Inc.
• QuickDNS Pro – Men & Mice
• SiteMonitor – Pacific Coast Software
• Web Server 4D – MDG Computer Services Inc.
• WebStar – Starnine Technologies

Rebound! monitors application crashes because it's possible that an application could crash leaving the rest of the system functioning properly. This could potentially put a system like a web server at risk of going down because if the web server software crashes, but Rebound! is still communicating with the system no action is taken. Rebound! can detect errors like, 'Application Unexpectedly Quit', and other errors which affect only one application. Rebound! uses two methods to achieve this, one by using 'Application Watchdog Timers' which can monitor any application's status, and tell if the application has encountered an unexpect quit. The second method, 'Application Timers' can detect situations where an application has only stalled without actually crashing. Based on a three-level fallback mechanism Rebound! can automatically re-launch a single failed application which leaves the operating system and other applications running. If the application fails to re-launch, Rebound! will activate a Finder restart, from the 'Special' menu, and if the restart still fails Rebound! will send the restart command via the command-control-power on keystroke.

How to crash your system

Once you've installed and configured Rebound! for your system how can you be sure that it will do it's job? In order to test Rebound! simply hold down the command key and hit the power on key. A dialog box will appear with the '>' symbol in it. This is the Mac's built in debugger, with it open no other tasks are allowed to run, and in a few minutes Rebound! should restart the system.

Methods of testing Rebound! are applications that purposely crash a system using a wide variety of system crashes. Crash Test is a small application that allows you to select the type of crash you want to test, and then crashes the system. Keep in mind that when doing any sort of crash testing on your system, you shouldn't have any files open as data can be lost. You may also want to run disk first aid on the system to insure there aren't any hidden problems caused by the crash testing.

Advanced Rebound! Features

Rebound! also offers features that go above and beyond what you'd expect from this little device. Features like advanced logging, event messaging, advancements in automatic crash detection, recovery, and prevention.

Advanced logging tracks both system and application activities. You can view a detailed log including the time, date, priority, and the type of activity. Activities that are logged include when an application was launched, when it was quit or automatically restarted, and when the system crashed and was restarted. These options give administrators the ability to know exactly when the system had a problem, and what the outcome of the specific problem was.

Accessing the log in some cases could be a lengthy connection process to a remote server, so to avoid this viewing the log of system activities can be done through any browser. Rebound! comes with a WebStar API plug-in that allows the administrator to get an up to the second view of the system's activities.

Being the first to know of a crash is of high value to any webmaster or system administrator. Rebound! can notify you via email if a server has encountered a problem. Rebound! sends the log file along with the high-priority email so you know exactly what the problem was.

Of course the best way to keep a system running is to avoid a crash altogether. Rebound! allows the administrator to set scheduled restarts of the entire system which helps eliminate problems before they occur. Problems like memory fragmentation, can be a regular cause of crashing. By restarting the system regularly these problem are less likely to occur.

Scriptable Rebound!

The nice thing about any great piece of software (and hardware in this case) is the ability to script it. Rebound's features can be controlled by AppleScript or any other OSA-compliant scripting language like Frontier. Scripting Rebound! is a valuable feature because different scripts could be set up based on different events. And since several of Rebound!'s features are accessible by Applescript the options are numerous. Creating an Applescript that changes any one or many of Rebound!'s settings is simple. Place a 'tell' block in your script like this:

tell application “Rebound! Extension”
...
end tell

When scripting Rebound! several of it's properties can be modified or read. The properties shown in figure 2 outline aspects of Rebound! that are scriptable.

System Crashes properties

• sysTimerEnabled - Set to true to enable system crash detection
• crashDetectionTimeout - Value of the "Restart if system doesn't respond" slider, in seconds
• restartInterval - Value of the "Time allowed for system to restart" slider, in seconds
• restartCount - Value of the "Number of restart attempts" slider

Application Crashes panel properties

• appCrashDetectEnabled - Set to true to enable application crash detection
• appTimersEnabled - Set to true to enable application timer monitoring
• appRelaunchCount - Number to times to attempt to relaunch a crashed application before restarting the system
• appRelaunchWindow - interval (in seconds) of repeated application crashes which will cause Rebound! to restart the system

Log Entry Priorities

You can customize the priorities of Rebound!'s log entries by setting these properties. Possible values are 'emergency' (0), 'alert' (1), 'critical' (2), 'err' (3), 'warning' (4), 'notice' (5), 'info' (6), 'debug' (7) and 'disabled' (-1).

• sysCrashRestartPriority (3) - "System crashes" log entries
• appCrashPriority (3) - "Application crashes/timed restart" log entries
• appTimerExpirePriority (3) - "Application crashes/timed restart" log entries
• appRelaunchPriority (5) - "Application crashes/timed restart" log entries
• appCrashRestartPriority (5) - "Application crashes/timed restart" log entries
• appLaunchPriority (6) - "Application is launched or quit" log entries
• appQuitPriority (6) quit" log entries - "Application is launched or quit" log entries
• daemonLaunchPriority (6) - "Rebound! Extension launches or quits" log entries
• daemonQuitPriority (6) - "Rebound! Extension launches or quits" log entries
• daemonQuitAEPriority (4) - Also "Rebound! Extension launches or quits," but reported when another application tells Rebound! to quit

Relaunching open applications after crash

• bookmarkingEnabled – Set to true to enable relaunching open applications after a crash

Periodically restarting the system

To set this value, add up the values for each day of the week as follows: Sunday=1, Monday= 2, Tuesday=4, Wednesdays=8, Thursday=16, Friday=32, Saturday=64. For example, to restart on Monday, Wednesday and Friday, set this value to 2+8+32=42.

• autoRestartDays - Bit mask telling which days of the week to automatically restart the system (set to 0 to disable)
• autoRestartTime - Time of day at which to automatically restart the system

In your script you can look at and modify the properties (Figure 2) by using Applescript's get and set commands. For example, to change the value of the "Restart if system doesn't respond" slider to ten minutes, place the following command in your tell block:

set crashDetectionTimeout to 600

Application crash detection in Rebound! works by creating appTimers for every monitored application. Each application sets it's appTimer before it reaches zero, so that if that application crashes the appTimer will not be set and will reach zero. Then Rebound! kicks in and assumes that the application has crashed and restarts the system. You may use Rebound!'s application crash detection to monitor databases, scripts or applications that you have written. Your application should send a tickleAppTimer Applescript command. The command looks like this:

tickleAppTimer 400

Rebound! then creates a new appTimer for your custom application, script, or database, and repeated commands will constantly update that timer.

Whether it's a file server that has to be up 24/7 for production purposes, or if it's a web server that hosts your e-commerce site, Rebound! will keep the machine consistently running with minimum down time. And with features that notify you of an application crash or a system crash with a log attached of the events that happened sent via email, you can be sure that you'll know at the first sign of trouble. With Rebound! installed your most important machines will be running smoothly, which allows you to concentrate more on other tasks at hand.

Rebound! plugs into the back of your Macintosh and allows for another ADB device to pass through.

You can find out more about Rebound! and Kick-off! at Sophisticated Circuits web site at <http://www.sophisticated.com>.

Kelly Konechny is the Publishing Systems Coordinator, and Webmaster at Farm Business Communications in Winnipeg, Manitoba, Canada. Kelly can be found most days working on http://www.agcanada.com, and he can be reached at kkonechny@fbc.unitedgrain.ca.