TweetFollow Us on Twitter

More on Logs

Volume Number: 23 (2007)
Issue Number: 02
Column Tag: Mac in the Shell

More on Logs

Watching log files without really watching them.

by Edward Marczak

Introduction

Last month's column dealt with reading and interpreting log files in a very generic sense. I typically go on at length about using logs to help troubleshoot and simply understand your system. Of course, each system you maintain will have logs to watch — multiple logs, in fact. So how could you possibly keep track of the activity in all of them at once? Well, we're talking about computers, right? This month, I'll be describing swatch, my favorite utility for letting me take my eyes off of scrolling text displays for a while, and know that I'll get alerted if there are any problems.

Do you have the time?

Of course, the other thing I tend to talk up a lot is automation. Turns out that computers can be pretty good at monitoring themselves — with the right software, of course. Enter swatch. Swatch began life as the "simple watcher" of logs. It's evolved past that quite a bit, but has retained the name. The current version is v3.2.1 and is written in perl.

Setup is incredibly straightforward for anyone comfortable on the command-line (which longtime readers now should be!), but does require some perl modules that are not included with the stock OS X install. To that end, we've created a package that installs swatch and all of its dependencies. You can download from ftp://ftp.mactech.com/src/. Double-click the .pkg file and let the installer do its thing. For those that want to handle this themselves, follow along. If you don't care about manual set up, feel free to skip ahead to "Running With the Files."

Installing Swatch

As always, bring up your favorite terminal app, and let's get started. Swatch has a repository on Sourceforge, and you can go hunt it down there. However, we already have a shell open, why not use it?!? Create a download directory and change into it. Then, download swatch using curl:

curl -O http://easynews.dl.sourceforge.net/sourceforge/¬ 
swatch/swatch-3.2.1.tar.gz

Untar it (tar xzvf swatch-3.2.1.tar.gz) and then change into the newly created swatch directory. Before we can successfully install swatch, however, we need to add some perl modules to our system. The easiest way is to utilize cpan — the Comprehensive Perl Archive Network. If you've never run cpan before, you'll need to go through a brief set up. sudo up to a root shell (sudo -s), as we're also going to install these modules now, and type cpan. You'll be asked if you're ready to perform manual setup. Take the default "[yes]" and off we go. Interestingly, you can literally take the defaults for the bulk of the questions — a testament to authors. You should, of course, read a little deeper. However, if you're not paying attention, you'll need to finally really answer a question when you're asked for mirror sites. Choose sites that are close to you, or that you know have a good chuck of bandwidth. Also, I like changing the default answer on "Policy on building prerequisites (follow, ask, or ignore)." The default is ask, however, if you want the module, you're going to want the dependency. So, after years of manually allowing the dependency each time, now, I set cpan up to "follow", which will automatically download and build the dependency. Let cpan go through its routine, and you'll finally be (unceremoniously) dumped at a "cpan>" prompt.

How do we know which modules to install? In addition to some information in the INSTALL file that comes with swatch, you'll get this if you try to make swatch with these modules:

Warning: prerequisite Date::Calc 0 not found.
Warning: prerequisite Date::Format 0 not found.
Warning: prerequisite Date::Manip 0 not found.
Warning: prerequisite File::Tail 0 not found.

So, from here, we need to install each one. Type:

install Date::Calc

at the "cpan>" prompt to install the first one. cpan will find and download the module, then compile and install it. It will find a dependency and either ask you, or if you told it to follow, just go download it itself. You will be asked if you want to add Object::Deadly to the tests. Just press return for the default of "no". Back to a cpan prompt with no errors? Great — let's repeat the process for the remaining three modules:

install Date::Format
install Date::Manip
install File::Tail

Once you're finally back at a cpan prompt after all of this, simply type "exit", "quit", or, if you're really hip, "q", to return to your shell prompt. You should still be in the swatch directory, and we can simply install swatch from here:

perl Makefile.pl
make
make install

There should be no errors reported, and you'll now have a working swatch in /usr/bin.

Running With the Files

Now that swatch is installed, you may be anxious to get going. If you simply type swatch, you'll be disappointed. You'll get a message that you need a config file to do anything useful. Well, that's where we'll head next. Long time readers will remember my many pleas for you to learn regular expressions. Swatch configuration will vindicate me once again!

Swatch uses a simple configuration file that, at its most basic, contains patterns to look for, and actions to take if said pattern is seen. When you hear "pattern," you should immediately think of regular expressions and ruling the universe. Additionally, swatch lets you configure throttles so you don't act on log lines that are quickly, and repeatedly sent to the log file.

More specifically, a trigger definition in swatch consists of three parts:

A trigger pattern, once matched causes the action to run.

Action(s), or, what to do when the pattern is matched.

A length of time to suppress duplicate log entries (optional).

So, let's set up a fictional entry, and then we'll force 'errors' into the log. First thing to do is to create our swatchrc file. I can expound on different methodologies regarding where this file should live, but we'll keep it easy for now and use the default: ~/.swatchrc. So, create a file in your home directory (and, if you're still following along, you're sudoed up to root, but thanks to sudo, this will create it in the home directory of account you sudoed from) called .swatchrc (note the leading dot). Give it these contents:

watchfor /anecho/
echo
throttle 0:15:00
watchfor /thebell/
bell 3

Save it, and run swatch thusly:

swatch -t /var/log/system.log

and you should be greeted with an "I'm running" message like this:

*** swatch version 3.2.1 (pid:3251) started at Mon Jan 15 14:29:33 EST 2007

We didn't tell swatch to go into the background, so, it'll monopolize our terminal while it runs. For the sake of this example, that's good. Grab a second terminal (Apple-N or File->New Shell), and let's put this example through its paces. We'll use the logger command to inject new lines into the log. In the new, non-swatch-running terminal, type:

logger anecho

...and you should see the swatch window have a reaction: it echoes the log file line to the display. Type:

logger thebell

...and you should hear your system beep three times. Now, once again, type logger anecho. You won't get a second line on the swatch console. But another "logger thebell" will make swatch beep (again and again). What's going on here?

First, the command line switch "-t" tells swatch which file to monitor, or tail. On the first line of the definition, we tell swatch the pattern (regex) to watch for in the log file. This can be any perl regex you like. The line(s) following consist of the action(s) to take when the pattern is found. There are several, and I'll hit the ones that I find most useful. Finally, you can set a throttle, so that multiple, repeated lines in the log file don't cause you to trigger an action too many times. In our first definition, we used a throttle, that said, "don't repeat these actions inside of 15 minutes." So, if there's a matching entry in your log that is repeating every minute, you'll only see an action 4 times an hour. We didn't give the second entry a throttle, so it'll repeat its action as many times as it's triggered.

Pattern, Trigger...Action!

So, about now, all sorts of ideas should be running through your brain regarding how this could be useful. Let's look at some of the more useful actions that swatch will perform based on a pattern being triggered. You've already seen two: echo and bell. These are of incredible use if you want to set up a dedicated swatch console on some machine. Otherwise, you'll want some other way of being notified. For the sake of completeness, let's look at those options first.

An action of echo will simply echo the matching log file line to the console. As OS X users, however, that may not be very useful if swatch is running daemonized in the background and we don't even have a console open! Similarly, bell will ring the console bell the number of times specified.

A 'simple' action is mail. This action sends the matching log file to the address you specify. Check it out:

mail addresses=marczak\@radiotope.com,subject=Some\ Bad\ Error

Notice the need to escape the usual suspects. This action, when triggered, will send a fairly uninspired mail message with the specified subject to the specified address. However, depending on the systems and workflow you have in place, this may be all you need (e.g. — Help desk systems that require input via mail to create a trouble ticket, etc.) The trigger that uses a mail action is clearly one that you also want to throttle!

pipe is another way to feed the log line to an external program. As you may expect, swatch will pipe a matched line into the command you feed it. This is similar, but not the same as using exec — my favorite.

exec will run an external program, and pass it any part of the matched log line, or the entire matched line. Now all of those ideas running around your brain can be made reality. Between pipe and exec, you can pretty much do anything. exec can also pass along variables to the program it is calling. Passing $* will pass the entire log line. $N will pass the "Nth" field of the log line. Here's a good example swatchrc using these statements:

watchfor /error/i
exec /Users/Shared/fs.sh err $* 
exec echo $* | cat >> /Users/Shared/err.txt
continue
throttle 0:60:0
watchfor /warning/i
exec /usr/local/bin/flagdb.sh $*
continue
throttle 12:0:0

Some new elements here. Notice that we're using the "i" flag on the watchfor regex. This makes the match case insensitive. This will match "ERROR", "error", "Error", or any combination of upper and lower case. When we find "error", we run two actions — nothing wrong with that. Also, note the use of "continue" — without it, a matched pattern triggers the actions in its block and then stops. "continue" tells swatch to keep looking for pattern matches in the current line. The final 'new' thing here is that the second block uses a twelve-hour throttle. Again, nothing wrong with setting a wide time to suppress duplicate actions, however, make sure what you're setting is appropriate. If you're running a script that takes corrective action based on the pattern it finds, lower might be better. You could always create two matches on the same pattern with different throttle values; one could open up wide for notifications, and the other could be lower and take action.

Think about exec for a minute and the options available to you. Shell scripts, AppleScript code, perl or php that interacts with a database...anything, really. If you really want to impress your date, tie swatch in with Growl using growlnotify, and set Growl to forward (sticky) notifications to machines on the LAN...that's a hot combination!

It's what you don't know

A quick note on an interesting swatch directive: ignore. Often, you know what errors in your logs look like. Sure, you want to be notified about them. But, what about log entries you've never seen before? Swatch requires that you already be somewhat familiar with the contents of the log you're monitoring. Since swatch is supposed to get us away from staring at a log all day, what can be done? Figure out what you know, and ignore it.

The ignore statement is the exact opposite of watchfor. If there's a log file that you're pretty comfortable with, and just want to look for anomalies, use swatch to ignore all of the items that you already know, and alert you when something new to you crops up. There are no actions for ignore, however, swatch stops processing the log line when it sees a matching ignore, so, put these at the beginning of your swatchrc file.

The Big Picture

The best thing about swatch may be its ability to monitor your entire organization. How? Remember, we're just feeding and watching syslog. A few months ago, MacTech published an article showing how you can have all of your machines log to a central Tiger server (which is now on-line at http://www.mactech.com/articles/mactech/Vol.21/21.09/ TipsNTidbits/). If this were any other Unix variant, I'd also have to tell you to create a new log that captures all entries. However, readers from last month will remember — we're special. We use OS X. We have the Apple System Log.

Use the "Centralized Device Logging" article (linked above) to allow syslogd to accept remote log entries. From there, you can set any syslog enabled device to send their log entries to it. This includes Mac clients, other servers (including Windows servers, using third-party software), hardware appliances such as firewalls, and more. Now, you can be alerted to warnings and errors (or whatever you please, really) from any device on your network, and potentially act on it. That's power!

Start 'Em Up

Naturally, we don't want to have start a program like swatch manually, or have to tie up a console or ssh session to have it run. Again, OS X users are special, and we can create a launchd item to have swatch run at every boot. This is where, though, we get back to style. For a daemon that will run at boot and be able to manage the machine, rather than a single user, the configuration file belongs in a more universal location, like "/etc".

Using your text editor of choice, or Lingon, create the following file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
¬ "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Disabled</key> <false/> <key>Label</key> <string>com.radiotope.swatch</string> <key>OnDemand</key> <false/> <key>ProgramArguments</key> <array> <string>/usr/bin/swatch</string> <string>-c</string> <string>/etc/swatchrc</string> <string>-t</string> <string>/var/log/asl.log</string> </array> <key>RunAtLoad</key> <true/> </dict> </plist>

Save it as /Library/LaunchDaemons/net.sourceforge.swatch.plist. Note that, as mentioned, this launchd plist will run swatch and have it look for a config file in /etc named "swatchrc", and it will monitor the Apple System Log at /var/log/asl.log.

Conclusion

Watching logs is an incredibly important activity for the health and security of your network and networked machines. However, as humans, there's only so long we can keep our eyes on scrolling text. Plus, since logs typically contain both warnings, errors and notices ("good" information), it can quickly lead to information overload. Using a tool like swatch allows us to be alerted to the things that we deem important.

As noted last month, just about every subsystem logs. However, not every system logs using the system log, and may just drop a file somewhere else in the system (like Samba and Apache). Feel free to run a second instance of swatch with a custom swatchrc to monitor a second log file. Also, check the swatch man page one you have it installed. You can run swatch in batch mode — not even tailing a file in 'real-time,' change the line separator, create a stand-alone script, have swatch restart automatically (to pickup rolled log files), use an alternate tail application (which can resolve the former issue) and even create custom actions.

I was really thrilled to meet many readers at MacWorld this year. It really was a great week — I learned a lot, and really had my confidence in the Mac platform boosted even higher. Nowadays, there's just no reason not to buy a Mac (on the client side...servers are subject to more variables). Now, we have other Mac-related events to attend during the year (WWDC, Mac Networkers Retreat, SoCal Mac Fair and more).

Media of the month: Battlestar Galactica Season 1 DVD set. SciFi enough to satisfy the geek in you, and well written enough to keep you significant other watching, too. If you haven't been watching, check out the mini-series and season 1.

So, stop watching your logs...but make sure you're monitoring them! See you next month!

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

BetterTouchTool 2.332 - Customize multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
Capture One 11.0.1.40 - RAW workflow sof...
Capture One is a professional RAW converter offering you ultimate image quality with accurate colors and incredible detail from more than 400 high-end cameras -- straight out of the box. It offers... Read more
Capture One 11.0.1.40 - RAW workflow sof...
Capture One is a professional RAW converter offering you ultimate image quality with accurate colors and incredible detail from more than 400 high-end cameras -- straight out of the box. It offers... Read more
GraphicConverter 10.5.4 - $39.95
GraphicConverter is an all-purpose image-editing program that can import 200 different graphic-based formats, edit the image, and export it to any of 80 available file formats. The high-end editing... Read more
Dash 4.1.3 - Instant search and offline...
Dash is an API documentation browser and code snippet manager. Dash helps you store snippets of code, as well as instantly search and browse documentation for almost any API you might use (for a full... Read more
Microsoft OneNote 16.9 - Free digital no...
OneNote is your very own digital notebook. With OneNote, you can capture that flash of genius, that moment of inspiration, or that list of errands that's too important to forget. Whether you're at... Read more
DEVONthink Pro 2.9.17 - Knowledge base,...
Save 10% with our exclusive coupon code: MACUPDATE10 DEVONthink Pro is your essential assistant for today's world, where almost everything is digital. From shopping receipts to important research... Read more
OmniGraffle 7.6 - Create diagrams, flow...
OmniGraffle helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use Graffle to... Read more
iFinance 4.3.7 - Comprehensively manage...
iFinance allows you to keep track of your income and spending -- from your lunchbreak coffee to your new car -- in the most convenient and fastest way. Clearly arranged transaction lists of all your... Read more
Opera 50.0.2762.58 - High-performance We...
Opera is a fast and secure browser trusted by millions of users. With the intuitive interface, Speed Dial and visual bookmarks for organizing favorite sites, news feature with fresh, relevant content... Read more

Latest Forum Discussions

See All

Around the Empire: What have you missed...
Around this time every week we're going to have a look at the comings and goings on the other sites in Steel Media's pocket-gaming empire. We'll round up the very best content you might have missed, so you're always going to be up to date with the... | Read more »
Everything about Hero Academy 2: Part 4...
In this part of our Hero Academy 2 guide, we're going to have a look at some of the tactics you're going to need to learn if you want to rise up the ranks. We're going to start off slow, then get more advanced in the next section. [Read more] | Read more »
All the best games on sale for iPhone an...
Another week has flown by. Sometimes it feels like the only truly unstoppable thing is time. Time will make dust of us all. But before it does, we should probably play as many awesome mobile videogames as we can. Am I right, or am I right? [Read... | Read more »
The 7 best games that came out for iPhon...
Well, it's that time of the week. You know what I mean. You know exactly what I mean. It's the time of the week when we take a look at the best games that have landed on the App Store over the past seven days. And there are some real doozies here... | Read more »
Popular MMO Strategy game Lords Mobile i...
Delve into the crowded halls of the Play Store and you’ll find mobile fantasy strategy MMOs-a-plenty. One that’s kicking off the new year in style however is IGG’s Lords Mobile, which has beaten out the fierce competition to receive Google Play’s... | Read more »
Blocky Racing is a funky and fresh new k...
Blocky Racing has zoomed onto the App Store and Google Play this week, bringing with it plenty of classic kart racing shenanigans that will take you straight back to your childhood. If you’ve found yourself hooked on games like Mario Kart or Crash... | Read more »
Cytus II (Games)
Cytus II 1.0.1 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0.1 (iTunes) Description: "Cytus II" is a music rhythm game created by Rayark Games. It's our fourth rhythm game title, following the footsteps of three... | Read more »
JYDGE (Games)
JYDGE 1.0.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.0 (iTunes) Description: Build your JYDGE. Enter Edenbyrg. Get out alive. JYDGE is a lawful but awful roguehate top-down shooter where you get to build your... | Read more »
Tako Bubble guide - Tips and Tricks to S...
Tako Bubble is a pretty simple and fun puzzler, but the game can get downright devious with its puzzle design. If you insist on not paying for the game and want to manage your lives appropriately, check out these tips so you can avoid getting... | Read more »
Everything about Hero Academy 2 - The co...
It's fair to say we've spent a good deal of time on Hero Academy 2. So much so, that we think we're probably in a really good place to give you some advice about how to get the most out of the game. And in this guide, that's exactly what you're... | Read more »

Price Scanner via MacPrices.net

Deals on clearance 15″ Apple MacBook Pros wit...
B&H Photo has clearance 2016 15″ MacBook Pros available for up to $800 off original MSRP. Shipping is free, and B&H charges NY & NJ sales tax only: – 15″ 2.7GHz Touch Bar MacBook Pro... Read more
Apple restocked Certified Refurbished 13″ Mac...
Apple has restocked a full line of Certified Refurbished 2017 13″ MacBook Airs starting at $849. An Apple one-year warranty is included with each MacBook, and shipping is free: – 13″ 1.8GHz/8GB/128GB... Read more
How to find the lowest prices on 2017 Apple M...
Apple has Certified Refurbished 13″ and 15″ 2017 MacBook Pros available for $200 to $420 off the cost of new models. Apple’s refurbished prices are the lowest available for each model from any... Read more
The lowest prices anywhere on Apple 12″ MacBo...
Apple has Certified Refurbished 2017 12″ Retina MacBooks available for $200-$240 off the cost of new models. Apple will include a standard one-year warranty with each MacBook, and shipping is free.... Read more
Apple now offering a full line of Certified R...
Apple is now offering Certified Refurbished 2017 10″ and 12″ iPad Pros for $100-$190 off MSRP, depending on the model. An Apple one-year warranty is included with each model, and shipping is free: –... Read more
27″ iMacs on sale for $100-$130 off MSRP, pay...
B&H Photo has 27″ iMacs on sale for $100-$130 off MSRP. Shipping is free, and B&H charges sales tax for NY & NJ residents only: – 27″ 3.8GHz iMac (MNED2LL/A): $2199 $100 off MSRP – 27″ 3.... Read more
2.8GHz Mac mini on sale for $899, $100 off MS...
B&H Photo has the 2.8GHz Mac mini (model number MGEQ2LL/A) on sale for $899 including free shipping plus NY & NJ sales tax only. Their price is $100 off MSRP. Read more
Apple offers Certified Refurbished iPad minis...
Apple has Certified Refurbished 128GB iPad minis available today for $339 including free shipping. Apple’s standard one-year warranty is included. Their price is $60 off MSRP. Read more
Amazon offers 13″ 256GB MacBook Air for $1049...
Amazon has the 13″ 1.8GHz/256B #Apple #MacBook Air on sale today for $150 off MSRP including free shipping: – 13″ 1.8GHz/256GB MacBook Air (MQD42LL/A): $1049.99, $150 off MSRP Read more
9.7-inch 2017 WiFi iPads on sale starting at...
B&H Photo has 9.7″ 2017 WiFi #Apple #iPads on sale for $30 off MSRP for a limited time. Shipping is free, and pay sales tax in NY & NJ only: – 32GB iPad WiFi: $299, $30 off – 128GB iPad WiFi... Read more

Jobs Board

*Apple* Retail - Multiple Positions - Apple,...
Job Description:SalesSpecialist - Retail Customer Service and SalesTransform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Data Center Site Selection and Strat...
# Apple Data Center Site Selection and Strategy Research Analyst Job Number: 83708609 Santa Clara Valley, California, United States Posted: 18-Jan-2018 Weekly Hours: Read more
Security Engineering Coordinator, *Apple* R...
# Security Engineering Coordinator, Apple Retail Job Number: 113237456 Santa Clara Valley, California, United States Posted: 18-Jan-2018 Weekly Hours: 40.00 **Job Read more
Firmware Engineer - *Apple* Accessories - A...
# Firmware Engineer - Apple Accessories Job Number: 113422485 Santa Clara Valley, California, United States Posted: 18-Jan-2018 Weekly Hours: 40.00 **Job Summary** Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.