TweetFollow Us on Twitter

On Logs

Volume Number: 23 (2007)
Issue Number: 01
Column Tag: Mac in the Shell

On Logs

Everything you need to know about your system

by Edward Marczak

Introduction

Log files are the heartbeat of your system. You can track what's going on in good times and bad - the latter hopefully pointing you to a way to make things right again. When I say 'heartbeat', however, I should be clear that it's really many hearts. All of the child processes jabbering away, speaking their part. This article aims to teach you how to read logs, show examples, explain the various log contents, and give some logging tips and tricks.

Log in the Shell?

Now, logs may not be a shell topic per se, they're often associated with work in the shell, and perhaps for good reason. On an OS X box, you have the excellent Console.app (/Applications/Utilities/Console.app). It gives you quick access to most important log files, provides quick searching capabilities, and will automatically decompress gzip-rolled logs. Plus, you don't need to open a shell just to access your logs, if that sort of thing bugs you. Additionally, on an OS X Server, the Server Admin application accesses various logs depending on context. While Server Admin is useful for a quick peek, it uses an HTTP call to pull down the log files at appropriate intervals. Due to this, it's not a good way to watch a file fill up quickly. This brings us back to the shell.

Longtime readers of this column already know where I'm heading with this: we're going to run Terminal.app and be looking at logs through a command-line interface. So, open it up now (/Applications/Utilities/Terminal.app).

Log Central

How do entries in log files appear? This may seem to be a simple question, but it isn't. A log is simply a text file. This text file runs in chronological order, with new entries appended to the end. Any application is certainly capable of performing this on its own, and sometimes, that is the case. More commonly, though, log entries will be logged to the syslog daemon, which in turn is responsible for writing the entries to the proper log or logs. Interestingly, but perhaps not surprisingly, Apple has their own logging facility intended to replace syslogd. In fact, the little secret here is that syslogd is the Apple System Log in disguise.

The Apple System Log, or 'asl' is a syslog-compatible replacement for standard syslog (which deviously is named 'syslogd'). Fortunately for everyone involved, you can still treat it just like syslog, which I will tend to do here.

Programs use an API to write log entries to the syslogd server. As the suffix may clue you in to, syslogd runs as a daemon in the background. As of 10.4, it's initially run, and subsequently controlled from launchd. Of course, applications are free to just drop a text file somewhere and write to it.

Fluency

Let's take a look at a typical log entry. We'll start out easy:

Nov  6 06:54:40 Jack-Kerouak lookupd[1775]: lookupd 
(version 369.5) starting - Mon Nov 6 06:54:40 2006

This is, of course, one long line, but you should get used to it. Often, log lines will wrap - on paper and in your terminal. Each line is read left to right, as you'd expect. Let's pick this entry apart:

Nov 6 06:54:40 - The date and time that this entry was logged. Do note that this is a 24-hour clock ("military time"). Also, read on for a note about GMT entries.

Jack-Kerouac - The hostname of the system that logged this entry. Jack-Kerouac is the name of my PowerBook as I'm often 'On the Road'.

lookupd[1775] - The name and process id or pid that sent the message to the log.

lookupd (version 369.5) starting - Mon Nov 6 06:54:40 2006 - The message itself. This is a simple, "hey, I'm starting up" message.

That was easy, right? The time is an obvious one: you need this to figure out the sequence of events, or, correlate log events with some other happening - be it electronic or real-world (like a power outage, etc.). Why do we need the hostname? If we're looking at the entry in a log on a particular system, it came from that system, right? End of story, no? Fortunately, no. syslogd can be set up to accept log entries from other machines and devices. This can be incredibly useful, but only if you know the host that's creating the message!

The next entry is the process name and process id that sent this message. While the process can set any name it likes, the syslog daemon is responsible for logging the pid that sent the message. Having the pid is interesting: some processes are long-lived, and you can easily correlate a log entry with a pid. Some processes are very ephemeral, and will have wisped away before you can hunt them down based on their pid. Also, if you're looking at an aged log, don't rely on the pid! Process IDs come and go, and may be recycled. Don't look at yesterday's log and try to line up events with pids - even from the same process. These entries may be hints, but until you have good evidence, they should be considered suspect.

Finally, we have the message, or, what the program is trying to tell you! Of course, this depends on the situation. In my example, lookupd is just letting us know that it is starting up. It's simply informational.

Who's on First?

Where does one find logs to watch? On a typical Unix system, you'll find all of your logs under one neat hierarchy: /var/log. As is typical for Apple, however, you'll find all of your Unixy logs in /var/log as you may expect, Apple-fied logs under /Library/Logs, and user space logs in each user's ~/Library/Logs directory. Wow. That's a lot of logs.

My example entry above came from /var/log/system.log. system.log is as close to the heartbeat of the system as possible with one log. Consider it the log. Of course, that viewpoint depends on what you're trying to troubleshoot. That said, I do like leaving system.log up in my face as much as possible. It's a great way to pick out issues before they cause havoc.

Which Way Did They Go?

For apps that log through syslog, how does syslog know in which logs to write each message? For that, you need to consult /etc/syslog.conf. This plain-text configuration file for syslog follows some very standard conventions. Lines beginning with a hash ("#") are ignored, and don't contribute to the configuration. The "*" is a wild-card, and can match multiple facilities. Repeating designations and specifying different log files.

One of my first articles for MacTech detailed syslogd, its conventions, how to get it to accept remote messages and how to get it to write messages to various log files. It's now on-line at http://www.mactech.com/articles/
mactech/Vol.20/20.12/UnixTipsForOSX/index.html
. One of the few things in that article that needs updating for 10.4 is how to get syslogd to allow remote messages to be logged. We had an article earlier this year that details this procedure, so I won't go into the detail that he did. For the sake of completeness, though, I'll give you the walkthrough. Remember: you do not need to do this to follow this article. You would typically only do this on a server that is up 24x7 and will centralize logs for an entire LAN or enterprise. With that out of the way:

1. Edit/System/Library/LaunchDaemons/
com.apple.syslogd.plist

2. Add "-u" to the ProgramArguments key, making it look like this:

   <key>ProgramArguments</key>
               <array>
                        <string>/usr/sbin/syslogd</string>
                        <string>-u</string>
               </array

3. Save the file

4.Reload syslogd:

# launchctl unload
/System/Library/LaunchDaemons/com.apple.syslogd.plist
# launchctl load
/System/Library/LaunchDaemons/com.apple.syslogd.plist

5. While that gets it going immediately, something wreaks havoc with it during the nightly cron routines. To counteract this, simply unload and load syslog each night in a script. Create /etc/daily.local and add the following lines:

launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
sleep 2
launchctl load 
/System/Library/LaunchDaemons/com.apple.syslogd.plist

6. Save this file.

7. Mark /etc/daily.local executable:

sudo chmod 744 /etc/daily.local                                      

8. Now you're done!

Again, this is just the how. For the 'why', please reference Aaron's article.

Watching the Detectives

So, how does one watch a log? As alluded to earlier, there are several ways. There's the very excellent Console.app. Of course, earlier, I listed reasons why I prefer the shell for such a task, so that's what I'll be covering. There are three distinct programs that let you follow what's going on with a log file.

First, there's the ubiquitous tail. tail displays the last lines of a file - 10 by default. More importantly, you can use the '-f' switch to have tail follow the output of a log file in "real-time". I put that in quotes simply because there's always a delay between when an app performs an action and posts a log entry, to the time when it actually shows up. That said, tail is about the best you can do. A listing in print will just not do this justice: you need to see it in action yourself. So, please, open an admin shell, and run:

tail -f /var/log/system.log

You'll initially see the last 10 lines of system log, and then, as new entries are posted, the display will add them to the listing.

Unfortunately, the OS X version of tail is missing a critical feature: following multiple files at once (you can, however, simply tail the last line of multiple files). Every other version of tail I've ever worked with could do this: Solaris, GNU/Linux and yes, even FreeBSD 6. Fortunately, if you're a fink or Mac Ports user (neé Darwin Ports), you can install the excellent multitail. multitail makes up for the lack of features in the current tail. multitail has some other nice features, too. I do find the syntax a little cumbersome, though. After you've run 'fink install multitail' or 'port install multitail', go check it out with:

multitail -I /var/log/system.log -I /var/log/mail.log

Very, very handy.

Another program that's great for viewing log files is less. Now, most people think of less as a better more, or vi-without-the-editing. Did you know that less can also follow a file? One thing that I love about using less to follow a file is highlighting. Let's say you're troubleshooting an e-mail issue for one user. On a busy server, the mail log fills up and scrolls away pretty quickly. Here's what you do: Run less /var/log/mail.log. This just brings mail.log into the buffer of less. Search on the user's login id. For example, you'd type /stevej (that's forward slash, then "stevej"). less will then jump to the first occurrence of "stevej" and highlight it. Something like this line:

Nov 6 05:16:45 xtc postfix/smtpd[4817]: 59D955B7E49:
client=topgun.apple.com[12.34.56.234], sasl_method=CRAM-MD5,
sasl_username=stevej

Now, type F (that's a capital F). less will tell you that it is "Waiting for data... (interrupt to abort)" and start following the file, displaying new lines as they're added to it. Interrupt is control-c. As new lines are added, any occurrences of "stevej' will also be highlighted. This can be of tremendous use in spotting a line you're looking for. If you do see something useful scroll by, just hit command-c, and then use the arrow keys or control-b (back one page), and control-f (forward one page) to navigate. When you're ready to follow again, just press shift-F.

The last way that I'm going to describe is an Apple-specific method. As I mentioned, syslogd in OS X is actually an Apple creation more properly called asl, the Apple System Log. In all other syslogd implementations you find, a message comes into syslogd, gets processed and ultimately written to the right file - end of story. asl does this, too, but then takes the extra step of writing all entries to a backing store. In this case, /var/log/asl.log, but this may change in future releases. This new syslogd-is-really-asl setup appeared in 10.4, so, don't go trying this on your Panther box.

That explanation is leading me to this: the syslog program. Note the lack of a trailing 'd', and the fact that you'll only find this on OS X machines. OK, try it: type syslog and press return. You'll get a screen full of log entries that looks like you just typed cat /var/log/system.log. So, what's the point? The syslog program can follow the entries, similar to less. The "-w" switch will wait for new data, and display it as it arrives. It'd be nice if, like tail, we saw a few previous entries as well, but syslog stoically sits there, cursor on the line below. OK, I can do this with less, what else?

Remember that backing store? Entries are actually written in there in a raw, key/value pair listing. To see that format, we override the default 'standard' format using the '-F' (capital F) switch. Try typing syslog -F raw. Looks a little different this time, doesn't it? You can also use the '-F' switch to define a custom format for your purposes - no awk required! Try this:

syslog -F '$Time $(Sender)[$(PID)]: $Message'

This basically just drops the hostname from the standard format. Useful, if the log you're viewing could only be from one host, and you want to focus on the other details.

While not quite as good as multitail, by default, syslog reports on anything, arriving via the syslog mechanism. This is very useful in conjuction with the -w switch. Of course, unlike multitail, you cannot choose to watch arbitrary files.

Overall, Apple's asl implementation gets more interesting the deeper you dive into it. It can filter the outout according to your criteria:

syslog -k Sender postfix

...to see only messages generated by postfix. It also has a full POSIX API, allowing apps to log to and search the asl data store.

All of that said, I rarely, if ever, use syslog to watch logs. Give me the flexibility and portability of tail, grep and less - that's pretty much all you need. Add to it the simple unfamiliarity, newness and slight clumsiness of syslog, well... perhaps we'll see more in 10.5.

What Now?

What about all of those other logs I pointed out earlier. Take a look in /Library/Logs. You should see something similar to the following:

drwxrwxr-x  3 root  admin  102 Apr 28  2005 AppleFileService
drwxrwxr-x 13 root  admin  442 Jul 17 10:02 Console
drwxrwxr-x  9 root  admin  306 Nov  5 08:07 CrashReporter
drwxrwxr-x 14 root  admin  476 Nov  8 17:19 DirectoryService
drwxr-xr-x  3 root  admin  102 Sep  7 15:48 NetUpdate
drwxrwxr-x  6 root  admin  204 Oct 19  2005 Qmaster
-rw-rw-rw-  1 root  admin 6653 Nov  2 15:04 Software Update.log
-rw-r-r-    1 root  admin 6910 Oct 19 10:21 panic.log

(If you also have a memberd_dump.log, it's because you, or someone on your machine, ran memberd -l at some point). All of these entries should look relatively familiar. The thing to note is that many are directories, each containing multiple, though related, log files. These files are ideal candidates to view via tail or less.

When you start going through these logs, you'll find that some logs are more useful than others. Sometimes, but not always, you can make a seemingly not-useful log very useful! How?

Some programs let you set the level of logging verbosity. Here's where people unfamiliar with Unix are going to say, "see! That's why I hate Unix!" The fact of the matter is, a) it's up to the application, how much or how little it will log, and if it will let you alter that amount, and b) each app handles this in its own way. There is no single mechanism that allows you to alter the logging level for any given app.

If you're using OS X Server, Server Admin will let you alter the log level of most services. Key here being most. If you're using plain 'ol OS X, you have to do it yourself. Doing it by hand can be better in either case as many programs have logging options that Server Admin won't let you set anyway.

Samba is a great example of many of the topics that we've discussed so far: it can log via syslog, its own mechanism or both. Additionally, you can configure the detail of the logs from minimal to very detailed. You can get more information right on your own disk at file:///usr/share/swat/using_samba/ch06.html, but here's the gist: In samba's /etc/smb.conf file, the following entries configure logging:

log level: sets the detail of the log. Ranges from

1(minimal) to 10 (extreme!).

log file: Where to store the logs file on disk.

max log size: Max size in Kb before samba rolls the log.

debug timestamp: Show time stamps in the logs? Values

of "yes" or "no".

syslog: Send log entries to syslog? Value is a number that

represents the syslog level - values of or below the

number you specify are sent to syslog. Anything higher is

only sent to the samba logs.

syslog only: says what it does, does what it says. Set to

"yes", you won't see any entries in the samba logs.

By default with OS X, samba does not log through syslog. This is the primary reason I stick with tail, grep, and less as my log analysis tools of choice. Samba also does something pretty cool with its config file: while most programs require a reload to pick up alterations, changes in /etc/smb.conf file are picked up every 60 seconds - no reload required! (This can be dangerous too - always work on a copy of smb.conf, check it with testparm, keep the good one on hand just in case, and then move the copy into place.)

My other favorite non-system.log log is the debug log generated by DirectoryServices (although DirectoryServices does log via asl). I mentioned this in the August 2006 Mac in the Shell column, but it bears repeating. Find Directory Service's logs at /Library/Logs/DirectoryService. To increase the verbosity, send the DirectoryService process a USR1 signal: killall -USR1 DirectoryService. Now have a gander at your /Library/Logs/DirectoryService/DirectoryService.server.log file (or, use syslog -k Sender DirectoryService - perhaps even with the -w flag).

Conclusion

Understanding how to read and monitor log files can make troubleshooting easier. More importantly, it is a harbinger of issues, before your users complain. For something that's just baked into me at this point, I tend to take logs for granted (just a little). Even I didn't realize that this topic was so deep and that I'd write this much. Hopefully, the importance of logs, and ways you can make your life easier with them is much more clear now. There are even a few things I just couldn't touch on this month regarding logs, as they really deserve more attention. So, next month (to my surprise), I'll be discussing my favorite log monitoring utilities. You can't stare a monitor watching logs 24x7, right? But, a computer can.

Media of the month: Neal Stephenson's Cryptonomicon. An 'oldie' (1999) but goodie. Now that it comes in paperback, it's even that much easier to read! Despite the Neal Stephenson book-ending-curse, Cryptonomicon is a great read, incredibly prescient; you'll recognize many of the elements, and ultimately learn something (especially if you continue on to the Bruce Schneier penned appendix!).

January! Happy New Year, and Happy MacWorld! If you're attending the show, please stop by the MacTech booth! If you picked this issue up while at the booth, stop back often and please say hello to the MacTech staff and authors that will be doing the same.


Ed Marczak owns and operates Radiotope, a technology consultancy that brings enterprise solutions to small and medium-sized businesses. Outside of this piece of the puzzle, he is Executive Editor of MacTech Magazine, a husband and father, and CTO of WheresSpot, among other things. Find the missing tech piece at http://www.radiotope.com

 
AAPL
$102.50
Apple Inc.
+0.25
MSFT
$45.43
Microsoft Corpora
+0.55
GOOG
$571.60
Google Inc.
+2.40

MacTech Search:
Community Search:

Software Updates via MacUpdate

Path Finder 6.5.5 - Powerful, award-winn...
Path Finder is a file browser that combines the familiar Finder interface with the powerful utilities and innovative features. Just a small selection of the Path Finder 6 feature set: Dual pane... Read more
QuarkXPress 10.2.1 - Desktop publishing...
With QuarkXPress, you can communicate in all the ways you need to -- and always look professional -- in print and digital media, all in a single tool. Features include: Easy to Use -- QuarkXPress is... Read more
Skype 6.19.0.450 - Voice-over-internet p...
Skype allows you to talk to friends, family and co-workers across the Internet without the inconvenience of long distance telephone charges. Using peer-to-peer data transmission technology, Skype... Read more
VueScan 9.4.41 - Scanner software with a...
VueScan is a scanning program that works with most high-quality flatbed and film scanners to produce scans that have excellent color fidelity and color balance. VueScan is easy to use, and has... Read more
Cloud 3.0.0 - File sharing from your men...
Cloud is simple file sharing for the Mac. Drag a file from your Mac to the CloudApp icon in the menubar and we take care of the rest. A link to the file will automatically be copied to your clipboard... Read more
LibreOffice 4.3.1.2 - Free Open Source o...
LibreOffice is an office suite (word processor, spreadsheet, presentations, drawing tool) compatible with other major office suites. The Document Foundation is coordinating development and... Read more
SlingPlayer Plugin 3.3.20.505 - Browser...
SlingPlayer is the screen interface software that works hand-in-hand with the hardware inside the Slingbox to make your TV viewing experience just like that at home. It features an array of... Read more
Get Lyrical 3.8 - Auto-magically adds ly...
Get Lyrical auto-magically add lyrics to songs in iTunes. You can choose either a selection of tracks, or the current track. Or turn on "Active Tagging" to get lyrics for songs as you play them.... Read more
Viber 4.2.2 - Send messages and make cal...
Viber lets you send free messages and make free calls to other Viber users, on any device and network, in any country! Viber syncs your contacts, messages and call history with your mobile device,... Read more
Cocktail 7.6 - General maintenance and o...
Cocktail is a general purpose utility for OS X that lets you clean, repair and optimize your Mac. It is a powerful digital toolset that helps hundreds of thousands of Mac users around the world get... Read more

Latest Forum Discussions

See All

Rhonna Designs Magic (Photography)
Rhonna Designs Magic 1.0 Device: iOS Universal Category: Photography Price: $1.99, Version: 1.0 (iTunes) Description: Want to sprinkle *magic* on your photos? With RD Magic, you can add colors, filters, light leaks, bokeh, edges,... | Read more »
This Week at 148Apps: August 25-29, 2014
Shiny Happy App Reviews   | Read more »
Qube Kingdom – Tips, Tricks, Strategies,...
Qube Kingdom is a tower defense game from DeNA. You rally your troops – magicians, archers, knights, barbarians, and others – and fight against an evil menace looking to dominate your kingdom of tiny squares. Planning a war isn’t easy, so here are a... | Read more »
Qube Kingdom Review
Qube Kingdom Review By Nadia Oxford on August 29th, 2014 Our Rating: :: KIND OF A SQUARE KINGDOMUniversal App - Designed for iPhone and iPad Qube Kingdom has cute visuals, but it’s a pretty basic tower defense game at heart.   | Read more »
Fire in the Hole Review
Fire in the Hole Review By Rob Thomas on August 29th, 2014 Our Rating: :: WALK THE PLANKUniversal App - Designed for iPhone and iPad Seafoam’s Fire in the Hole looks like a bright, 8-bit throwback, but there’s not enough booty to... | Read more »
Alien Creeps TD is Now Available Worldwi...
Alien Creeps TD is Now Available Worldwide Posted by Ellis Spice on August 29th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Dodo Master Review
Dodo Master Review By Jordan Minor on August 29th, 2014 Our Rating: :: NEST EGGiPad Only App - Designed for the iPad Dodo Master is tough but fair, and that’s what makes it a joy to play.   | Read more »
Motorsport Manager Review
Motorsport Manager Review By Lee Hamlet on August 29th, 2014 Our Rating: :: MARVELOUS MANAGEMENTUniversal App - Designed for iPhone and iPad Despite its depth and sense of tactical freedom, Motorsport Manager is one of the most... | Read more »
Motorsport Manager – Beginner Tips, Tric...
The world of Motorsport management can be an unforgiving and merciless one, so to help with some of the stress that comes with running a successful race team, here are a few hints and tips to leave your opponents in the dust. | Read more »
CalPal Update Brings the App to 2.0, Add...
CalPal Update Brings the App to 2.0, Adds Lots of New Stuff Posted by Ellis Spice on August 29th, 2014 [ permalink ] | Read more »

Price Scanner via MacPrices.net

Apple now offering refurbished 21-inch 1.4GHz...
The Apple Store is now offering Apple Certified Refurbished 21″ 1.4GHz iMacs for $929 including free shipping plus Apple’s standard one-year warranty. Their price is $170 off the cost of new models,... Read more
Save $50 on the 2.5GHz Mac mini, on sale for...
B&H Photo has the 2.5GHz Mac mini on sale for $549.99 including free shipping. That’s $50 off MSRP, and B&H will also include a free copy of Parallels Desktop software. NY sales tax only. Read more
Save up to $300 on an iMac with Apple refurbi...
The Apple Store has Apple Certified Refurbished iMacs available for up to $300 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free. These are the best prices on... Read more
The Rise of Phablets
Carlisle & Gallagher Consulting Group, a businesses and technology consulting firm focused solely on the financial services industry, has released an infographic depicting the convergence of... Read more
Bad Driver Database App Allows Good Drivers t...
Bad Driver Database 1.4 by Facile Group is a new iOS and Android app that lets users instantly input and see how many times a careless, reckless or just plain stupid driver has been added to the... Read more
Eddy – Cloud Music Player for iPhone/iPad Fre...
Ukraine based CapableBits announces the release of Eddy, its tiny, but smart and powerful cloud music player for iPhone and iPad that allows users to stream or download music directly from cloud... Read more
A&D Medical Launches Its WellnessConnecte...
For consumers and the healthcare providers and loved ones who care for them, A&D Medical, a leader in connected health and biometric measurement devices and services, has launched its... Read more
Anand Lal Shimpi Retires From AnandTech
Anand Lal Shimpi, whose AnandTech Website is famous for its meticulously detailed and thoroughgoing reviews and analysis, is packing it in. Lal Shimpi, who founded the tech site at age 14 in 1997,... Read more
2.5GHz Mac mini, Apple refurbished, in stock...
The Apple Store has Apple Certified Refurbished 2.5GHz Mac minis available for $509, $90 off MSRP. Apple’s one-year warranty is included, and shipping is free. Read more
13-inch 2.5GHz MacBook Pro on sale for $999,...
B&H Photo has the 13″ 2.5GHz MacBook Pro on sale for $999.99 including free shipping plus NY sales tax only. Their price is $100 off MSRP. Read more

Jobs Board

*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Senior Event Manager, *Apple* Retail Market...
…This senior level position is responsible for leading and imagining the Apple Retail Team's global event strategy. Delivering an overarching brand story; in-store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.