TweetFollow Us on Twitter

Mac In The Shell -- Which Log?

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Mac in the Shell

Mac In The Shell -- Which Log?

Following up on which log shows you what

by Edward Marczak

Introduction

I've had incredible response to my last two columns that talked about logs: what they are, how to interpret them and how to notify yourself if a log is telling you something important. More than any other question, however, people have asked, "which log does what? While I gave an overview of some logs, there are plenty more that I haven't gone into, and more locations for logs than I could describe previously. That's what I'll be following up on this month. So, read on for even more on logs.

Kinda I Want To

Just as a review for anyone who didn't read the previous columns, logs are text files that running programs write to that keep track of their activity. Text files, that's all. (OK, an app could keep a binary log, but for the most part, text it is). This allows other apps and, more importantly, humans to read their contents. Apps are free to deal with logging on their own, or, they can use syslog to hand off their data to the system logger. It's a bit of a style issue, and usage is roughly split on which method is chosen. There's nothing that says that an application can't do both. Let's look quickly again at the syslog method.

The system logger, aka "syslog, is a daemon run at boot that should be running all of the time. It listens for logging data locally, and knows where to put a logging entry based on its configuration file /etc/syslog.conf. "syslogd can also be setup to listen for logging data from other machines as well. Log data can be categorized by facility and level (a "selector). The facility is basically where the data is coming from ("mail, "ftp, etc., along with some generic facilities). The level describes the severity of the log message. The currently defined levels are:

Emergency (level 0) Alert (level 1) Critical (level 2) Error (level 3) Warning (level 4) Notice (level 5) Info (level 6) Debug (level 7)

(Don't you love it when things neatly fit into a byte?) Looking at /etc/syslog.conf, you can then see which messages will be directed to which files. Note that some will go to multiple files.

This is a bit of a review, as all of this (and more) was covered in previous articles in this column.

Down In It

Apple has created a little bit of a split with logging: "Unix-y files in /var/log, and everything else. While that may be a bit of an overgeneralization, it's a good general guideline.

syslogd, as you can see from /etc/syslog.conf, will dump just about everything in some log residing in /var/log. Additionally, several non-syslog sending services also drop their files somewhere into the /var/log hierarchy. The two big notables in this category would be apache logging into /var/httpd and samba, which logs into /var/samba. You'll also find some other non-syslog-ish files hanging around in /var/log, and we'll address those a little later.

The other place you'll find good logging information falls into the "everything else category. You'll find these in /Library/Logs or ~/Library/Logs. System processes will log to /Library/Logs and user processes, when the user home directory is writable, will log to ~/Library/Logs. For example, anything that needs admin level rights to run AFP logs, Software Update and, my favorite, Directory Services will log to the System directory /Library/Logs. Other user-space apps Logic, SyncServices, Parallels, etc. will log into the user's own Library/Logs.

If you've peeked into either of these directories, you'll notice that there's another category of logs: logs that get generated as the result of a crash.

Something I Can Never Have

For those of you obsessed with the running process list, you'll no doubt have noticed the persistent /usr/libexec/crashreporterd. Apple's crash reporter daemon hangs around waiting for an application to crash. Technically, it listens for a Mach exception to be generated, and, upon that happening, launches crashdump. crashdump then logs and reports the event to the user. If the user can be determined, and they have a writable home, the report goes into ~/Library/Logs/CrashReporter. You really forget how much stuff takes a dive until you peek in there:

Jack-Kerouac:~/Library/Logs/CrashReporter marczak$ ls -1
BOMArchiveHelper.crash.log
Cyberduck.crash.log
Dreamweaver.crash.log
Finder.crash.log
GrowlTunes.crash.log
Mail.crash.log
Now Up-To-Date.crash.log
Pages.crash.log
Parallels.crash.log
Preview.crash.log
Quicksilver.crash.log
Safari.crash.log
VirtueDesktops.crash.log
Workgroup Manager.crash.log
firefox-bin.crash.log
iSnip.crash.log
mdimportserver.crash.log
vmware.crash.log

Wow. When an app's owner can't be determined, is a system process, or the user's home directory is not writeable, crashdump logs into /Library/Logs:

Jack-Kerouac:/Library/Logs/CrashReporter root# ls -1 
ARDAgent.crash.log
Exited process.crash.log
httpd.crash.log
iSnip.crash.log
loginwindow.crash.log

LoginWindow crash...I like that one! crashreporter itself, however, logs its actions into /var/log/crashreporter.log. Finally, crashreporterd is also responsible for writing panic logs when the system is rebooted after a panic find the log at /Library/Logs/panic.log.

Terrible Lie

If you root around /var/log long enough, you'll even start to notice some other files that aren't syslog generated at all. The most notable of the bunch are daily.out, weekly.out and monthly.out. These log files are generated by the daily, weekly and monthly periodic jobs that run overnight (if your machine is up and running, but not necessarily logged in).

Also of note are wtmp and lastlog. Interestingly, these are binary log files, and must be read with another application. These are not present for troubleshooting per se, but instead track login activity. User IDs, along with their login time get written to lastlog. From there, the utmp file gets updated with the user login information, and the same utmp record gets written to /var/log/wtmp. users, w and who use utmp to provide their information, while last and ac use wtmp.

That's What I Get

So, back to the original question: which logs record which bit of information? Here's a 40,000 ft. view:

/var/log/system.log

Along with asl.log, these are the big kahunas. Most system related activity is logged here. Specifically, the following selectors log to system.log:

*.notice
authpriv,remoteauth,ftp,install.none
kern.debug
mail.crit

As mentioned in January, asl.log pretty much gets everything, and displays the facility and level with the log message.

/var/log/secure.log

Messages from Apple's SecurityServer get logged here along with anything in the authpriv facility. Good one to keep an eye on. Interestingly, though, some security-related information logs to system log as well (such as sshd authentication errors through PAM).

/var/log/mail.log and /var/log/mailaccess.log

Complimentary logs that describe mail (if you're running OS X Server's built-in mail OS X client' also uses /var/log/mail.log). mail.log contains activity for postfix, SMTP sending and receiving while mailaccess.log contains cyrus' imap and pop activity. mail.log is the place to look when clients are telling you that they sent mail that the intended recipients never received (mail going out), or that someone sends them e-mail which they themselves never received (mail coming in).

/Library/Logs/DirectoryService/*

Huge. HUGE I say! Anyone operating in an OD, AD or other LDAP-reliant environment should be checking here occasionally. Of course, if you're trying to bind a workstation to a directory, and it's just not working for some reason, this is the place to check. Don't forget to throw DirectoryService into debug mode (killall -USR1 DirectoryService, logging to DirectoryService.debug.log), or use boot-time debugging (touch /Library/Preferences/DirectoryService/.DSLogDebugAtStart which will also log to DirectoryService.debug.log) if the situation calls for it covered in detail in past Mac In The Shell articles.

Logs that you don't need explained

There are some logs that simply don't need any explanation. These include:

/var/log/ftp.log (contents of the ftp server)
/var/log/httpd/* (Apache's log files)
/var/log/samba/* (Samba's log files)
/var/log/lpr.log (lpr printing activity)
/var/log/cups/* (CUPS web sever activity)
/Library/Logs/Software Update.log (Software Update install history)

/Library/Logs/AppleFileService OK, perhaps this one does need some explanation. But not by me from the engineers at Apple. This should be the Mecca of information that you reach for when you want to track activity coming in through AFP. However, details in this log are so sparse to render it useless. Despite the fact that AppleFileServer runs as one grand, monolithic process, we need it to give up the goods on what's happening internally in these logs.

The mega-log

Remember, that you can always have all syslog messages be delivered to a single file by using a wildcard selector in /etc/syslog.conf:

*.*      /var/log/mega.log

…although asl.log is already catching most of that. Further remember that this will not capture everything else that is performing logging on its own and not using syslog/asl.

The Only Time

While I made a strong case for watching and examining logs using good ol Unix tools grep, tail and less and I would still make that assertion, there are some other very useful tools out there.


Figure 1: Splunk searching for events (detail)

Console.app is the most obvious for us Mac users; it's "built-in and clearly Mac-like. I still think it's a miserable way to follow log activity as it happens, but it is a fantastic tool for exploring the various logs that exist on the system. Fire up Console.app (located in the Utilities folder) and poke around. It's a great learning experience.

I've found a newer tool on the market to also be very useful, and while it's not Mac-specific, it's nice that there's an OS X version at all. Splunk is billed as a logfile search engine. It comes in a free version that will sift through up to 500MB of logs per day, and a paid version which not only removes that restriction altogether, but also adds Splunk-to-Splunk logging for log correlation across many machines, and also authentication which is missing in the free version.

Splunk alone could take up an entire column (and may very well one day soon); it is so easy to get going, that I'd recommend the download. Excellent documentation exists on the site as well. Of course, Splunk makes it really easy to search for events, but I've found that it's a nice exploratory tool in general. Splunk quickly categorizes events and can let you filter on those events. Figure 1 shows the search engine, running in a browser, looking at event type 9 on my machine.

Even cooler, perhaps, is the ability to look at the frequency of events. Figure 2 shows the frequency of events that appear in the secure.log of my machine clearly a good one to keep an eye on and reign in this kind of data.


Figure 2 Splunk showing the frequency of various events that appear in a given source.

In a small-to-medium sized organization, you could set up a single server as a log host (check the prior logging articles for instructions) and have Splunk access that log. 500MB is a lot of logging information. Enough that you shouldn't have a problem using the free version of Splunk.

Sanctified

So here's the real story with logs: you can ignore them, sure. Then when a problem crops up, you can just tell your client/employer that this is just the way computers are. There are bugs, there are problems; such is life. If you watch the logs, you'll know there's a problem, and then you'll actually have to do something about it.

Or, you can realize that this is the only way the system can speak to you and more often that not, you're warned well in advance of any catastrophic problems. You can head these off at the pass. You can keep the system providing services without interruption and keep up with real work.

Of course, the unfortunate cases do arise where you need to ascertain what happened after the fact. Did that mail get sent? Did that volume bomb on space and then recover without us really knowing? Was someone trying to brute-force a login through ssh at the same time our web server was under attack? In those cases, knowing which logs to look in and how to read them are your only resource for piecing together information after an event took place.

Media of the month: you might suspect it would be some Nine Inch Nails title, however, I'd have to disappoint: that was only a passing fancy. The real Media of the Month is, "Innumeracy: Mathematical Illiteracy and Its Consequences by John Allen Paulos. A book that I forgot I had until I lent it out. Easy, enjoyable, powerful reading.

WWDC time again! Apple has announced the dates (June 11th through the 15th, if you'd missed it), so, get ready. If you're attending, I hope to see you there in person! Until then, though, I'll see you in print next month...


Ed Marczak owns and operates Radiotope, a technology consultancy that brings enterprise solutions to small and medium-sized businesses. Outside of this piece of the puzzle, he is Executive Editor of MacTech Magazine, a husband and father, and CTO of WheresSpot, among other things. Find the missing tech piece at .

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

How to become the ultimate robot warrior...
Chrono Strike is a delightfully immersive beat ‘em up with a sense of humor (any game with a good Sims reference gets points in my book). [Read more] | Read more »
Tips and tricks to get a higher score in...
Snow Roll is a devilish endless runner very much in the vein of Flappy Bird. It revels in its dastardly level of difficulty, and doesn’t really care how angry you get at it as it knows you’ll keep coming back for more. [Read more] | Read more »
How to win big in Slots Deluxe
Cheating while gambling is illegal and morally wrong, and in some parts of the world it leads to men with names like Vinnie "Six Knuckles" Manchenzo beating you to a pulp in a dark alley. [Read more] | Read more »
How to take over the world in Dictator 2
Running a country isn't easy - especially when you're a dictator who wants to take over the world and crush everyone in your path while you do it. [Read more] | Read more »
Tips and tricks to get a higher score in...
Tank.iois - you guessed it! - another multiplayer arena battler likeAgar.io and Slither.io. It does differentiate itself by putting you in a tiny tank though, so it's not exactly the same. To help you get that all-important high score, we've got a... | Read more »
How to unlock characters in One Tap Tenn...
As the title suggests, One Tap Tennis requires only a single tap to play its particular brand of tennis, and rewards you with a ton of unlockable characters if you perform well. Fortunately for you, we at 148Apps have got a few tips and tricks to... | Read more »
Grab it now: Game Craft’s Legend of War...
The real time strategy game is now available for you to sink your teeth into, through the App Store and Google Play. Combining elements of skill, strategy and empire building, Legend of War is a real gamers’ game. [Read more] | Read more »
Skateboard Party 3 ft. Greg Lutzka (Gam...
Skateboard Party 3 ft. Greg Lutzka 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Skateboard Party is back! This third edition of the popular sports franchise features professional skater... | Read more »
Cubious (Games)
Cubious 1.0 Device: iOS Universal Category: Games Price: $.99, Version: 1.0 (iTunes) Description: Cubious – How smart are you? How high is your IQube? Solve the impossible puzzles to find out, and help a lost little cube find his... | Read more »
Goat Simulator Waste of Space (Games)
Goat Simulator Waste of Space 1.1 Device: iOS Universal Category: Games Price: $4.99, Version: 1.1 (iTunes) Description: ** IMPORTANT - SUPPORTED DEVICESiPhone 4S, iPad 2, iPod Touch 5 or better.** | Read more »

Price Scanner via MacPrices.net

12-inch 1.2GHz Space Gray Retina MacBook avai...
Amazon has the new 12″ 1.2GHz/512GB Space Gray Retina MacBook (Apple model #MLH82LL/A) available for $1549 including free shipping. Their price is $50 off MSRP. B&H has the 12″ 1.2GHz/512GB Space... Read more
Apple refurbished Mac Pros available for up t...
Apple has Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The following... Read more
Enterprise Workers Pick Technology Over Perks...
New Adobe study shows surprising attitudes about office jobs and where the future of work is heading. Adobe has released survey findings revealing that a surprising 70 percent of U.S. office workers... Read more
Memorial Day Weekend Sale: $50-$100 off 11-in...
B&H Photo has 13″ and 11″ MacBook Airs with 256GB SSDs on sale for $50-$100 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 11″ 1.6GHz/256GB MacBook Air: $999 $100 off MSRP... Read more
Memorial Day Weekend Sales: Apple MacBook Pro...
B&H Photo has 13″ and 15″ Retina MacBook Pros on sale for up to $210 off MSRP. Shipping is free, and B&H charges NY tax only: - 15″ 2.2GHz Retina MacBook Pro: $1799 $200 off MSRP - 15″ 2.5GHz... Read more
Memorial Day Weekend Sales: Apple iMacs and M...
Take up to $150 off the price of a new iMac or Mac mini at the following Apple resellers this Memorial Day weekend: iMacs: B&H Photo has 21″ and 27″ iMacs on sale for up to $150 off MSRP... Read more
Apple refurbished Retina MacBook Pros availab...
Apple has Certified Refurbished 2015 13″ and 15″ Retina MacBook Pros available for up to $380 off the cost of new models. An Apple one-year warranty is included with each model, and shipping is free... Read more
Apple refurbished 11-inch MacBook Airs availa...
Apple has Certified Refurbished 11″ MacBook Airs (the latest models), available for up to $170 off the cost of new models. An Apple one-year warranty is included with each MacBook, and shipping is... Read more
Goal Zero and OtterBox Partner to Expand iPh...
Goal Zero, specialists in portable power, have announced a partnership with OtterBox, brand smartphone case protection, to offer the Slide and Slide Plus Batteries as modules compatible with the new... Read more
15-inch Retina MacBook Pros on sale for up to...
B&H Photo has 15″ Retina MacBook Pros on sale for up to $210 off MSRP. Shipping is free, and B&H charges NY tax only: - 15″ 2.2GHz Retina MacBook Pro: $1799 $200 off MSRP - 15″ 2.5GHz Retina... Read more

Jobs Board

Senior *Apple* Engineer - Signature Technol...
One-year contract for an Apple consultant. The resource should be an Apple Certified Technical Coordinator or get the certification within 60 days of starting Read more
Senior *Apple* Engineer - Signature Technol...
One-year contract for an Apple consultant. The resource should be an Apple Certified Technical Coordinator or get the certification within 60 days of starting Read more
*Apple* Architect - AECOM (United States)
**Requisition/Vacancy No.** 132759BR **Position Title** Apple Architect **Job Category** Information Technology **Business Line** Government **Country** United Read more
*Apple* Solutions Consultant - APPLE (United...
Job Summary As an Apple Solutions Consultant, you'll be the link between our future customers and our products. You'll showcase your entrepreneurial spirit as you Read more
*Apple* Project Engineer - Smart Source Inc...
SmartSource is in need of an Apple Project Engineer for a 12 month contract opportunity in Pittsburg, PA. Role: Apple Project Engineer Location: Pittsburg, PA Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.