TweetFollow Us on Twitter

Mac In The Shell -- Which Log?

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Mac in the Shell

Mac In The Shell -- Which Log?

Following up on which log shows you what

by Edward Marczak

Introduction

I've had incredible response to my last two columns that talked about logs: what they are, how to interpret them and how to notify yourself if a log is telling you something important. More than any other question, however, people have asked, "which log does what? While I gave an overview of some logs, there are plenty more that I haven't gone into, and more locations for logs than I could describe previously. That's what I'll be following up on this month. So, read on for even more on logs.

Kinda I Want To

Just as a review for anyone who didn't read the previous columns, logs are text files that running programs write to that keep track of their activity. Text files, that's all. (OK, an app could keep a binary log, but for the most part, text it is). This allows other apps and, more importantly, humans to read their contents. Apps are free to deal with logging on their own, or, they can use syslog to hand off their data to the system logger. It's a bit of a style issue, and usage is roughly split on which method is chosen. There's nothing that says that an application can't do both. Let's look quickly again at the syslog method.

The system logger, aka "syslog, is a daemon run at boot that should be running all of the time. It listens for logging data locally, and knows where to put a logging entry based on its configuration file /etc/syslog.conf. "syslogd can also be setup to listen for logging data from other machines as well. Log data can be categorized by facility and level (a "selector). The facility is basically where the data is coming from ("mail, "ftp, etc., along with some generic facilities). The level describes the severity of the log message. The currently defined levels are:

Emergency (level 0) Alert (level 1) Critical (level 2) Error (level 3) Warning (level 4) Notice (level 5) Info (level 6) Debug (level 7)

(Don't you love it when things neatly fit into a byte?) Looking at /etc/syslog.conf, you can then see which messages will be directed to which files. Note that some will go to multiple files.

This is a bit of a review, as all of this (and more) was covered in previous articles in this column.

Down In It

Apple has created a little bit of a split with logging: "Unix-y files in /var/log, and everything else. While that may be a bit of an overgeneralization, it's a good general guideline.

syslogd, as you can see from /etc/syslog.conf, will dump just about everything in some log residing in /var/log. Additionally, several non-syslog sending services also drop their files somewhere into the /var/log hierarchy. The two big notables in this category would be apache logging into /var/httpd and samba, which logs into /var/samba. You'll also find some other non-syslog-ish files hanging around in /var/log, and we'll address those a little later.

The other place you'll find good logging information falls into the "everything else category. You'll find these in /Library/Logs or ~/Library/Logs. System processes will log to /Library/Logs and user processes, when the user home directory is writable, will log to ~/Library/Logs. For example, anything that needs admin level rights to run AFP logs, Software Update and, my favorite, Directory Services will log to the System directory /Library/Logs. Other user-space apps Logic, SyncServices, Parallels, etc. will log into the user's own Library/Logs.

If you've peeked into either of these directories, you'll notice that there's another category of logs: logs that get generated as the result of a crash.

Something I Can Never Have

For those of you obsessed with the running process list, you'll no doubt have noticed the persistent /usr/libexec/crashreporterd. Apple's crash reporter daemon hangs around waiting for an application to crash. Technically, it listens for a Mach exception to be generated, and, upon that happening, launches crashdump. crashdump then logs and reports the event to the user. If the user can be determined, and they have a writable home, the report goes into ~/Library/Logs/CrashReporter. You really forget how much stuff takes a dive until you peek in there:

Jack-Kerouac:~/Library/Logs/CrashReporter marczak$ ls -1
BOMArchiveHelper.crash.log
Cyberduck.crash.log
Dreamweaver.crash.log
Finder.crash.log
GrowlTunes.crash.log
Mail.crash.log
Now Up-To-Date.crash.log
Pages.crash.log
Parallels.crash.log
Preview.crash.log
Quicksilver.crash.log
Safari.crash.log
VirtueDesktops.crash.log
Workgroup Manager.crash.log
firefox-bin.crash.log
iSnip.crash.log
mdimportserver.crash.log
vmware.crash.log

Wow. When an app's owner can't be determined, is a system process, or the user's home directory is not writeable, crashdump logs into /Library/Logs:

Jack-Kerouac:/Library/Logs/CrashReporter root# ls -1 
ARDAgent.crash.log
Exited process.crash.log
httpd.crash.log
iSnip.crash.log
loginwindow.crash.log

LoginWindow crash...I like that one! crashreporter itself, however, logs its actions into /var/log/crashreporter.log. Finally, crashreporterd is also responsible for writing panic logs when the system is rebooted after a panic find the log at /Library/Logs/panic.log.

Terrible Lie

If you root around /var/log long enough, you'll even start to notice some other files that aren't syslog generated at all. The most notable of the bunch are daily.out, weekly.out and monthly.out. These log files are generated by the daily, weekly and monthly periodic jobs that run overnight (if your machine is up and running, but not necessarily logged in).

Also of note are wtmp and lastlog. Interestingly, these are binary log files, and must be read with another application. These are not present for troubleshooting per se, but instead track login activity. User IDs, along with their login time get written to lastlog. From there, the utmp file gets updated with the user login information, and the same utmp record gets written to /var/log/wtmp. users, w and who use utmp to provide their information, while last and ac use wtmp.

That's What I Get

So, back to the original question: which logs record which bit of information? Here's a 40,000 ft. view:

/var/log/system.log

Along with asl.log, these are the big kahunas. Most system related activity is logged here. Specifically, the following selectors log to system.log:

*.notice
authpriv,remoteauth,ftp,install.none
kern.debug
mail.crit

As mentioned in January, asl.log pretty much gets everything, and displays the facility and level with the log message.

/var/log/secure.log

Messages from Apple's SecurityServer get logged here along with anything in the authpriv facility. Good one to keep an eye on. Interestingly, though, some security-related information logs to system log as well (such as sshd authentication errors through PAM).

/var/log/mail.log and /var/log/mailaccess.log

Complimentary logs that describe mail (if you're running OS X Server's built-in mail OS X client' also uses /var/log/mail.log). mail.log contains activity for postfix, SMTP sending and receiving while mailaccess.log contains cyrus' imap and pop activity. mail.log is the place to look when clients are telling you that they sent mail that the intended recipients never received (mail going out), or that someone sends them e-mail which they themselves never received (mail coming in).

/Library/Logs/DirectoryService/*

Huge. HUGE I say! Anyone operating in an OD, AD or other LDAP-reliant environment should be checking here occasionally. Of course, if you're trying to bind a workstation to a directory, and it's just not working for some reason, this is the place to check. Don't forget to throw DirectoryService into debug mode (killall -USR1 DirectoryService, logging to DirectoryService.debug.log), or use boot-time debugging (touch /Library/Preferences/DirectoryService/.DSLogDebugAtStart which will also log to DirectoryService.debug.log) if the situation calls for it covered in detail in past Mac In The Shell articles.

Logs that you don't need explained

There are some logs that simply don't need any explanation. These include:

/var/log/ftp.log (contents of the ftp server)
/var/log/httpd/* (Apache's log files)
/var/log/samba/* (Samba's log files)
/var/log/lpr.log (lpr printing activity)
/var/log/cups/* (CUPS web sever activity)
/Library/Logs/Software Update.log (Software Update install history)

/Library/Logs/AppleFileService OK, perhaps this one does need some explanation. But not by me from the engineers at Apple. This should be the Mecca of information that you reach for when you want to track activity coming in through AFP. However, details in this log are so sparse to render it useless. Despite the fact that AppleFileServer runs as one grand, monolithic process, we need it to give up the goods on what's happening internally in these logs.

The mega-log

Remember, that you can always have all syslog messages be delivered to a single file by using a wildcard selector in /etc/syslog.conf:

*.*      /var/log/mega.log

…although asl.log is already catching most of that. Further remember that this will not capture everything else that is performing logging on its own and not using syslog/asl.

The Only Time

While I made a strong case for watching and examining logs using good ol Unix tools grep, tail and less and I would still make that assertion, there are some other very useful tools out there.


Figure 1: Splunk searching for events (detail)

Console.app is the most obvious for us Mac users; it's "built-in and clearly Mac-like. I still think it's a miserable way to follow log activity as it happens, but it is a fantastic tool for exploring the various logs that exist on the system. Fire up Console.app (located in the Utilities folder) and poke around. It's a great learning experience.

I've found a newer tool on the market to also be very useful, and while it's not Mac-specific, it's nice that there's an OS X version at all. Splunk is billed as a logfile search engine. It comes in a free version that will sift through up to 500MB of logs per day, and a paid version which not only removes that restriction altogether, but also adds Splunk-to-Splunk logging for log correlation across many machines, and also authentication which is missing in the free version.

Splunk alone could take up an entire column (and may very well one day soon); it is so easy to get going, that I'd recommend the download. Excellent documentation exists on the site as well. Of course, Splunk makes it really easy to search for events, but I've found that it's a nice exploratory tool in general. Splunk quickly categorizes events and can let you filter on those events. Figure 1 shows the search engine, running in a browser, looking at event type 9 on my machine.

Even cooler, perhaps, is the ability to look at the frequency of events. Figure 2 shows the frequency of events that appear in the secure.log of my machine clearly a good one to keep an eye on and reign in this kind of data.


Figure 2 Splunk showing the frequency of various events that appear in a given source.

In a small-to-medium sized organization, you could set up a single server as a log host (check the prior logging articles for instructions) and have Splunk access that log. 500MB is a lot of logging information. Enough that you shouldn't have a problem using the free version of Splunk.

Sanctified

So here's the real story with logs: you can ignore them, sure. Then when a problem crops up, you can just tell your client/employer that this is just the way computers are. There are bugs, there are problems; such is life. If you watch the logs, you'll know there's a problem, and then you'll actually have to do something about it.

Or, you can realize that this is the only way the system can speak to you and more often that not, you're warned well in advance of any catastrophic problems. You can head these off at the pass. You can keep the system providing services without interruption and keep up with real work.

Of course, the unfortunate cases do arise where you need to ascertain what happened after the fact. Did that mail get sent? Did that volume bomb on space and then recover without us really knowing? Was someone trying to brute-force a login through ssh at the same time our web server was under attack? In those cases, knowing which logs to look in and how to read them are your only resource for piecing together information after an event took place.

Media of the month: you might suspect it would be some Nine Inch Nails title, however, I'd have to disappoint: that was only a passing fancy. The real Media of the Month is, "Innumeracy: Mathematical Illiteracy and Its Consequences by John Allen Paulos. A book that I forgot I had until I lent it out. Easy, enjoyable, powerful reading.

WWDC time again! Apple has announced the dates (June 11th through the 15th, if you'd missed it), so, get ready. If you're attending, I hope to see you there in person! Until then, though, I'll see you in print next month...


Ed Marczak owns and operates Radiotope, a technology consultancy that brings enterprise solutions to small and medium-sized businesses. Outside of this piece of the puzzle, he is Executive Editor of MacTech Magazine, a husband and father, and CTO of WheresSpot, among other things. Find the missing tech piece at .

 
AAPL
$99.02
Apple Inc.
+1.35
MSFT
$43.97
Microsoft Corpora
-0.53
GOOG
$590.60
Google Inc.
+1.58

MacTech Search:
Community Search:

Software Updates via MacUpdate

OS X Yosemite Wallpaper 1.0 - Desktop im...
OS X Yosemite Wallpaper is the gorgeous new background image for Apple's upcoming OS X 10.10 Yosemite. This wallpaper is available for all screen resolutions with a source file that measures 5,418... Read more
Acorn 4.4 - Bitmap image editor. (Demo)
Acorn is a new image editor built with one goal in mind - simplicity. Fast, easy, and fluid, Acorn provides the options you'll need without any overhead. Acorn feels right, and won't drain your bank... Read more
Bartender 1.2.20 - Organize your menu ba...
Bartender lets you organize your menu bar apps. Features: Lets you tidy your menu bar apps how you want. See your menu bar apps when you want. Hide the apps you need to run, but do not need to... Read more
TotalFinder 1.6.2 - Adds tabs, hotkeys,...
TotalFinder is a universally acclaimed navigational companion for your Mac. Enhance your Mac's Finder with features so smart and convenient, you won't believe you ever lived without them. Tab-based... Read more
Vienna 3.0.0 RC 2 :be5265e: - RSS and At...
Vienna is a freeware and Open-Source RSS/Atom newsreader with article storage and management via a SQLite database, written in Objective-C and Cocoa, for the OS X operating system. It provides... Read more
VLC Media Player 2.1.5 - Popular multime...
VLC Media Player is a highly portable multimedia player for various audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, MP3, OGG, ...) as well as DVDs, VCDs, and various streaming protocols. It... Read more
Default Folder X 4.6.7 - Enhances Open a...
Default Folder X attaches a toolbar to the right side of the Open and Save dialogs in any OS X-native application. The toolbar gives you fast access to various folders and commands. You just click... Read more
TinkerTool 5.3 - Expanded preference set...
TinkerTool is an application that gives you access to additional preference settings Apple has built into Mac OS X. This allows to activate hidden features in the operating system and in some of the... Read more
Audio Hijack Pro 2.11.0 - Record and enh...
Audio Hijack Pro drastically changes the way you use audio on your computer, giving you the freedom to listen to audio when you want and how you want. Record and enhance any audio with Audio Hijack... Read more
Intermission 1.1.1 - Pause and rewind li...
Intermission allows you to pause and rewind live audio from any application on your Mac. Intermission will buffer up to 3 hours of audio, allowing users to skip through any assortment of audio... Read more

Latest Forum Discussions

See All

Traps n’ Gemstones Review
Traps n’ Gemstones Review By Campbell Bird on July 28th, 2014 Our Rating: :: CASTLEVANIA JONESUniversal App - Designed for iPhone and iPad Fight mummies, dig tunnels, and ride a runaway minecart to discover ancient secrets in this... | Read more »
The Phantom PI Mission Apparition Review
The Phantom PI Mission Apparition Review By Jordan Minor on July 28th, 2014 Our Rating: :: GHOSTS BUSTEDUniversal App - Designed for iPhone and iPad The Phantom PI is an exceedingly clever and well-crafted adventure game.   | Read more »
More Stubies Are Coming Your Way in a Ne...
More Stubies Are Coming Your Way in a New Update Posted by Jessica Fisher on July 28th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
The Great Prank War Review
The Great Prank War Review By Nadia Oxford on July 28th, 2014 Our Rating: :: PRANKING IS SERIOUS BUSINESSUniversal App - Designed for iPhone and iPad Though short, The Great Prank War offers an interesting and fun mix of action and... | Read more »
Marvel Contest of Champions Announced at...
Marvel Contest of Champions Announced at Comic-Con Posted by Jennifer Allen on July 28th, 2014 [ permalink ] Announced over the weekend at San Diego Comic-Con was the fairly exciting looking Marvel Contest of Champions. | Read more »
Teenage Mutant Ninja Turtles Review
Teenage Mutant Ninja Turtles Review By Jennifer Allen on July 28th, 2014 Our Rating: :: DULL SWIPINGUniversal App - Designed for iPhone and iPad The pizza power is weak when it comes to this Teenage Mutant Ninja Turtles game.   | Read more »
Exploration Focused Puzzle Game Beatbudd...
Exploration Focused Puzzle Game Beatbuddy Set to Make Transition from PC to iOS this September Posted by Jennifer Allen on July 28th, 2014 [ permalink ] | Read more »
PlanetHD
PlanetHD By Nadia Oxford on July 28th, 2014 Our Rating: :: SPACE MADNESSUniversal App - Designed for iPhone and iPad PlanetHD will keep players busy for a while, though its unpredictable physics are a handful to deal with.   | Read more »
This Week at 148Apps: July 21-25, 2014
Another Week of Expert App Reviews   At 148Apps, we help you sort through the great ocean of apps to find the ones we think you’ll like and the ones you’ll need. Our top picks become Editor’s Choice, our stamp of approval for apps with that little... | Read more »
Reddme for iPhone - The Reddit Client (...
Reddme for iPhone - The Reddit Client 1.0 Device: iOS iPhone Category: News Price: $.99, Version: 1.0 (iTunes) Description: Reddme for iPhone is an iOS 7-optimized Reddit client that offers a refreshing new way to experience Reddit... | Read more »

Price Scanner via MacPrices.net

iOS 8 and OS X 10.10 To Support DuckDuckGo As...
Writing for Quartz, Dan Frommer reports that Apple’s forthcoming iOS 8 and OS X 10.10 operating systems version updates will allow users to select DuckDuckGo as their default search engine. He notes... Read more
U.K. Hospital Using iPods and iPads To Record...
British news journal GazetteLive’s. Ian McNeal notes that the old “an apple a day keeps the doctor away” proverb is being turned on its head at http://southtees.nhs.uk/hospitals/james-cook/ James... Read more
13-inch 2.5GHz MacBook Pro on sale for $1099,...
Best Buy has the 13″ 2.5GHz MacBook Pro available for $1099.99 on their online store. Choose free shipping or free instant local store pickup (if available). Their price is $100 off MSRP. Price is... Read more
Roundup of Apple refurbished MacBook Pros, th...
The Apple Store has Apple Certified Refurbished 13″ and 15″ MacBook Pros available for up to $400 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free. Their prices... Read more
Record Mac Shipments In Q2/14 Confound Analys...
A Seeking Alpha Trefis commentary notes that Apple’s fiscal Q3 2014 results released July 22, beat market predictions on earnings, although revenues were slightly lower than anticipated. Apple’s Mac’... Read more
Intel To Launch Core M Silicon For Use In Not...
Digitimes’ Monica Chen and Joseph Tsai, report that Intel will launch 14nm-based Core M series processors specifically for use in fanless notebook/tablet 2-in-1 models in Q4 2014, with many models to... Read more
Apple’s 2014 Back to School promotion: $100 g...
 Apple’s 2014 Back to School promotion includes a free $100 App Store Gift Card with the purchase of any new Mac (Mac mini excluded), or a $50 Gift Card with the purchase of an iPad or iPhone,... Read more
iMacs on sale for $150 off MSRP, $250 off for...
Best Buy has iMacs on sale for up to $160 off MSRP for a limited time. Choose free home shipping or free instant local store pickup (if available). Prices are valid for online orders only, in-store... Read more
Mac minis on sale for $100 off MSRP, starting...
Best Buy has Mac minis on sale for $100 off MSRP. Choose free shipping or free instant local store pickup. Prices are for online orders only, in-store prices may vary: 2.5GHz Mac mini: $499.99 2.3GHz... Read more
Global Tablet Market Grows 11% in Q2/14 Notwi...
Worldwide tablet sales grew 11.0 percent year over year in the second quarter of 2014, with shipments reaching 49.3 million units according to preliminary data from the International Data Corporation... Read more

Jobs Board

Sr Software Lead Engineer, *Apple* Online S...
Sr Software Lead Engineer, Apple Online Store Publishing Systems Keywords: Company: Apple Job Code: E3PCAK8MgYYkw Location (City or ZIP): Santa Clara Status: Full Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
Sr. Product Leader, *Apple* Store Apps - Ap...
**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.