TweetFollow Us on Twitter

Mac In The Shell -- Which Log?

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Mac in the Shell

Mac In The Shell -- Which Log?

Following up on which log shows you what

by Edward Marczak

Introduction

I've had incredible response to my last two columns that talked about logs: what they are, how to interpret them and how to notify yourself if a log is telling you something important. More than any other question, however, people have asked, "which log does what? While I gave an overview of some logs, there are plenty more that I haven't gone into, and more locations for logs than I could describe previously. That's what I'll be following up on this month. So, read on for even more on logs.

Kinda I Want To

Just as a review for anyone who didn't read the previous columns, logs are text files that running programs write to that keep track of their activity. Text files, that's all. (OK, an app could keep a binary log, but for the most part, text it is). This allows other apps and, more importantly, humans to read their contents. Apps are free to deal with logging on their own, or, they can use syslog to hand off their data to the system logger. It's a bit of a style issue, and usage is roughly split on which method is chosen. There's nothing that says that an application can't do both. Let's look quickly again at the syslog method.

The system logger, aka "syslog, is a daemon run at boot that should be running all of the time. It listens for logging data locally, and knows where to put a logging entry based on its configuration file /etc/syslog.conf. "syslogd can also be setup to listen for logging data from other machines as well. Log data can be categorized by facility and level (a "selector). The facility is basically where the data is coming from ("mail, "ftp, etc., along with some generic facilities). The level describes the severity of the log message. The currently defined levels are:

Emergency (level 0) Alert (level 1) Critical (level 2) Error (level 3) Warning (level 4) Notice (level 5) Info (level 6) Debug (level 7)

(Don't you love it when things neatly fit into a byte?) Looking at /etc/syslog.conf, you can then see which messages will be directed to which files. Note that some will go to multiple files.

This is a bit of a review, as all of this (and more) was covered in previous articles in this column.

Down In It

Apple has created a little bit of a split with logging: "Unix-y files in /var/log, and everything else. While that may be a bit of an overgeneralization, it's a good general guideline.

syslogd, as you can see from /etc/syslog.conf, will dump just about everything in some log residing in /var/log. Additionally, several non-syslog sending services also drop their files somewhere into the /var/log hierarchy. The two big notables in this category would be apache logging into /var/httpd and samba, which logs into /var/samba. You'll also find some other non-syslog-ish files hanging around in /var/log, and we'll address those a little later.

The other place you'll find good logging information falls into the "everything else category. You'll find these in /Library/Logs or ~/Library/Logs. System processes will log to /Library/Logs and user processes, when the user home directory is writable, will log to ~/Library/Logs. For example, anything that needs admin level rights to run AFP logs, Software Update and, my favorite, Directory Services will log to the System directory /Library/Logs. Other user-space apps Logic, SyncServices, Parallels, etc. will log into the user's own Library/Logs.

If you've peeked into either of these directories, you'll notice that there's another category of logs: logs that get generated as the result of a crash.

Something I Can Never Have

For those of you obsessed with the running process list, you'll no doubt have noticed the persistent /usr/libexec/crashreporterd. Apple's crash reporter daemon hangs around waiting for an application to crash. Technically, it listens for a Mach exception to be generated, and, upon that happening, launches crashdump. crashdump then logs and reports the event to the user. If the user can be determined, and they have a writable home, the report goes into ~/Library/Logs/CrashReporter. You really forget how much stuff takes a dive until you peek in there:

Jack-Kerouac:~/Library/Logs/CrashReporter marczak$ ls -1
BOMArchiveHelper.crash.log
Cyberduck.crash.log
Dreamweaver.crash.log
Finder.crash.log
GrowlTunes.crash.log
Mail.crash.log
Now Up-To-Date.crash.log
Pages.crash.log
Parallels.crash.log
Preview.crash.log
Quicksilver.crash.log
Safari.crash.log
VirtueDesktops.crash.log
Workgroup Manager.crash.log
firefox-bin.crash.log
iSnip.crash.log
mdimportserver.crash.log
vmware.crash.log

Wow. When an app's owner can't be determined, is a system process, or the user's home directory is not writeable, crashdump logs into /Library/Logs:

Jack-Kerouac:/Library/Logs/CrashReporter root# ls -1 
ARDAgent.crash.log
Exited process.crash.log
httpd.crash.log
iSnip.crash.log
loginwindow.crash.log

LoginWindow crash...I like that one! crashreporter itself, however, logs its actions into /var/log/crashreporter.log. Finally, crashreporterd is also responsible for writing panic logs when the system is rebooted after a panic find the log at /Library/Logs/panic.log.

Terrible Lie

If you root around /var/log long enough, you'll even start to notice some other files that aren't syslog generated at all. The most notable of the bunch are daily.out, weekly.out and monthly.out. These log files are generated by the daily, weekly and monthly periodic jobs that run overnight (if your machine is up and running, but not necessarily logged in).

Also of note are wtmp and lastlog. Interestingly, these are binary log files, and must be read with another application. These are not present for troubleshooting per se, but instead track login activity. User IDs, along with their login time get written to lastlog. From there, the utmp file gets updated with the user login information, and the same utmp record gets written to /var/log/wtmp. users, w and who use utmp to provide their information, while last and ac use wtmp.

That's What I Get

So, back to the original question: which logs record which bit of information? Here's a 40,000 ft. view:

/var/log/system.log

Along with asl.log, these are the big kahunas. Most system related activity is logged here. Specifically, the following selectors log to system.log:

*.notice
authpriv,remoteauth,ftp,install.none
kern.debug
mail.crit

As mentioned in January, asl.log pretty much gets everything, and displays the facility and level with the log message.

/var/log/secure.log

Messages from Apple's SecurityServer get logged here along with anything in the authpriv facility. Good one to keep an eye on. Interestingly, though, some security-related information logs to system log as well (such as sshd authentication errors through PAM).

/var/log/mail.log and /var/log/mailaccess.log

Complimentary logs that describe mail (if you're running OS X Server's built-in mail OS X client' also uses /var/log/mail.log). mail.log contains activity for postfix, SMTP sending and receiving while mailaccess.log contains cyrus' imap and pop activity. mail.log is the place to look when clients are telling you that they sent mail that the intended recipients never received (mail going out), or that someone sends them e-mail which they themselves never received (mail coming in).

/Library/Logs/DirectoryService/*

Huge. HUGE I say! Anyone operating in an OD, AD or other LDAP-reliant environment should be checking here occasionally. Of course, if you're trying to bind a workstation to a directory, and it's just not working for some reason, this is the place to check. Don't forget to throw DirectoryService into debug mode (killall -USR1 DirectoryService, logging to DirectoryService.debug.log), or use boot-time debugging (touch /Library/Preferences/DirectoryService/.DSLogDebugAtStart which will also log to DirectoryService.debug.log) if the situation calls for it covered in detail in past Mac In The Shell articles.

Logs that you don't need explained

There are some logs that simply don't need any explanation. These include:

/var/log/ftp.log (contents of the ftp server)
/var/log/httpd/* (Apache's log files)
/var/log/samba/* (Samba's log files)
/var/log/lpr.log (lpr printing activity)
/var/log/cups/* (CUPS web sever activity)
/Library/Logs/Software Update.log (Software Update install history)

/Library/Logs/AppleFileService OK, perhaps this one does need some explanation. But not by me from the engineers at Apple. This should be the Mecca of information that you reach for when you want to track activity coming in through AFP. However, details in this log are so sparse to render it useless. Despite the fact that AppleFileServer runs as one grand, monolithic process, we need it to give up the goods on what's happening internally in these logs.

The mega-log

Remember, that you can always have all syslog messages be delivered to a single file by using a wildcard selector in /etc/syslog.conf:

*.*      /var/log/mega.log

…although asl.log is already catching most of that. Further remember that this will not capture everything else that is performing logging on its own and not using syslog/asl.

The Only Time

While I made a strong case for watching and examining logs using good ol Unix tools grep, tail and less and I would still make that assertion, there are some other very useful tools out there.


Figure 1: Splunk searching for events (detail)

Console.app is the most obvious for us Mac users; it's "built-in and clearly Mac-like. I still think it's a miserable way to follow log activity as it happens, but it is a fantastic tool for exploring the various logs that exist on the system. Fire up Console.app (located in the Utilities folder) and poke around. It's a great learning experience.

I've found a newer tool on the market to also be very useful, and while it's not Mac-specific, it's nice that there's an OS X version at all. Splunk is billed as a logfile search engine. It comes in a free version that will sift through up to 500MB of logs per day, and a paid version which not only removes that restriction altogether, but also adds Splunk-to-Splunk logging for log correlation across many machines, and also authentication which is missing in the free version.

Splunk alone could take up an entire column (and may very well one day soon); it is so easy to get going, that I'd recommend the download. Excellent documentation exists on the site as well. Of course, Splunk makes it really easy to search for events, but I've found that it's a nice exploratory tool in general. Splunk quickly categorizes events and can let you filter on those events. Figure 1 shows the search engine, running in a browser, looking at event type 9 on my machine.

Even cooler, perhaps, is the ability to look at the frequency of events. Figure 2 shows the frequency of events that appear in the secure.log of my machine clearly a good one to keep an eye on and reign in this kind of data.


Figure 2 Splunk showing the frequency of various events that appear in a given source.

In a small-to-medium sized organization, you could set up a single server as a log host (check the prior logging articles for instructions) and have Splunk access that log. 500MB is a lot of logging information. Enough that you shouldn't have a problem using the free version of Splunk.

Sanctified

So here's the real story with logs: you can ignore them, sure. Then when a problem crops up, you can just tell your client/employer that this is just the way computers are. There are bugs, there are problems; such is life. If you watch the logs, you'll know there's a problem, and then you'll actually have to do something about it.

Or, you can realize that this is the only way the system can speak to you and more often that not, you're warned well in advance of any catastrophic problems. You can head these off at the pass. You can keep the system providing services without interruption and keep up with real work.

Of course, the unfortunate cases do arise where you need to ascertain what happened after the fact. Did that mail get sent? Did that volume bomb on space and then recover without us really knowing? Was someone trying to brute-force a login through ssh at the same time our web server was under attack? In those cases, knowing which logs to look in and how to read them are your only resource for piecing together information after an event took place.

Media of the month: you might suspect it would be some Nine Inch Nails title, however, I'd have to disappoint: that was only a passing fancy. The real Media of the Month is, "Innumeracy: Mathematical Illiteracy and Its Consequences by John Allen Paulos. A book that I forgot I had until I lent it out. Easy, enjoyable, powerful reading.

WWDC time again! Apple has announced the dates (June 11th through the 15th, if you'd missed it), so, get ready. If you're attending, I hope to see you there in person! Until then, though, I'll see you in print next month...


Ed Marczak owns and operates Radiotope, a technology consultancy that brings enterprise solutions to small and medium-sized businesses. Outside of this piece of the puzzle, he is Executive Editor of MacTech Magazine, a husband and father, and CTO of WheresSpot, among other things. Find the missing tech piece at .

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

iFFmpeg 6.2.5 - Convert multimedia files...
iFFmpeg is a comprehensive media tool to convert movie, audio and media files between formats. The FFmpeg command line instructions can be very hard to master/understand, so iFFmpeg does all the hard... Read more
DaisyDisk 4.4 - $9.99
DaisyDisk allows you to visualize your disk usage and free up disk space by quickly finding and deleting big unused files. The program scans your disk and displays its content as a sector diagram... Read more
iFFmpeg 6.2.5 - Convert multimedia files...
iFFmpeg is a comprehensive media tool to convert movie, audio and media files between formats. The FFmpeg command line instructions can be very hard to master/understand, so iFFmpeg does all the hard... Read more
DaisyDisk 4.4 - $9.99
DaisyDisk allows you to visualize your disk usage and free up disk space by quickly finding and deleting big unused files. The program scans your disk and displays its content as a sector diagram... Read more
BetterTouchTool 2.07 - Customize Multi-T...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
BetterTouchTool 2.071 - Customize Multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
BetterTouchTool 2.07 - Customize Multi-T...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
PDFpen 8.3.2 - $74.95
PDFpen allows users to easily edit PDF's. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Features... Read more
DiskCatalogMaker 6.5.20 - Catalog your d...
DiskCatalogMaker is a simple disk management tool which catalogs disks. Simple, light-weight, and fast Finder-like intuitive look and feel Super-fast search algorithm Can compress catalog data for... Read more
Things 2.8.9 - Elegant personal task man...
Things is a task management solution that helps to organize your tasks in an elegant and intuitive way. Things combines powerful features with simplicity through the use of tags and its intelligent... Read more

Latest Forum Discussions

See All

Pokémon GO Generation 2 evolution guide
At long last, Niantic Labs finally unleashed the Generation 2 Pokémon into the wild. Pokémon GO trainers are scrambling to grab up this new set of 80 Pokémon. There are some special new tricks required to catch all of these new beasties, though.... | Read more »
The best new games we played this week
It feels as though the New Year got off to a creaking start as far as mobile games go, but that's changed over the past few weeks. The last few days alone have seen the debut of a number of wonderful games, so we thought we'd take the time to... | Read more »
Recruit more scallywags and discover new...
Get ready to show off your sea legs all over again in Oceans & Empires’ new grand update, which aims to make the act of rising to the role of seven seas ruler even more fresh and appealing, thanks to a richness of new content on both iOS and... | Read more »
Mage the Ascension: Refuge (Games)
Mage the Ascension: Refuge 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: The groundbreaking roleplaying game Mage: The Ascension manifests in our turbulent present with Refuge, an... | Read more »
Vampire: Prelude (Games)
Vampire: Prelude 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: The classic roleplaying game Vampire: The Masquerade returns to digital games with a Prelude of things to come. Experience a... | Read more »
Digby Forever Guide: How to dig to the d...
Digby Forever is a sparkling homage to arcade classics, and while you may be tiring of the number of arcade games being thrown at you, this endless digger finds many ways to stand out from the rest of the pack. The game manages to be challenging... | Read more »
The best sales on the App Store this wee...
It's been quite the week in mobile games, but if the latest releases(there were some pretty darn good ones, in case you missed out) aren't really doing the trick, perhaps some of these discounted games will. Many of these premium games had their... | Read more »
Why the new Fire Emblem Heroes update sh...
It’s exciting to see Nintendo delving into the mobile sphere, regardless of whether it’s to give fans another platform to enjoy their fans or simply a sound business venture. Two of the company's announced mobile games have finally come to... | Read more »
New Fire Emblem Heroes update adds new h...
Fire Emblem Heroes received a sizeable update first thing this morning. The update features a batch of fresh content along with a few updates to the game's systems. [Read more] | Read more »
The Deep Paths (Games)
The Deep Paths 1.0 Device: iOS iPhone Category: Games Price: $3.99, Version: 1.0 (iTunes) Description: 25% off launch sale!!! The Deep Paths: Labyrinth Of Andokost is a first-person, dungeon crawling RPG, with traditional grid-based... | Read more »

Price Scanner via MacPrices.net

15-inch Touch Bar MacBook Pros on sale for up...
Amazon has 15″ Touch Bar MacBook Pros in stock today and on sale for up to $150 off MSRP including free shipping: - 15″ 2.7GHz Touch Bar MacBook Pro Space Gray: $2720.38 $79 off MSRP - 15″ 2.7GHz... Read more
Apple’s Education discount saves up to $300 o...
Purchase a new Mac or iPad using Apple’s Education Store and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free: -... Read more
13-inch 1.6GHz/256GB MacBook Air on sale for...
Newegg has the 13″ 1.6GHz/256GB MacBook Air (MMGG2LL/A) on sale for $1029.99 including free shipping. Their price is $170 off MSRP, and it’s the lowest price available for this model. Choose Newegg... Read more
Save up to $600 with Apple refurbished Mac Pr...
Apple has Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The following... Read more
12-inch 1.1GHz Retina MacBooks on sale for $1...
B&H has 12″ 1.1GHz Retina MacBooks on sale for $150 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.1GHz Space Gray Retina MacBook: $1149 $150 off MSRP - 12″ 1.1GHz... Read more
InTouch Health Expands iOS And Windows Produc...
Specialty telehealth enterprise provider InTouch Health has announced an expanded range of FDA Class I listed medical devices and software solutions for ambulatory, non-acute and non-emergent... Read more
iMobie Airs World’s 1st iCloud Manager with M...
iMobie Inc., an Apple-related software company, announced their newly-updated iPhone manager AnyTrans with exclusive feature to sync and manage contents across multiple iCloud accounts. With it,... Read more
New Proactive Apple Support Professional Cert...
Watchman Monitoring has announced Proactive Support Professional Certification at MacTech Pro. Watchman Monitoring is a premier Proactive Support Software as a Service (SaaS) tool for IT... Read more
13-inch 2.7GHz Retina MacBook Pro on sale for...
B&H Photo has the 2015 13″ 2.7GHz/128GB Retina Apple MacBook Pro on sale for $100 off MSRP. Shipping is free, and B&H charges NY tax only: - 13″ 2.7GHz/128GB Retina MacBook Pro (MF839LL/A): $... Read more
Back in stock: Apple refurbished 13-inch Reti...
Apple has Certified Refurbished 2015 13″ Retina MacBook Pros available for up to $360 off original MSRP, starting at $1099. An Apple one-year warranty is included with each model, and shipping is... Read more

Jobs Board

*Apple* Technician - nfrastructure (United S...
Let’s Work Together Apple Technician This position is based in Portland, ME Life at nfrastructure At nfrastructure, we understand that our success results from our Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Manager *Apple* Systems Administration - Pu...
Req ID 3315BR Position Title Manager, Apple Systems Administration Job Description The Manager of Apple Systems Administration oversees the administration and Read more
*Apple* Retail - Multiple Positions - Apple,...
SalesSpecialist - Retail Customer Service and SalesTransform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* macOS Systems Integration Administra...
…most exceptional support available in the industry. SCI is seeking an Junior Apple macOS systems integration administrator that will be responsible for providing Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.