TweetFollow Us on Twitter

Mac In The Shell -- Which Log?

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Mac in the Shell

Mac In The Shell -- Which Log?

Following up on which log shows you what

by Edward Marczak

Introduction

I've had incredible response to my last two columns that talked about logs: what they are, how to interpret them and how to notify yourself if a log is telling you something important. More than any other question, however, people have asked, "which log does what? While I gave an overview of some logs, there are plenty more that I haven't gone into, and more locations for logs than I could describe previously. That's what I'll be following up on this month. So, read on for even more on logs.

Kinda I Want To

Just as a review for anyone who didn't read the previous columns, logs are text files that running programs write to that keep track of their activity. Text files, that's all. (OK, an app could keep a binary log, but for the most part, text it is). This allows other apps and, more importantly, humans to read their contents. Apps are free to deal with logging on their own, or, they can use syslog to hand off their data to the system logger. It's a bit of a style issue, and usage is roughly split on which method is chosen. There's nothing that says that an application can't do both. Let's look quickly again at the syslog method.

The system logger, aka "syslog, is a daemon run at boot that should be running all of the time. It listens for logging data locally, and knows where to put a logging entry based on its configuration file /etc/syslog.conf. "syslogd can also be setup to listen for logging data from other machines as well. Log data can be categorized by facility and level (a "selector). The facility is basically where the data is coming from ("mail, "ftp, etc., along with some generic facilities). The level describes the severity of the log message. The currently defined levels are:

Emergency (level 0) Alert (level 1) Critical (level 2) Error (level 3) Warning (level 4) Notice (level 5) Info (level 6) Debug (level 7)

(Don't you love it when things neatly fit into a byte?) Looking at /etc/syslog.conf, you can then see which messages will be directed to which files. Note that some will go to multiple files.

This is a bit of a review, as all of this (and more) was covered in previous articles in this column.

Down In It

Apple has created a little bit of a split with logging: "Unix-y files in /var/log, and everything else. While that may be a bit of an overgeneralization, it's a good general guideline.

syslogd, as you can see from /etc/syslog.conf, will dump just about everything in some log residing in /var/log. Additionally, several non-syslog sending services also drop their files somewhere into the /var/log hierarchy. The two big notables in this category would be apache logging into /var/httpd and samba, which logs into /var/samba. You'll also find some other non-syslog-ish files hanging around in /var/log, and we'll address those a little later.

The other place you'll find good logging information falls into the "everything else category. You'll find these in /Library/Logs or ~/Library/Logs. System processes will log to /Library/Logs and user processes, when the user home directory is writable, will log to ~/Library/Logs. For example, anything that needs admin level rights to run AFP logs, Software Update and, my favorite, Directory Services will log to the System directory /Library/Logs. Other user-space apps Logic, SyncServices, Parallels, etc. will log into the user's own Library/Logs.

If you've peeked into either of these directories, you'll notice that there's another category of logs: logs that get generated as the result of a crash.

Something I Can Never Have

For those of you obsessed with the running process list, you'll no doubt have noticed the persistent /usr/libexec/crashreporterd. Apple's crash reporter daemon hangs around waiting for an application to crash. Technically, it listens for a Mach exception to be generated, and, upon that happening, launches crashdump. crashdump then logs and reports the event to the user. If the user can be determined, and they have a writable home, the report goes into ~/Library/Logs/CrashReporter. You really forget how much stuff takes a dive until you peek in there:

Jack-Kerouac:~/Library/Logs/CrashReporter marczak$ ls -1
BOMArchiveHelper.crash.log
Cyberduck.crash.log
Dreamweaver.crash.log
Finder.crash.log
GrowlTunes.crash.log
Mail.crash.log
Now Up-To-Date.crash.log
Pages.crash.log
Parallels.crash.log
Preview.crash.log
Quicksilver.crash.log
Safari.crash.log
VirtueDesktops.crash.log
Workgroup Manager.crash.log
firefox-bin.crash.log
iSnip.crash.log
mdimportserver.crash.log
vmware.crash.log

Wow. When an app's owner can't be determined, is a system process, or the user's home directory is not writeable, crashdump logs into /Library/Logs:

Jack-Kerouac:/Library/Logs/CrashReporter root# ls -1 
ARDAgent.crash.log
Exited process.crash.log
httpd.crash.log
iSnip.crash.log
loginwindow.crash.log

LoginWindow crash...I like that one! crashreporter itself, however, logs its actions into /var/log/crashreporter.log. Finally, crashreporterd is also responsible for writing panic logs when the system is rebooted after a panic find the log at /Library/Logs/panic.log.

Terrible Lie

If you root around /var/log long enough, you'll even start to notice some other files that aren't syslog generated at all. The most notable of the bunch are daily.out, weekly.out and monthly.out. These log files are generated by the daily, weekly and monthly periodic jobs that run overnight (if your machine is up and running, but not necessarily logged in).

Also of note are wtmp and lastlog. Interestingly, these are binary log files, and must be read with another application. These are not present for troubleshooting per se, but instead track login activity. User IDs, along with their login time get written to lastlog. From there, the utmp file gets updated with the user login information, and the same utmp record gets written to /var/log/wtmp. users, w and who use utmp to provide their information, while last and ac use wtmp.

That's What I Get

So, back to the original question: which logs record which bit of information? Here's a 40,000 ft. view:

/var/log/system.log

Along with asl.log, these are the big kahunas. Most system related activity is logged here. Specifically, the following selectors log to system.log:

*.notice
authpriv,remoteauth,ftp,install.none
kern.debug
mail.crit

As mentioned in January, asl.log pretty much gets everything, and displays the facility and level with the log message.

/var/log/secure.log

Messages from Apple's SecurityServer get logged here along with anything in the authpriv facility. Good one to keep an eye on. Interestingly, though, some security-related information logs to system log as well (such as sshd authentication errors through PAM).

/var/log/mail.log and /var/log/mailaccess.log

Complimentary logs that describe mail (if you're running OS X Server's built-in mail OS X client' also uses /var/log/mail.log). mail.log contains activity for postfix, SMTP sending and receiving while mailaccess.log contains cyrus' imap and pop activity. mail.log is the place to look when clients are telling you that they sent mail that the intended recipients never received (mail going out), or that someone sends them e-mail which they themselves never received (mail coming in).

/Library/Logs/DirectoryService/*

Huge. HUGE I say! Anyone operating in an OD, AD or other LDAP-reliant environment should be checking here occasionally. Of course, if you're trying to bind a workstation to a directory, and it's just not working for some reason, this is the place to check. Don't forget to throw DirectoryService into debug mode (killall -USR1 DirectoryService, logging to DirectoryService.debug.log), or use boot-time debugging (touch /Library/Preferences/DirectoryService/.DSLogDebugAtStart which will also log to DirectoryService.debug.log) if the situation calls for it covered in detail in past Mac In The Shell articles.

Logs that you don't need explained

There are some logs that simply don't need any explanation. These include:

/var/log/ftp.log (contents of the ftp server)
/var/log/httpd/* (Apache's log files)
/var/log/samba/* (Samba's log files)
/var/log/lpr.log (lpr printing activity)
/var/log/cups/* (CUPS web sever activity)
/Library/Logs/Software Update.log (Software Update install history)

/Library/Logs/AppleFileService OK, perhaps this one does need some explanation. But not by me from the engineers at Apple. This should be the Mecca of information that you reach for when you want to track activity coming in through AFP. However, details in this log are so sparse to render it useless. Despite the fact that AppleFileServer runs as one grand, monolithic process, we need it to give up the goods on what's happening internally in these logs.

The mega-log

Remember, that you can always have all syslog messages be delivered to a single file by using a wildcard selector in /etc/syslog.conf:

*.*      /var/log/mega.log

…although asl.log is already catching most of that. Further remember that this will not capture everything else that is performing logging on its own and not using syslog/asl.

The Only Time

While I made a strong case for watching and examining logs using good ol Unix tools grep, tail and less and I would still make that assertion, there are some other very useful tools out there.


Figure 1: Splunk searching for events (detail)

Console.app is the most obvious for us Mac users; it's "built-in and clearly Mac-like. I still think it's a miserable way to follow log activity as it happens, but it is a fantastic tool for exploring the various logs that exist on the system. Fire up Console.app (located in the Utilities folder) and poke around. It's a great learning experience.

I've found a newer tool on the market to also be very useful, and while it's not Mac-specific, it's nice that there's an OS X version at all. Splunk is billed as a logfile search engine. It comes in a free version that will sift through up to 500MB of logs per day, and a paid version which not only removes that restriction altogether, but also adds Splunk-to-Splunk logging for log correlation across many machines, and also authentication which is missing in the free version.

Splunk alone could take up an entire column (and may very well one day soon); it is so easy to get going, that I'd recommend the download. Excellent documentation exists on the site as well. Of course, Splunk makes it really easy to search for events, but I've found that it's a nice exploratory tool in general. Splunk quickly categorizes events and can let you filter on those events. Figure 1 shows the search engine, running in a browser, looking at event type 9 on my machine.

Even cooler, perhaps, is the ability to look at the frequency of events. Figure 2 shows the frequency of events that appear in the secure.log of my machine clearly a good one to keep an eye on and reign in this kind of data.


Figure 2 Splunk showing the frequency of various events that appear in a given source.

In a small-to-medium sized organization, you could set up a single server as a log host (check the prior logging articles for instructions) and have Splunk access that log. 500MB is a lot of logging information. Enough that you shouldn't have a problem using the free version of Splunk.

Sanctified

So here's the real story with logs: you can ignore them, sure. Then when a problem crops up, you can just tell your client/employer that this is just the way computers are. There are bugs, there are problems; such is life. If you watch the logs, you'll know there's a problem, and then you'll actually have to do something about it.

Or, you can realize that this is the only way the system can speak to you and more often that not, you're warned well in advance of any catastrophic problems. You can head these off at the pass. You can keep the system providing services without interruption and keep up with real work.

Of course, the unfortunate cases do arise where you need to ascertain what happened after the fact. Did that mail get sent? Did that volume bomb on space and then recover without us really knowing? Was someone trying to brute-force a login through ssh at the same time our web server was under attack? In those cases, knowing which logs to look in and how to read them are your only resource for piecing together information after an event took place.

Media of the month: you might suspect it would be some Nine Inch Nails title, however, I'd have to disappoint: that was only a passing fancy. The real Media of the Month is, "Innumeracy: Mathematical Illiteracy and Its Consequences by John Allen Paulos. A book that I forgot I had until I lent it out. Easy, enjoyable, powerful reading.

WWDC time again! Apple has announced the dates (June 11th through the 15th, if you'd missed it), so, get ready. If you're attending, I hope to see you there in person! Until then, though, I'll see you in print next month...


Ed Marczak owns and operates Radiotope, a technology consultancy that brings enterprise solutions to small and medium-sized businesses. Outside of this piece of the puzzle, he is Executive Editor of MacTech Magazine, a husband and father, and CTO of WheresSpot, among other things. Find the missing tech piece at .

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Skype 8.52.0.138 - Voice-over-internet p...
Skype allows you to talk to friends, family and co-workers across the Internet without the inconvenience of long distance telephone charges. Using peer-to-peer data transmission technology, Skype... Read more
Bookends 13.2.6 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more
BusyContacts 1.4.0 - Fast, efficient con...
BusyContacts is a contact manager for OS X that makes creating, finding, and managing contacts faster and more efficient. It brings to contact management the same power, flexibility, and sharing... Read more
Chromium 77.0.3865.75 - Fast and stable...
Chromium is an open-source browser project that aims to build a safer, faster, and more stable way for all Internet users to experience the web. Version 77.0.3865.75: A list of changes is available... Read more
DiskCatalogMaker 7.5.5 - Catalog your di...
DiskCatalogMaker is a simple disk management tool which catalogs disks. Simple, light-weight, and fast Finder-like intuitive look and feel Super-fast search algorithm Can compress catalog data for... Read more
Alfred 4.0.4 - Quick launcher for apps a...
Alfred is an award-winning productivity application for OS X. Alfred saves you time when you search for files online or on your Mac. Be more productive with hotkeys, keywords, and file actions at... Read more
A Better Finder Rename 10.45 - File, pho...
A Better Finder Rename is the most complete renaming solution available on the market today. That's why, since 1996, tens of thousands of hobbyists, professionals and businesses depend on A Better... Read more
iFinance 4.5.11 - Comprehensively manage...
iFinance allows you to keep track of your income and spending -- from your lunchbreak coffee to your new car -- in the most convenient and fastest way. Clearly arranged transaction lists of all your... Read more
OmniGraffle Pro 7.11.3 - Create diagrams...
OmniGraffle Pro helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use... Read more
BBEdit 12.6.7 - Powerful text and HTML e...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more

Latest Forum Discussions

See All

Five Nights at Freddy's AR: Special...
Five Nights at Freddy's AR: Special Delivery is a terrifying new nightmare from developer Illumix. Last week, FNAF fans were sent into a frenzy by a short teaser for what we now know to be Special Delivery. Those in the comments were quick to... | Read more »
Rush Rally 3's new live events are...
Last week, Rush Rally 3 got updated with live events, and it’s one of the best things to happen to racing games on mobile. Prior to this update, the game already had multiplayer, but live events are more convenient in the sense that it’s somewhat... | Read more »
Why your free-to-play racer sucks
It’s been this way for a while now, but playing Hot Wheels Infinite Loop really highlights a big issue with free-to-play mobile racing games: They suck. It doesn’t matter if you’re trying going for realism, cart racing, or arcade nonsense, they’re... | Read more »
Steam Link Spotlight - The Banner Saga 3
Steam Link Spotlight is a new feature where we take a look at PC games that play exceptionally well using the Steam Link app. Our last entry talked about Terry Cavanaugh’s incredible Dicey Dungeons. Read about how it’s a great mobile experience... | Read more »
PSA: GRIS has some issues
You may or may not have seen that Devolver Digital just released GRIS on the App Store, but we wanted to do a quick public service announcement to say that you might not want to hop on buying it just yet. The puzzle platformer has come to small... | Read more »
Explore the world around you in new matc...
Got a hankering for a fresh-feeling Match-3 puzzle game that offers a unique twist? You might find exactly what you’re looking for with What a Wonderful World, a new spin on the classic mobile genre which merges entertaining puzzles with global... | Read more »
Combo Quest (Games)
Combo Quest 1.0 Device: iOS Universal Category: Games Price: $.99, Version: 1.0 (iTunes) Description: Combo Quest is an epic, time tap role-playing adventure. In this unique masterpiece, you are a knight on a heroic quest to retrieve... | Read more »
Hero Emblems (Games)
Hero Emblems 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: ** 25% OFF for a limited time to celebrate the release ** ** Note for iPhone 6 user: If it doesn't run fullscreen on your device... | Read more »
Puzzle Blitz (Games)
Puzzle Blitz 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Puzzle Blitz is a frantic puzzle solving race against the clock! Solve as many puzzles as you can, before time runs out! You have... | Read more »
Sky Patrol (Games)
Sky Patrol 1.0.1 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0.1 (iTunes) Description: 'Strategic Twist On The Classic Shooter Genre' - Indie Game Mag... | Read more »

Price Scanner via MacPrices.net

Save $150-$250 on 10.2″ WiFi + Cellular iPads...
Verizon is offering $150-$250 discounts on Apple’s new 10.2″ WiFi + Cellular iPad with service. Buy the iPad itself and save $150. Save $250 on the purchase of an iPad along with an iPhone. The fine... Read more
Apple continues to offer 13″ 2.3GHz Dual-Core...
Apple has Certified Refurbished 2017 13″ 2.3GHz Dual-Core non-Touch Bar MacBook Pros available starting at $1019. An standard Apple one-year warranty is included with each model, outer cases are new... Read more
Apple restocks 2018 MacBook Airs, Certified R...
Apple has restocked Certified Refurbished 2018 13″ MacBook Airs starting at only $849. Each MacBook features a new outer case, comes with a standard Apple one-year warranty, and is shipped free. The... Read more
Sunday Sale! 2019 27″ 5K 6-Core iMacs for $20...
B&H Photo has the new 2019 27″ 5K 6-Core iMacs on stock today and on sale for up to $250 off Apple’s MSRP. Overnight shipping is free to many locations in the US. These are the same iMacs sold by... Read more
Weekend Sale! 2019 13″ MacBook Airs for $200...
Amazon has new 2019 13″ MacBook Airs on sale for $200 off Apple’s MSRP, with prices starting at $899, each including free shipping. Be sure to select Amazon as the seller during checkout, rather than... Read more
2019 15″ MacBook Pros now on sale for $350-$4...
B&H Photo has Apple’s 2019 15″ 6-Core and 8-Core MacBook Pros on sale today for $350-$400 off MSRP, starting at $2049, with free overnight shipping available to many addresses in the US: – 2019... Read more
Buy one Apple Watch Series 5 at Verizon, get...
Buy one Apple Watch Series 5 at Verizon, and get a second Watch for 50% off. Plus save $10 on your first month of service. The fine print: “Buy Apple Watch, get another up to 50% off on us. Plus $10... Read more
Sprint offers 64GB iPhone 11 for free to new...
Sprint will include the 64GB iPhone 11 for free for new customers with an eligible trade-in in of the iPhone 7 or newer through September 19, 2019. The fine print: “iPhone 11 64GB $0/mo. iPhone 11... Read more
Verizon offers new iPhone 11 models for up to...
Verizon is offering Apple’s new iPhone 11 models for $500 off MSRP to new customers with an eligible trade-in (see list below). Discount is applied via monthly bill credits over 24 months. Verizon is... Read more
AT&T offers free $300 reward card + free...
AT&T Wireless will include a second free 64GB iPhone 11 with the purchase of one eligible iPhone at full price. They will also include a free $300 rewards card. The fine print: “Buy an elig.... Read more

Jobs Board

Student Employment (Blue *Apple* Cafe) Spri...
Student Employment (Blue Apple Cafe) Spring 2019 Penn State University Campus/Location: Penn State Brandywine Campus City: Media, PA Date Announced: 12/20/2018 Date Read more
Best Buy *Apple* Computing Master - Best Bu...
**732359BR** **Job Title:** Best Buy Apple Computing Master **Job Category:** Store Associates **Location Number:** 000171-Winchester Road-Store **Job Description:** Read more
*Apple* Mobile Master - Best Buy (United Sta...
**732324BR** **Job Title:** Apple Mobile Master **Job Category:** Store Associates **Location Number:** 000013-Fargo-Store **Job Description:** **What does a Best Read more
Best Buy *Apple* Computing Master - Best Bu...
**732455BR** **Job Title:** Best Buy Apple Computing Master **Job Category:** Sales **Location Number:** 000449-Auburn Hills-Store **Job Description:** **What does a Read more
*Apple* Mobility Pro - Best Buy (United Stat...
**732490BR** **Job Title:** Apple Mobility Pro **Job Category:** Store Associates **Location Number:** 000449-Auburn Hills-Store **Job Description:** At Best Buy, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.