TweetFollow Us on Twitter

Mac In The Shell -- Which Log?

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Mac in the Shell

Mac In The Shell -- Which Log?

Following up on which log shows you what

by Edward Marczak

Introduction

I've had incredible response to my last two columns that talked about logs: what they are, how to interpret them and how to notify yourself if a log is telling you something important. More than any other question, however, people have asked, "which log does what?Ó While I gave an overview of some logs, there are plenty more that I haven't gone into, and more locations for logs than I could describe previously. That's what I'll be following up on this month. So, read on for even more on logs.

Kinda I Want To

Just as a review for anyone who didn't read the previous columns, logs are text files that running programs write to that keep track of their activity. Text files, that's all. (OK, an app could keep a binary log, but for the most part, text it is). This allows other apps and, more importantly, humans to read their contents. Apps are free to deal with logging on their own, or, they can use syslog to hand off their data to the system logger. It's a bit of a style issue, and usage is roughly split on which method is chosen. There's nothing that says that an application can't do both. Let's look quickly again at the syslog method.

The system logger, aka "syslogÓ, is a daemon run at boot that should be running all of the time. It listens for logging data locally, and knows where to put a logging entry based on its configuration file /etc/syslog.conf. "syslogdÓ can also be setup to listen for logging data from other machines as well. Log data can be categorized by facility and level (a "selectorÓ). The facility is basically where the data is coming from ("mailÓ, "ftpÓ, etc., along with some generic facilities). The level describes the severity of the log message. The currently defined levels are:

Emergency (level 0) Alert (level 1) Critical (level 2) Error (level 3) Warning (level 4) Notice (level 5) Info (level 6) Debug (level 7)

(Don't you love it when things neatly fit into a byte?) Looking at /etc/syslog.conf, you can then see which messages will be directed to which files. Note that some will go to multiple files.

This is a bit of a review, as all of this (and more) was covered in previous articles in this column.

Down In It

Apple has created a little bit of a split with logging: "Unix-yÓ files in /var/log, and everything else. While that may be a bit of an overgeneralization, it's a good general guideline.

syslogd, as you can see from /etc/syslog.conf, will dump just about everything in some log residing in /var/log. Additionally, several non-syslog sending services also drop their files somewhere into the /var/log hierarchy. The two big notables in this category would be apache Ð logging into /var/httpd Ð and samba, which logs into /var/samba. You'll also find some other non-syslog-ish files hanging around in /var/log, and we'll address those a little later.

The other place you'll find good logging information falls into the "everything elseÓ category. You'll find these in /Library/Logs or ~/Library/Logs. System processes will log to /Library/Logs and user processes, when the user home directory is writable, will log to ~/Library/Logs. For example, anything that needs admin level rights to run Ð AFP logs, Software Update and, my favorite, Directory Services Ð will log to the System directory /Library/Logs. Other user-space apps Ð Logic, SyncServices, Parallels, etc. Ð will log into the user's own Library/Logs.

If you've peeked into either of these directories, you'll notice that there's another category of logs: logs that get generated as the result of a crash.

Something I Can Never Have

For those of you obsessed with the running process list, you'll no doubt have noticed the persistent /usr/libexec/crashreporterd. Apple's crash reporter daemon hangs around waiting for an application to crash. Technically, it listens for a Mach exception to be generated, and, upon that happening, launches crashdump. crashdump then logs and reports the event to the user. If the user can be determined, and they have a writable home, the report goes into ~/Library/Logs/CrashReporter. You really forget how much stuff takes a dive until you peek in there:

Jack-Kerouac:~/Library/Logs/CrashReporter marczak$ ls -1
BOMArchiveHelper.crash.log
Cyberduck.crash.log
Dreamweaver.crash.log
Finder.crash.log
GrowlTunes.crash.log
Mail.crash.log
Now Up-To-Date.crash.log
Pages.crash.log
Parallels.crash.log
Preview.crash.log
Quicksilver.crash.log
Safari.crash.log
VirtueDesktops.crash.log
Workgroup Manager.crash.log
firefox-bin.crash.log
iSnip.crash.log
mdimportserver.crash.log
vmware.crash.log

Wow. When an app's owner can't be determined, is a system process, or the user's home directory is not writeable, crashdump logs into /Library/Logs:

Jack-Kerouac:/Library/Logs/CrashReporter root# ls -1 
ARDAgent.crash.log
Exited process.crash.log
httpd.crash.log
iSnip.crash.log
loginwindow.crash.log

LoginWindow crash...I like that one! crashreporter itself, however, logs its actions into /var/log/crashreporter.log. Finally, crashreporterd is also responsible for writing panic logs when the system is rebooted after a panic Ð find the log at /Library/Logs/panic.log.

Terrible Lie

If you root around /var/log long enough, you'll even start to notice some other files that aren't syslog generated at all. The most notable of the bunch are daily.out, weekly.out and monthly.out. These log files are generated by the daily, weekly and monthly periodic jobs that run overnight (if your machine is up and running, but not necessarily logged in).

Also of note are wtmp and lastlog. Interestingly, these are binary log files, and must be read with another application. These are not present for troubleshooting per se, but instead track login activity. User IDs, along with their login time get written to lastlog. From there, the utmp file gets updated with the user login information, and the same utmp record gets written to /var/log/wtmp. users, w and who use utmp to provide their information, while last and ac use wtmp.

That's What I Get

So, back to the original question: which logs record which bit of information? Here's a 40,000 ft. view:

/var/log/system.log

Along with asl.log, these are the big kahunas. Most system related activity is logged here. Specifically, the following selectors log to system.log:

*.notice
authpriv,remoteauth,ftp,install.none
kern.debug
mail.crit

As mentioned in January, asl.log pretty much gets everything, and displays the facility and level with the log message.

/var/log/secure.log

Messages from Apple's SecurityServer get logged here Ð along with anything in the authpriv facility. Good one to keep an eye on. Interestingly, though, some security-related information logs to system log as well (such as sshd authentication errors through PAM).

/var/log/mail.log and /var/log/mailaccess.log

Complimentary logs that describe mail (if you're running OS X Server's built-in mail Ð OS X Ôclient' also uses /var/log/mail.log). mail.log contains activity for postfix, SMTP sending and receiving while mailaccess.log contains cyrus' imap and pop activity. mail.log is the place to look when clients are telling you that they sent mail that the intended recipients never received (mail going out), or that someone sends them e-mail which they themselves never received (mail coming in).

/Library/Logs/DirectoryService/*

Huge. HUGE I say! Anyone operating in an OD, AD or other LDAP-reliant environment should be checking here occasionally. Of course, if you're trying to bind a workstation to a directory, and it's just not working for some reason, this is the place to check. Don't forget to throw DirectoryService into debug mode (killall -USR1 DirectoryService, logging to DirectoryService.debug.log), or use boot-time debugging (touch /Library/Preferences/DirectoryService/.DSLogDebugAtStart Ð which will also log to DirectoryService.debug.log) if the situation calls for it Ð covered in detail in past Mac In The Shell articles.

Logs that you don't need explained

There are some logs that simply don't need any explanation. These include:

/var/log/ftp.log (contents of the ftp server)
/var/log/httpd/* (Apache's log files)
/var/log/samba/* (Samba's log files)
/var/log/lpr.log (lpr printing activity)
/var/log/cups/* (CUPS web sever activity)
/Library/Logs/Software Update.log (Software Update install history)

/Library/Logs/AppleFileService Ð OK, perhaps this one does need some explanation. But not by me Ð from the engineers at Apple. This should be the Mecca of information that you reach for when you want to track activity coming in through AFP. However, details in this log are so sparse to render it useless. Despite the fact that AppleFileServer runs as one grand, monolithic process, we need it to give up the goods on what's happening internally in these logs.

The mega-log

Remember, that you can always have all syslog messages be delivered to a single file by using a wildcard selector in /etc/syslog.conf:

*.*      /var/log/mega.log

…although asl.log is already catching most of that. Further remember that this will not capture everything else that is performing logging on its own and not using syslog/asl.

The Only Time

While I made a strong case for watching and examining logs using good Ôol Unix tools Ð grep, tail and less Ð and I would still make that assertion, there are some other very useful tools out there.


Figure 1: Splunk searching for events (detail)

Console.app is the most obvious for us Mac users; it's "built-inÓ and clearly Mac-like. I still think it's a miserable way to follow log activity as it happens, but it is a fantastic tool for exploring the various logs that exist on the system. Fire up Console.app (located in the Utilities folder) and poke around. It's a great learning experience.

I've found a newer tool on the market to also be very useful, and while it's not Mac-specific, it's nice that there's an OS X version at all. Splunk is billed as a logfile search engine. It comes in a free version that will sift through up to 500MB of logs per day, and a paid version which not only removes that restriction altogether, but also adds Splunk-to-Splunk logging for log correlation across many machines, and also authentication which is missing in the free version.

Splunk alone could take up an entire column (and may very well one day soon); it is so easy to get going, that I'd recommend the download. Excellent documentation exists on the site as well. Of course, Splunk makes it really easy to search for events, but I've found that it's a nice exploratory tool in general. Splunk quickly categorizes events and can let you filter on those events. Figure 1 shows the search engine, running in a browser, looking at event type 9 on my machine.

Even cooler, perhaps, is the ability to look at the frequency of events. Figure 2 shows the frequency of events that appear in the secure.log of my machine Ð clearly a good one to keep an eye on and reign in this kind of data.


Figure 2 Ð Splunk showing the frequency of various events that appear in a given source.

In a small-to-medium sized organization, you could set up a single server as a log host (check the prior logging articles for instructions) and have Splunk access that log. 500MB is a lot of logging information. Enough that you shouldn't have a problem using the free version of Splunk.

Sanctified

So here's the real story with logs: you can ignore them, sure. Then when a problem crops up, you can just tell your client/employer that this is just the way computers are. There are bugs, there are problems; such is life. If you watch the logs, you'll know there's a problem, and then you'll actually have to do something about it.

Or, you can realize that this is the only way the system can speak to you and more often that not, you're warned well in advance of any catastrophic problems. You can head these off at the pass. You can keep the system providing services without interruption and keep up with real work.

Of course, the unfortunate cases do arise where you need to ascertain what happened after the fact. Did that mail get sent? Did that volume bomb on space and then recover without us really knowing? Was someone trying to brute-force a login through ssh at the same time our web server was under attack? In those cases, knowing which logs to look in and how to read them are your only resource for piecing together information after an event took place.

Media of the month: you might suspect it would be some Nine Inch Nails title, however, I'd have to disappoint: that was only a passing fancy. The real Media of the Month is, "Innumeracy: Mathematical Illiteracy and Its ConsequencesÓ by John Allen Paulos. A book that I forgot I had until I lent it out. Easy, enjoyable, powerful reading.

WWDC time again! Apple has announced the dates (June 11th through the 15th, if you'd missed it), so, get ready. If you're attending, I hope to see you there in person! Until then, though, I'll see you in print next month...


Ed Marczak owns and operates Radiotope, a technology consultancy that brings enterprise solutions to small and medium-sized businesses. Outside of this piece of the puzzle, he is Executive Editor of MacTech Magazine, a husband and father, and CTO of WheresSpot, among other things. Find the missing tech piece at .

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Latest Forum Discussions

See All

Whitethorn Games combines two completely...
If you have ever gone fishing then you know that it is a lesson in patience, sitting around waiting for a bite that may never come. Well, that's because you have been doing it wrong, since as Whitehorn Games now demonstrates in new release Skate... | Read more »
Call of Duty Warzone is a Waiting Simula...
It's always fun when a splashy multiplayer game comes to mobile because they are few and far between, so I was excited to see the notification about Call of Duty: Warzone Mobile (finally) launching last week and wanted to try it out. As someone who... | Read more »
Albion Online introduces some massive ne...
Sandbox Interactive has announced an upcoming update to its flagship MMORPG Albion Online, containing massive updates to its existing guild Vs guild systems. Someone clearly rewatched the Helms Deep battle in Lord of the Rings and spent the next... | Read more »
Chucklefish announces launch date of the...
Chucklefish, the indie London-based team we probably all know from developing Terraria or their stint publishing Stardew Valley, has revealed the mobile release date for roguelike deck-builder Wildfrost. Developed by Gaziter and Deadpan Games, the... | Read more »
Netmarble opens pre-registration for act...
It has been close to three years since Netmarble announced they would be adapting the smash series Solo Leveling into a video game, and at last, they have announced the opening of pre-orders for Solo Leveling: Arise. [Read more] | Read more »
PUBG Mobile celebrates sixth anniversary...
For the past six years, PUBG Mobile has been one of the most popular shooters you can play in the palm of your hand, and Krafton is celebrating this milestone and many years of ups by teaming up with hit music man JVKE to create a special song for... | Read more »
ASTRA: Knights of Veda refuse to pump th...
In perhaps the most recent example of being incredibly eager, ASTRA: Knights of Veda has dropped its second collaboration with South Korean boyband Seventeen, named so as it consists of exactly thirteen members and a video collaboration with Lee... | Read more »
Collect all your cats and caterpillars a...
If you are growing tired of trying to build a town with your phone by using it as a tiny, ineffectual shover then fear no longer, as Independent Arts Software has announced the upcoming release of Construction Simulator 4, from the critically... | Read more »
Backbone complete its lineup of 2nd Gene...
With all the ports of big AAA games that have been coming to mobile, it is becoming more convenient than ever to own a good controller, and to help with this Backbone has announced the completion of their 2nd generation product lineup with their... | Read more »
Zenless Zone Zero opens entries for its...
miHoYo, aka HoYoverse, has become such a big name in mobile gaming that it's hard to believe that arguably their flagship title, Genshin Impact, is only three and a half years old. Now, they continue the road to the next title in their world, with... | Read more »

Price Scanner via MacPrices.net

B&H has Apple’s 13-inch M2 MacBook Airs o...
B&H Photo has 13″ MacBook Airs with M2 CPUs and 256GB of storage in stock and on sale for up to $150 off Apple’s new MSRP, starting at only $849. Free 1-2 day delivery is available to most US... Read more
M2 Mac minis on sale for $100-$200 off MSRP,...
B&H Photo has Apple’s M2-powered Mac minis back in stock and on sale today for $100-$200 off MSRP. Free 1-2 day shipping is available for most US addresses: – Mac mini M2/256GB SSD: $499, save $... Read more
Mac Studios with M2 Max and M2 Ultra CPUs on...
B&H Photo has standard-configuration Mac Studios with Apple’s M2 Max & Ultra CPUs in stock today and on Easter sale for $200 off MSRP. Their prices are the lowest available for these models... Read more
Deal Alert! B&H Photo has Apple’s 14-inch...
B&H Photo has new Gray and Black 14″ M3, M3 Pro, and M3 Max MacBook Pros on sale for $200-$300 off MSRP, starting at only $1399. B&H offers free 1-2 day delivery to most US addresses: – 14″ 8... Read more
Department Of Justice Sets Sights On Apple In...
NEWS – The ball has finally dropped on the big Apple. The ball (metaphorically speaking) — an antitrust lawsuit filed in the U.S. on March 21 by the Department of Justice (DOJ) — came down following... Read more
New 13-inch M3 MacBook Air on sale for $999,...
Amazon has Apple’s new 13″ M3 MacBook Air on sale for $100 off MSRP for the first time, now just $999 shipped. Shipping is free: – 13″ MacBook Air (8GB RAM/256GB SSD/Space Gray): $999 $100 off MSRP... Read more
Amazon has Apple’s 9th-generation WiFi iPads...
Amazon has Apple’s 9th generation 10.2″ WiFi iPads on sale for $80-$100 off MSRP, starting only $249. Their prices are the lowest available for new iPads anywhere: – 10″ 64GB WiFi iPad (Space Gray or... Read more
Discounted 14-inch M3 MacBook Pros with 16GB...
Apple retailer Expercom has 14″ MacBook Pros with M3 CPUs and 16GB of standard memory discounted by up to $120 off Apple’s MSRP: – 14″ M3 MacBook Pro (16GB RAM/256GB SSD): $1691.06 $108 off MSRP – 14... Read more
Clearance 15-inch M2 MacBook Airs on sale for...
B&H Photo has Apple’s 15″ MacBook Airs with M2 CPUs (8GB RAM/256GB SSD) in stock today and on clearance sale for $999 in all four colors. Free 1-2 delivery is available to most US addresses.... Read more
Clearance 13-inch M1 MacBook Airs drop to onl...
B&H has Apple’s base 13″ M1 MacBook Air (Space Gray, Silver, & Gold) in stock and on clearance sale today for $300 off MSRP, only $699. Free 1-2 day shipping is available to most addresses in... Read more

Jobs Board

Medical Assistant - Surgical Oncology- *Apple...
Medical Assistant - Surgical Oncology- Apple Hill Location: WellSpan Medical Group, York, PA Schedule: Full Time Sign-On Bonus Eligible Remote/Hybrid Regular Apply Read more
Omnichannel Associate - *Apple* Blossom Mal...
Omnichannel Associate - Apple Blossom Mall Location:Winchester, VA, United States (https://jobs.jcp.com/jobs/location/191170/winchester-va-united-states) - Apple Read more
Cashier - *Apple* Blossom Mall - JCPenney (...
Cashier - Apple Blossom Mall Location:Winchester, VA, United States (https://jobs.jcp.com/jobs/location/191170/winchester-va-united-states) - Apple Blossom Mall Read more
Operations Associate - *Apple* Blossom Mall...
Operations Associate - Apple Blossom Mall Location:Winchester, VA, United States (https://jobs.jcp.com/jobs/location/191170/winchester-va-united-states) - Apple Read more
Business Analyst | *Apple* Pay - Banco Popu...
Business Analyst | Apple PayApply now " Apply now + Apply Now + Start applying with LinkedIn Start + Please wait Date:Mar 19, 2024 Location: San Juan-Cupey, PR Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.