Volume Number: 17 (2001)
Issue Number: 08
Column Tag: Mac OS X

What happened to my System Folder?

by Dan Wood, Alameda CA

Learning The Directory Structure of Mac OS X

Longtime Mac users have gotten used to the layout of a Macintosh system. The System Folder holds all the files distributed by Apple, along with anything that you or your applications installed. Your files and applications can go anywhere, though predefined Applications and Documents folders have given you some suggestions on where to put your files. But overall, you had complete control of the layout and contents of your hard disk.

With Mac OS X, all of this has changed. The disk layout has little in common with the structure that users and developers on Mac OS 9 are used to. Getting used to this may take a little time and patience, but there is some rationality behind these changes. The goal of this article is to familiarize the developer or technical user with this new structure. In this article, I'll be introducing the new structure of the Mac OS X volume and concepts of domains; I'll delve into the sacred Library folders; I'll touch on file packaging, and I'll expose the hidden Unix layers underneath.

Early Mac systems only had a few files on a volume — after all, we were working with 800K floppies —but by the age of hard drives, a volume was likely to contain hundreds of files. It was difficult to keep track of the contents of a System Folder in System 6. System 7 alleviated this problem by creating sub-folders to hold fonts, control panels, startup items, and so forth. This certainly helped, but soon this categorization got out of hand. Just take a look in any Extensions folder in Mac OS 9 and you'll see what I mean.

A major problem with the "classic" System Folder layout is that it was next to impossible to keep track of the files installed by Apple and the files installed by your own applications. If you were repairing a hard disk and you needed to re-install your system, the process of making a clean install and then moving over the user-installed preferences, extensions, fonts, control panels, and other files could easily suck up a day.

Older versions of Mac OS were never designed for a multi-user environment. Sure, Mac OS 9 gave us multiple users, but some inelegant retrofitting had to be performed on Apple's part to get this to work; many applications were not written with multiple users in mind and do not perform correctly in a multi-user environment.

On Mac OS X, many of the problems of maintainability and working in a multiple-user environment are gone. Although this may take some getting used to, one important paradigm shift is that the disk is not really "owned" by the users as it has been in the past. (Some of the reasons that the disk is laid out the way that it is comes from Mac OS X's BSD Unix underpinnings; other reasons come out of the OS's ancestry from NeXT.) Perhaps this seems draconian, but the fact is that computers are a lot less forgiving of missing files than humans are. By protecting the files that the Mac needs to operate from the average user, the Mac is much more likely to stay functional.

So rather than allowing the average user to put any file anywhere, the user has a home directory normally located within the /Users directory. That home directory is pre-populated with several sub-folders. Up at the top level, there is an unwritable folder called System that doesn't hold the files one would have seen in older versions of Mac OS, along with a Library directory as well. The Network icon at the highest level beckons mysteriously with several empty folders. And there are dozens of hidden directories holding even more files. Rest assured, this will all be explained.

A user looking inside the System folder may be surprised to see only one item in it, a folder called Library. What happened to all the folders and files that lived in the System Folder? On Mac OS X, a new paradigm is that these kinds of files are hidden, rather than being visible and available for accidental destruction.

Solving the problem of separating the user's files from Apple-provided folders is another paradigm shift. On Mac OS 9, many users like to label all Apple-provided files with a distinctive color after a clean install so that it's easy to visually separate the user-installed files from the Apple-installed files within the System Folder. By sorting by label in the Finder, for example, one could quickly determine which of their control panels were installed initially and which were installed by them. Mac OS X, on the other hand, separates all of the Apple-provided files from the user-installed files.

One innovation descending from NeXT is the concept of parallel directories with different owners. Once this concept is understood, much of the overall structure becomes clear. On Mac OS X, there are essentially four places, which called domains, where files that need to be found by the OS itself can live. According to Mac OS X:System Overview, a domain is "an area of the file system segregated from other domains and with structural elements identical to other domains." (Don't you love circular definitions?) Perhaps it's clearer to examine domains by enumerating through them.

The first domain where files can live is on a per-user basis, on the local hard drive. This is where a user's personal documents and preferences reside in a typical setup of Mac OS X.

The second domain is in a place accessible to everybody on the local machine, regardless of who is logged in. Useful things to put in such a directory are drivers for peripherals, Internet configurations, fonts, documents to be served by the machine if it is being used as a Web server, and so forth.

The third domain, one that is largely unused unless you happen to have your Mac configured to run on a network of Macintoshes, is the local area network. Documents, applications, and configurations relevant to an entire group of users and computers (say, a classroom or office environment) can all live on the network rather than being copied to everybody's local machines. This is a powerful but underutilized capability of Mac OS X, one that makes network administration much easier than ever before. (I expect that Apple will begin promoting this capability in the near future.)

The fourth domain is in a directory owned entirely by Apple; that is, only Apple-supplied files live there. This directory is write-protected from the non-administrative user, because a user really has no business changing the contents of that directory. Think of how much easier it is to upgrade your system if one doesn't have to sort out the Apple-supplied files from the third-party files!

These four domains for files to reside in correspond to specific directories on a Mac OS X system. If you haven't already guessed, it boils down to these locations:

Per-user: A user's home directory, located in the /Users folder  (frequently abbreviated as "~")
Per-Machine, often called "Local": / (the root, top-level directory; the contents of your hard disk)
Local area network: /Network directory
Apple-provided: /System directory

Oh yes, there's sort of a fifth domain, and that is your iDisk. Apple has been pushing integration between Mac OS X and iDisk, so naturally there is some overlap here. Many of the top-level folders in your home directory have the same names and purposes as the top-level folders on your iDisk. Most of the iDisk folders are merely suggested locations for a user's files, so they will not be discussed further.

If you start to browse around in these directories, you will begin to see some similarities in their contents. Let's look at the some of the sub-folders to be found in the four domains.

User   Local   Network   System
Applications   Applications
Desktop
Library   Library   Library   Library
Sites
Users   Users

In Mac OS 9, you will recall that the System Folder on your hard disk contain all sorts of files — Apple-installed files, files shared by everybody on one machine, documents for each user, and so forth — all overlapping. On Mac OS X, these are separated, but then still categorized by this second level. (These second-level folders are somewhat akin to the System Folder sub-folder separation found in Mac OS 9.)

• Applications: Applications can really live anywhere, but in a few cases where applications need to be found by the system (such as those providing services), they do need to reside in the Applications folders (or sub-folders thereof). Unfortunately (for consistency) or fortunately (for ease of use), Apple broke their own "rules" described here by putting both Apple-supplied apps and user-supplied apps in the same folder: There is no /System/Applications. In earlier versions of Mac OS X and Mac OS X Server, Apple apps and third-party apps lived in separate folders. Apple apparently decided it wasn't worth the user confusion of having to look in two places for their applications. Note the directory for network-based applications; this is likely an empty directory on your own installation. If you are on a Mac OS X network that is configured properly, your office-wide applications can be accessed as transparently as your own local applications. And individuals can use applications available only to themselves by installing applications in their home directory (preferably by making an Applications folder for searchability reasons described above).
• Desktop: Each user has their own Desktop folder, so that each user has a different set of documents living on their desktop. There is no single shared Desktop Folder as there is on Mac OS 9.
• Library: The Library directories, across all domains, are as close to the Mac OS 9 System Folder as you will find on Mac OS X. The library is the catch-all place for documents meant to be read and written by applications, not people. (It is described in a header file as "various user-visible documentation, support, and configuration files, resources.") In the System domain, Library holds files that are supplied by Apple and not meant to be modified. At the Local domain, Library holds files that may be modified on the local system, but do not exist on a per-user basis. Library files can live on the network, for files to be shared across a network. Finally, each user can have their own files in their own Library directory; per-user preferences files live in this folder. Library folders will be discussed in detail below.
• Sites: A Sites folder is used to hold documents to be shared over the web. Files in a user's Sites directory will be served by Apache if Apache is enabled and the URL of http://hostname/~username is accessed. (Documents to be served by the machine's "main" web site, at http://hostname, actually live in /Library/WebServer/Documents. Ah, another inconsistency.)
• Users: Users on the local hard disk contains home directories, along with a single folder for shared items. But the Network domain can contain users, which means that one's home directory can live on a network server. This means that somebody could log into any machine on a properly configured network and access their home directory, including their own files, preferences, and desktop. Yes, Mac OS X can be configured to be a pure "network computer."

It's a powerful concept to have parallel domains in this fashion. Thanks to the concept of searching domains, it's possible to place the files in one of the four main domains (System, Local, Network, and User) and have Mac OS X find the appropriate item. There is an order of searching the domains, such that more local files can override more global files. The search order moves from the most local domain (a user's home directory) then to the local machine, then to the network, and then finally the unmodifiable files supplied by Apple.

As an example, let's imagine that we want to install a new screen saver, perhaps one of the nice ones available at www.epicware.com. The folder /System/Library/Screen Savers contains the Apple-provided screen savers (Abstract, Aqua Icons, Beach, Cosmos, Custom, and Forest). It's not appropriate to place your screen saver there, because if you were to reinstall Mac OS X, you'd have to worry about losing your new file.

If a network administrator were generous enough to make some screen savers available to everybody on the network, she would install it such that it appeared in /Network/Library/Screen Savers. If you (or your machine's administrator) wanted to make the screensaver available to everybody on your local machine, you would put it into /Library/Screen Savers. (You might have to create the sub-folder called Screen Savers in /Library; that's OK.) If you want to install the screen saver to be available only to you (which might be your only option if you aren't an administrator and don't have access to the other domains), you'd put it in your home folder's Library folder in a folder called Screen Savers.

Once the new screensaver is installed in one of these places, it will be available when you go to the Screen Saver panel of System Preferences. Pretty cool!

OK, this is a magazine for developers, so it's high time we look at some code, or at least an API. Apple has provided an API for accessing the list of directories for searching in Cocoa. If you look in NSPathUtilities.h, you will see the following method:

FOUNDATION_EXPORT NSArray *NSSearchPathForDirectoriesInDomains(NSSearchPathDirectory directory, 
NSSearchPathDomainMask domainMask, BOOL expandTilde);

NSSearchPathForDirectoriesInDomains returns an array of strings which you can manipulate for searching for files (or actually writing to, if that is appropriate. The first parameter, NSSearchPathDirectory, is a constant that represents the second-level folder within a domain, as described above. The second parameter, NSSearchPathDomainMask, is a mask that determines whether all domains should be returned (including unmodifiable ones), or just user-writable domains. The third parameter is a BOOL indicating whether the "~" representing the user's home directory should be expanded to the full path or not.

Careful examination of the first parameter, NSSearchPathDirectory, reveals that there are more special directories than those listed above. Briefly, it should be noted that the GrabBag and Utilities directories, directories for developer, and documentation directories can be accessed via the above API. Such a path might come in handy for, say, an installer looking for the appropriate directory to place a particular type of application.

From Carbon, you can access domains using the familiar FindFolder() in the Folder Manager; this has been retrofitted to work with domains.

Recall the comment that Mac OS 9's Extensions folder, without further subcategorization, has too many items in it to keep track of. On Mac OS X, the Library domain is a catch-all, but fortunately, it is already divided into multiple sub-folders — several dozen, actually. In the interest of brevity, only a few interesting sub-folders will be discussed here.

Application Support is for applications to put cross-application plug-ins, miscellaneous support files, and so forth. For example, DropStuff puts its Stuffit Engine in to the Application Support folder at the Local domain; OmniWeb places bookmarks and cookie files at the User domain. (Though Apple's HI guidelines state that this is where support files should go, many applications ignore this and place their files and folders directly in the Library folder at the User domain)

ColorPickers contains different color-picking implementations. At the System domain, the four basic types (list, slider, user, and wheel) can be found. Others could be installed at other library directories in the other domains.

ColorSync holds ColorSync profiles, not surprisingly. The primitive profiles are found at the System domain; more standard profiles are found at the Local domain; others could be installed for a network or user. This sub-folder is an example of one that has its own API for finding profiles; you would use CMIterateColorSyncFolder() to iterate through all available profiles at all domains. (If you need to work with a particular third-level Library folder; check your package's API for something similar.)

CoreServices (at the System domain at /System/Library/CoreServices) is a useful folder to know about, only because many "system" applications such as the Finder, the Dock, and Software Update reside here.

Extensions holds kernel extensions, more commonly known as drivers. Apple provides many at the System domain; user-installed drivers would live at the User, Network, or Local domains.

Fonts contains, you guessed it, fonts. Basic "system" fonts are found at the System domain; more fonts reside at the Local domain; even more could reside for the network or user.

Java is used for Java support. At the System domain, you'll find Apple's Java classes. At the Local domain is a Java's "home." At the User domain, Jar files can be placed for an individual user. (Note that Apple hasn't gotten the whole java classpath mechanism fully integrated into the Mac OS X domains as of this writing.)

Preferences holds preference files at the User or Local domains, but (inconsistently) holds the preference panel packages in the System domain.

Printers holds printer drivers. Third-party drivers could be installed at the User, Local, or Network domains.

Screen Savers and Sounds contain the obvious screen savers and sound files.

StartupItems contain directories (with a specific structure) for items to be executed when the system is started. (Note that this is different from a user logging in!) At the System domain, all of the system-provided processes and services are loaded. Installing items into the Local domain allows third-party services to be launched when the machine is booted up. For example, a database server such as PostgreSQL or MySQL could be started up by placing the appropriate script and configuration file into a folder within /Library/StartupItems. (For more details, see Inside Mac OS X:System Overview on Apple's developer Website.)

One advantage of Mac OS X (which is also available to Mac OS 9 but not earlier systems) is the concept of packaging. Applications on the Mac have become so complex that a typical program tends to have dozens of support files installed alongside the main application file. Packaging means that multiple files can live in a single folder and appear as if they were a single file from the user's perspective. Files that would have lived inside the System Folder or alongside of an application can now live in the application. This simplifies the user experience, but may confound developers who are looking for the components of an application. You can look inside of a package (usually a bundled application but sometimes a document, such as an ".rtfd" file or Interface Builder ".nib" file) by using the "control" key to bring up a contextual menu and choosing Show Package Contents.

We have all heard that beneath Mac OS X lies the "Power of UNIX." Well, this means that the BSD Unix directory structure is there as well. The directories are hidden, but the Unix-specific directories are there, waiting to be discovered by developers and system administrators.

The Unix-like directories are there because so much of Mac OS X (and the software that runs on OS X) requires these directories to exist. They are hidden because for the most part, only somebody who knows anything about Unix is actually going to need to access them. To navigate around these directories, you can use the Terminal and standard Unix commands like ls and cd. If you want to see these hidden files and directories from the Finder, you can enter the following command in the Terminal:

defaults write com.apple.Finder AppleShowAllFiles YES

After restarting the Finder (by logging out and then logging in, or using Force Quit... from the Apple Menu), you will see all hidden directories as well as the hidden files that begin with ".". When you want to go back to a simpler view, repeat the above instructions with NO instead of YES.

You will see many root-level directories familiar to users of Unix systems, such as /etc, /bin, /sbin, /var, /tmp, and /usr. Apple does not encourage developers to use these directories, but provides them for compatibility with other Unix-based operating systems. (Porting and installing generic BSD Unix software — say, from the FreeBSD Ports collection at http://www.freebsd.org/ports/ — is often trivially easy, and there are countless opportunities for adding a Mac UI over command-line tools!)

Explaining the Unix directories is beyond the scope of this article — a good book on BSD Unix will help. (Be sure to read up on BSD unix variants like FreeBSD, not on Linux — they have evolved along separate paths.) There are some significant differences, such as the way that Mac OS X manages users through NetInfo, but almost everything in BSD Unix has an analog in Mac OS X.

So what happened to your System Folder? Well, it's still there in functionality, but it has been partially hidden and partially rearranged. It is a new way of thinking, but in the long run, both the average user and the power user are better off. Maintenance is simplified by separating the components that make up the operating system into separate domains. By separating Apple-provided resources from user-installed resources and by allowing such resources to be available for an individual user, a particular machine, or even an entire network of machines, administration of Macs can be much easier than it has ever been. As a developer, you need to consider these new paradigms (and new APIs) when writing your killer app.

Dan Wood is a long-time Macintosh developer who started learning about Mac OS X technologies immediately after Apple acquired NeXT. After developing on the Mac using MacForth, Lightspeed Pascal, Think Class Library, Metrowerks PowerPlant, and Java, he has recently gained experience in the Cocoa frameworks and says that "there's no going back." Mr. Wood was also the lead courseware developer on Apple's week-long training course for Mac OS X Server administration. You can reach him at dwood@karelia.com.