Volume Number: 23 (2007)
Issue Number: 01
Column Tag: Network Administration

Can RAID be used for Backup?

Try splitting off a RAID drive as part of a backup strategy.

By Tim Standing

It's 4:30 PM on a Friday and you get a call that a server you administer is down. The users are going to be working through the weekend on a project which is due on Monday and are getting frantic because many of the files they need are on the server. This is the time when you'll see whether the backup of that server is any good. The two criteria which matter to your users are: 1) How fast can you get the server back online and 2) How old are the files you are restoring. These two items will determine whether the users think you are a miracle worker or just someone who tinkers with computers. This article describes how you can use mirror volumes (RAID 1), as part of a backup strategy which can get your users to think of you as a miracle worker.

Before I get in trouble with the semanticists reading this article, I need to define what I mean by "backup." For this article, I define backup as the process of protecting files or volumes from any disaster which can destroy data. This includes: a hard disk failing, a user accidentally deleting a file, theft of a laptop, a file system getting corrupted, a virus infection, or your place of business burning to the ground (can you tell I'm a volunteer fire fighter).

For me, a backup does not include saving multiple copies of a given file or volume for several years or saving copies of all the work you've ever done; I call that archiving. If you need to maintain an audit trail for financial or legal reasons or if you want to create an offline library of all the work you have ever created, you need some method of archiving your files. You should use software which allows you to archive to tape, for long term storage or simply copy files to CD-R or DVD-R media. In either case, the media should be stored in a safe deposit box at the bank or offsite in a fireproof vault.

Most people think of mirror volumes as providing protection only from disk failures. This is the "set it and forget it" attitude to RAID; you set up the mirror volume with two disks and don't touch it until one of the disks fails. At that point, the other disk provides data protection because your files are redundantly stored on it.

The problem with the "set it and forget it" approach to mirror volumes is that it only protects you from one of the data disasters listed above: hard disk failure. If you accidentally delete a file, the file is deleted from both disks. If your mirror volume's file system becomes corrupted, the corruption is written to both disks. Likewise if your building burns down or your Mac is stolen, you have no backup copy of your data to continue your business with. These are the reasons many people think that RAID cannot be part of a good backup solution.

Let's start by reviewing some of the features of a mirror volume. Once the volume is set up, every write goes to all the disks in the volume. Therefore, every disk contains exactly the same data; if the volume is bootable, every disk in the volume is bootable. At a later date, if a disk is added to the volume, a rebuild is started. This involves reading every byte of the volume and writing it to the new disk. Since this occurs at a layer under the file system, the copying is unaffected by whether a file is open, currently being written to or locked by one of the users. At the end of the rebuild, all of the bytes on the newly added disk are identical to the bytes on all the other disks in the volume. This ensures that this new disk contains the intact copy volume, with the correct permissions, modification dates, boot data structures etc. After the rebuild completes, writes to the volume continue to go to the new disk as well as all the other disks in the mirror volume, ensuring that the new disk continues to contain an accurate copy of the volume's data.

So how can we use the fact that a RAID 1 volume has identical data on two or more disks as the cornerstone of a disk to disk backup strategy? As soon as you remove one of the disks from the mirror volume, that disk becomes a backup copy of the entire volume. It is an instantaneous snapshot of all the data on that volume taken at the moment the disk is disconnected. If something happens to the original volume, you can always attach the backup disk to another computer and mount the copy of your mirror volume. In SoftRAID, we call the main disk of a mirror volume, the primary disk and all the others are referred to as secondary disks. I will use this terminology for the rest of this article.

Let's look at a couple of ways that people are using mirror volumes for backup and, what it protects them from.

The Laptop User:

First let's look at a laptop user who travels a lot. She is out of the office most of the week and returns every couple of days to attend meetings and work with the creative personnel to develop new ideas for clients. She has set up her startup volume as a mirror with the internal disk in her laptop as the primary disk and, an external FireWire disk as a secondary disk. Every time she comes back to the office, she plugs in her secondary disk. The mirror volume starts a rebuild which copies all data from her primary disk to her secondary disk. She can continue to use her laptop as the rebuild is in progress. (The SoftRAID driver minimizes the performance degradation during a rebuild by only copying bytes when the volume is not being accessed.) When the rebuild is done, she can unplug her secondary disk and leave the office on another trip.

If anything happens to her laptop when she is on the road, it gets stolen or someone spills coffee on it, she is confident that she can get back to work as soon as she returns to the office. All she has to do is attach the secondary disk to a new computer and boot up holding down the option key. She can then select the secondary disk as her startup disk and boot up using a copy of the volume which was on her original laptop. The only data she will have lost are those changes which she made since she last connected her secondary disk.

There are several advantages to this backup method, compared to backup strategies which don't involve RAID: the secondary disk is guaranteed to be bootable if the volume is bootable, the backup can take place regardless of which files are open or in use, the backup has very little impact on the performance of the Mac (usually less than 1% of the CPU capacity when a rebuild is in progress), and there is no restore operation required before the backup can be used.

The Desktop Mac User:

The second user I want to look at is the professional who relies on his desktop Mac for his business. He is a lawyer, developer, architect or other professional whose business relies totally on the contents of his computer. If he loses the files on his computer, his business will really suffer. In addition, the time lost when dealing with a failed hard disk is money down the drain. This user relies on a three disk mirror volume. His Mac has two internal SATA disks which are used as the primary and secondary disks of a mirror volume. If one of the disks fails at any given point, the other one will take over and become the new primary disk, so his files are always protected from a hard disk failure. The third disk is an external FireWire disk which is also a secondary disk. It is stored offsite (either at home in a fireproof safe or at another safe location). Every Friday, he brings this third disk into his office and connects it to his desktop Mac. Once this disk is connected, a mirror rebuild starts automatically. He can perform this rebuild as he continues to use his Mac during the day. At the end of the day, when he shuts his Mac down, he disconnects his external secondary disk and returns it to its offsite location. This offsite secondary disk becomes his insurance against his building getting broken into or his building burning down. Like the laptop user, if his Mac gets destroyed, he can just purchase a replacement Mac and then boot up from his external secondary disk. At most, he will use one week's worth of work.

The Server administrator:

The last user I want to look at is a server administrator, running a mail server on a XServe. She can't afford to have the server go down due to a disk failure so she is also using a mirror volume with two internal disks. She also can't afford to lose a week's worth of mail, so she has chosen a more aggressive backup strategy than the professional user. Her solution is a mirror volume with 4 disks, 2 internal, and 2 external FireWire disks. The two external secondary disks are normally stored offsite in a secure location. Every morning, the FireWire disk which contains the oldest copy of the volume is attached to the server and the mirror rebuild starts. Once the mirror rebuild has completed a few hours later, the external secondary disk is returned to its safe offsite location. This ensures that at least one copy of the data on the mirror volume is always at a secure offsite location.

There is a SoftRAID user in the military who has 8 external disks and performs his daily server volume backup by using these 8 disks in rotation. This allows him to restore his server volume to any state it was in, during the last 8 days.

SoftRAID, LLC, had users of its Mac OS 9 product who were using SoftRAID for backing up their servers. We found out how they were doing their backups and listened to their requests for new features. These features were: the ability to make write protected copies of a mirror volume, and a mechanism to regulate the speed of a mirror rebuild. These requested features are designed to facilitate disk-to-disk and disk-to-disk-to-tape backups. They have been incorporated into SoftRAID 3 for Mac OS X.

SoftRAID 3 allows users to split a secondary disk off of a mirror volume as a read only copy of the volume; called a "Read-Only Secondary" volume. These disks then mount on the desktop as separate volumes which are write protected by the SoftRAID driver. Read-Only Secondary volumes are the most secure form of disk to disk backup as the user cannot modify these volumes once they are split from the original mirror volume. At a later date, the user can choose either: to add these Read-Only Secondary volumes back to the original mirror volume or to convert them to normal read/write volumes.

SoftRAID 3 also allows the user to determine how a given volume will be used; called the volume optimization setting. This setting determines how much of the disk bandwidth the SoftRAID driver will use during a mirror rebuild. The possible settings are: Server - uses the disks as much as possible, Workstation - uses the disk 50% of the time, and Digital Audio or Digital Video - waits for many seconds of inactivity on the volume before performing any i/o for a mirror rebuild.

Let's see how to setup a mirror startup volume for each of these users. I will be illustrating this by describing the steps you perform with the SoftRAID application. You can perform some of these steps with AppleRAID, the RAID software build into Mac OS X, but you will have to resort to using terminal to run the diskutil tool.

Since a mirror volume is just a volume with the identical data on more than one disk, it is easy to convert a standard Apple Disk Utility non-RAID volume into a SoftRAID mirror without erasing any of the files on the volume. The only changes which need to take place are: the volume's partition needs to be changed to a SoftRAID partition, and a small SoftRAID specific partition must be created to contain the volume's metadata. The SoftRAID application performs these steps with one menu selection.

For security reasons, Mac OS X will not allow a disk's partition map to be modified if the disk contains mounted volumes. This means that all the volumes on a disk must be unmounted before the partition map can be changed. If the volume is not your startup volume, this is easy; the SoftRAID application can just unmount the volume, change the partition map, and then remount the volume. If the disk contains your startup volume, you have to start up your Mac using a different startup volume (e.g.: the SoftRAID Startup CD or a volume on an external FireWire disk). Once the partition map has been changed, all the other steps of creating and managing a mirror volume can be performed while your Mac has started up this volume.

Setting up a Mirror Volume for the Laptop User:

1) Startup your Mac using the SoftRAID Startup CD. As I described above, you must use a different startup volume than the one on the disk you will be converting to SoftRAID. You can use either the SoftRAID Startup CD (available for purchase from www.softraid.com), use an external FireWire disk which contains a startup volume or startup your laptop in target disk mode and connect it to a second Mac. If you purchase the electronic version of SoftRAID, you will have to use the second or third method for starting up your laptop.

2) Launch the SoftRAID application. If you did not use the SoftRAID Startup CD, you will have to double click on the SoftRAID application to launch it.

3) Convert your startup volume to a SoftRAID volume. Click on the tile for the internal disk in your Mac in the Disks column. You will see that it gets connected to a volume tile which represents your startup volume in the Volumes column. Select Convert to SoftRAID 3 in the Disks menu to convert your volumes partition to a SoftRAID volume partition. This process does not change any of the files in your volume, and all your volume's data will remain intact.

4) Restart your Mac using your normal startup volume. Restart your Mac, so it is using the original startup volume. This will now be a SoftRAID volume and will have a SoftRAID non-RAID volume icon.

5) Connect the second disk you will be using for the mirror volume. Connect a second disk to your Mac. This disk must be the same size or larger than the internal disk in your laptop. If your Mac contains a PowerPC CPU, you will have to use a FireWire disk as USB disks cannot be used for startup volumes. If you Mac contains an Intel CPU, you can use either a FireWire or USB disk. (Our experience has been that USB disks are much less reliable on Mac OS X than FireWire ones on either type of CPU.)

6) Initialize the second disk. Click on the tile for the external disk you have connected and select Initialize from the Disk menu. This tells the SoftRAID application to create a partition map on the disk you have selected. You can specify the partition map type to use or let the SoftRAID application choose the appropriate one by using the Default button (APM for PowerPC Macs and GPT for Intel Macs). The partition maps for all the disks of a given volume must be the same type.

7) Convert the startup volume to a mirror. Click on the tile for your startup volume and select Convert to Mirror in the Volume menu. Then select the external disk you have just initialized. This will convert the volume to a mirror volume which contains two disks: the internal as the mirror primary disk and the external as the secondary one. It will also start a rebuild of the mirror volume. The rebuild process copies all of the data on the internal disk to the external one. Since a mirror volume always sends all writes to all the mirror disks, the two disks will contain identical volume data after the rebuild has completed, even if you have written files to the volume during the rebuild process. Both disks will continue to contain identical volume data as you use the volume, as long as they remain connected.

8) Wait for the mirror rebuild to complete. The rebuild will continue even if you quit the SoftRAID application. This is because the rebuild process is actually handled by the SoftRAID driver; the application just displays a progress indicator for the rebuild, and allows you to stop a rebuild which is in progress. Once the rebuild has finished, the driver will display a dialog telling you that the mirror volume is in sync, and all the disks contain identical data. The driver also writes an entry to the system.log file when the rebuild completes. You can view the system.log file using the Console application located in the Utilities folder.

Leaving the office with your Laptop:

1) Disconnect the external disk from your laptop. You can either shutdown your laptop, and then disconnect the external disk or disconnect it while your Mac is still running. The external disk becomes a snapshot of your internal volume taken at the exact time the external disk gets disconnected. Note that if you disconnect the external disk while your laptop is asleep, your Mac may hang when you wake it back up. This is due to a bug in the kernel which fails to keep track of external disks correctly if they are removed while the Mac is asleep.

Returning from your trip and reconnecting your second disk:

1) Reconnect the external disk to your laptop. When you return to your office and want to rebuild your mirror, you connect the external disk to your Mac. You can do this before you restart your Mac, when it is asleep or while it's running. When the SoftRAID driver detects this disk, it will recognize it as part of an existing mirror volume, and automatically start a rebuild. It will also display a dialog telling you that it has started the rebuild, and write an entry to the system.log file with the time the rebuild started. Once the rebuild has finished, the driver will display a dialog stating that the mirror volume is in sync, and all the disks contain identical volume data. It will also write another entry to the system.log file. At no point do you have to run the SoftRAID application to complete the rebuild.

Setting up a desktop or server Mac:

The process of converting the startup volume to a mirror for a desktop or server is identical to that of a laptop. For these applications, the first two disks for a mirror volume are probably going to be internal or in the slide out trays in the case of an XServe. Like with the laptop, once the startup disk has been converted to SoftRAID, all of the remaining steps can be performed while the Mac is in use. You can have a server on line and have users logged in ,while you initialize your second disk and convert your startup volume to a mirror.

Adding additional disks to your Mirror Volume:

You can add another disk to your mirror volume at any time. This means you can add disks to your mirror volume as your backup strategy becomes more advanced or as your equipment budget grows. SoftRAID allows you to have up to 16 disks associated with a given volume. (SoftRAID also allows up to 60 volumes per disk, and has been tested with over 100 disks connected to a single Mac.)

To add another disk to a mirror volume:

1) Connect the new disk to your Mac. This disk must be the same size or larger than your mirror volume. For startup volumes, this disk must be on a bus which supports booting (i.e. not USB on a PowerPC Mac). It should also offer similar performance to the other disks in the volume. For instance, adding a FireWire 800 disk to a mirror volume, which contained SATA II disks would be okay, but adding a USB disk to the same volume would impair performance.

2) Launch the SoftRAID application.

3) Initialize this new disk. Click on the tile for the external disk you have connected, and select Initialize from the Disk menu.

4) Add the disk to your mirror volume. Click on the tile in the Volumes column, which represents your mirror volume. Then select Add Secondary Disk from the Volume menu. You can then select the newly initialized disk, and it will be added to your mirror volume. It will also start a rebuild of the mirror volume.

5) Wait for the rebuild to complete. The rebuild will continue even if you quit the SoftRAID application. For faster desktop Macs and servers, a mirror rebuild will take less than 60 minutes for every 100 GB of volume size.

While a disk can simply be unplugged from a mirror volume to create a snapshot of that volume, the most reliable way of creating a copy is to split a disk off as a Read-Only Secondary volume. This creates a write protected copy of the original mirror volume on a single disk. When the disk is connected to another Mac for use with a tape backup system or to recover a group of files, the file system is locked, and none of the files on it can be modified. (The SoftRAID driver actually blocks writes to that volume, so even attempts to write to the volumes partition will fail.) At a later time, the Read-Only Secondary volume can be added back to a mirror volume. When this happens, the SoftRAID driver will start a mirror rebuild to copy all the data from the volume back onto the disk, which has just been reconnected.

The mirror volume on a server could therefore be backed using two or more external disks, both of which were split from the mirror volume to create Read-Only Secondary volumes. Every time a backup was required, the disk with the oldest Read-Only Secondary volume could be reattached and added back to the original mirror volume. This would start a mirror rebuild. Once the rebuild had completed, the disk could be split from the mirror volume as a Read-Only Secondary and stored offsite.

To Create a Read-Only Secondary Volume:

1) Launch the SoftRAID application.

2) Split one of the disks off as a Read-Only Secondary volume. Click on the tile in the Volumes column which represents your mirror volume. Then select Split Mirror from the Volume menu. You can then select the secondary disk you want to use as a backup copy of the mirror volume. Select the Read-Only Secondary option, and then click the Split button. In 5 - 10 seconds, your disk will be split off as a Read-Only Secondary volume.

3) Disconnect the disk containing the Read-Only Secondary volume. You can unmount the Read-Only Secondary volume and disconnect the disk from the Mac. It is now ready to be stored offsite.

To Add a Read-Only Secondary Volume back to the Original Mirror Volume:

1) Attach the disk containing the Read-Only Secondary volume. Connect the backup disk with the Read-Only Secondary disk to your Mac. The volume will automatically mount on the desktop.

2) Launch the SoftRAID application.

3) Add the Read-Only Secondary Volume back to the Original Mirror Volume. Click on the tile in the Volumes column which represents the Read-Only Secondary Volume. Then select Convert Read-Only Secondary Volume from the Volume menu. Use the default setting of Add back to original Mirror volume, and click the Convert button. The Read-Only Secondary volume will be added to the original mirror volume in 5 - 10 seconds, and a rebuild will be started.

Recovering from Data Disasters:

All of the data disasters you can encounter can be divided into two types: those where you need to restore an entire volume intact, and those where you only need to restore a group of files or folders. When you are restoring an entire volume, you do not have access to the original volume or cannot trust its contents. This would happen if your Mac was stolen or if all of the mirror disks inside your Mac were destroyed by a power spike. You would also want to restore an entire volume if you had a corrupted volume or your Mac was kernel panicking each time you started up.

If you are restoring a group of files or folders from a volume, chances are that the original volume is still fully functional. In this case, you only need to copy a file or group of files which were accidentally erased to your mirror volume or another safe place.

Recovering an entire volume:

If your backup disk contains a Read-Only Secondary volume, you will have to convert it to a normal volume before you can use it as a startup volume. You can do this using the SoftRAID Startup CD or by connecting it to another Mac, and running the SoftRAID application. To convert a Read-Only Secondary volume to a normal startup volume:

1) Launch the SoftRAID application.

2) Convert the Read-Only Secondary volume to a non-RAID volume. Click on the tile in the Volumes column which represents the Read-Only Secondary Volume. Then select Convert Read-Only Secondary Volume from the Volume menu. Select the Convert to non-RAID Volume, and click the Convert button. The Read-Only Secondary volume will be converted to a non-RAID volume, and will mount on the desktop.

3) Startup the Mac using your backup disk. Attach the backup disk to the replacement Mac you want to use. Then startup the Mac, and select the backup volume as your startup volume.

If your backup disk was simply disconnected from the Mac, and does not contain a Read-Only Secondary volume, you should connect it to different Mac before using it. If you are using the original Mac, you should disconnect all of the original mirror disks or the SoftRAID driver may try and start a mirror rebuild, possibly overwriting your backup of the volume.

To startup using a backup disk which contains a snapshot of the mirror volume:

1) Connect the backup disk to the Mac you want to use. Make sure the replacement Mac you are using can be started up using the partition map type, and Mac OS X version on the backup disk.

2) Disconnect any disks which were part of the mirror volume. Make sure all of the disks which were used with the original mirror volume are disconnected from the Mac. These disks contain copies of the corrupted file system, virus infection or other malady which is causing your Mac to kernel panic. If they are connected when you try and startup from the backup disk, the SoftRAID driver may start a mirror rebuild, possibly overwriting the backup disk with data from the corrupted volume.

3) Startup the Mac using your backup disk. The startup process will take 15 seconds longer than normal, the first time. This delay will occur when you see the grey apple logo. This delay is normal, and is caused by the SoftRAID driver waiting to see if any of the other disks in the mirror volume will appear. If they don't appear in 15 seconds, the SoftRAID driver will use the backup disk for the mirror volume, and start the Mac.

Remember that the usual startup disk rules apply: You can't startup an Intel Mac from a disk with an APM partition map, and similarly you can't start up a PowerPC Mac from a disk with a GPT partition map. In addition, the version of Mac OS on the backup disk must be able to boot the Mac you are trying to startup. So, don't try and startup your new replacement Intel XServe with the backup disk from your G4 XServe, which was running Panther Server.

Recovering a group of files or folders:

If your backup disk contains a Read-Only Secondary volume, you can just connect the disk to your Mac, and copy the files and folders you want off of the volume. The volume will have the same name as your mirror volume, but the icon will have a small paddle lock on it.

If your backup disk was simply disconnected from the Mac, and does not contain a Read-Only Secondary volume, you should connect it to different Mac and copy the files off using this second Mac. If you connect this backup disk to a Mac, which contains the original mirror volume, the SoftRAID driver might start a rebuild which would overwrite the files you are trying to recover.

There are several advantages to using mirror volumes as part of a backup strategy: 1)The backup preserves all permissions, file system links, and aliases perfectly. 2) Open and locked files can be backed up. 3) There is no restore step. 4) It takes less than 5 minutes to get a server back on line from the backup disk. 5) It is easy to see if your backup worked.

There are a few disadvantages to using a mirror volume for a backup: 1)The backup disk must be the same size or larger than the size of the volume. 2) The backup time is dependent on the size of the volume, not the amount of data on the volume.

Recovering the Server on Friday Afternoon:

Let's revisit the scenario I started this article with. If you had backed up this server by creating backup disks containing Read-Only Secondary volumes, you could have the server back up within 10 minutes. The hardest part would be getting a replacement Mac, the backup disk, and the necessary cables together. All you would do is convert the Read-Only Secondary volume to a non-RAID volume, and then set it to be the startup volume on your replacement Mac. You would then have a server which was exactly the same as the original server (at the time you created the backup).

If you had backed up that server with a more traditional type of backup software, how far into the restore process would you be in 10 minutes?

This article showed you viable ways to make RAID a part of your overall backup strategy. While its original use concerned high-availability, being able to split off an in sync mirror volume opens up a new avenue of data protection.

Tim Standing is the Vice President of Engineering for SoftRAID. Outside of writing code, he spends time with his family, and as a volunteer fireman