TweetFollow Us on Twitter

OSX Failover - Part 1

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Network Administration

OSX Failover - Part 1

A Beginner's Guide

By Ben Greisler

Introduction

OS X Server has the capability to provide IP failover, a high availability feature that allows a secondary backup server to take over for a failed primary server. It is a great feature and can be very handy keeping your services available, but it has its limitations and constraints. We will review the basics of IP failover in this article and then expand on the concept in later issues. This is aimed at getting the beginner up and running with a minimum of hassle.

IP Failover Concepts

There are two major parts to the failover process: The primary server sending out notification that it is up and running and the secondary server monitoring the signal from the primary server. Kind of like, "Can you hear me now?" but without the primary server repeating "Good" after each question. This process is done via two daemons, heartbeatd and failoverd. Both are available on OS X Server, but not on OS X client.

On the primary server, heartbeatd sends out a message every second via port 1694 on both of the network interfaces involved in the process. This is the signal to the other machine in the failover pair that the primary is still alive and well, or at least well enough to keep a heartbeat going.

On the secondary server, failoverd listens for the heartbeat message on port 1694 on both network interfaces. If it stops receiving the heartbeat message it will start the failover process.

Initial configuration of IP failover starts in /etc/hostconfig where you define what role each server will be. We'll get into the specifics in the next section. There is a startup item at /System/Library/StartupItems/IPFailover that checks for configuration specifications and starts either heartbeatd or failoverd located in /usr/sbin as appropriate.

When failoverd on the secondary server realizes that it isn't receiving a heartbeat message, it sets off a series of events based on scripts located in /usr/libexec. The script NotifyFailover grabs the email address of failover recipient from /etc/hostconfig and sends a message to that address. It then utilizes the ProcessFailover script which will make an IP alias on a network interface, allowing the secondary server to take the IP address of the primary server. Both of these scripts are available for examination and are pretty well commented.

Another purpose of the ProcessFailover script is to execute scripts located in the /Library/IPFailover/ folder. This folder does not exist in a standard install of OS X Server and has to be created if needed. Within that folder can be 4 subfolders: PreAcq, PostAcq, PreRel and PostRel. You can utilize these folders to perform certain actions. The names are self-explanatory and define when the content scripts will be used (i.e.: before IP acquisition or after the IP release, etc). This is where the power and flexibility of IP failover resides.

More information can be found in the High Availability Administration document http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf , but it does have some incorrect information as referenced in this Apple tech article: http://docs.info.apple.com/article.html?artnum=305066

Setting up IP Failover

In this article, we will set up the most basic IP failover configuration to show that it works. In general, IP failover can be done in three easy steps:

1. Set up OSX Server on two machines with appropriate network configurations.

2. Add the appropriate entries to /etc/hostconfig on both machines.

3. Reboot each machine and have a working IP failover pair.

Easy, huh? Ok, now to the steps needed to accommodate the above.

It is best that the two machines in the failover pair be as identical as possible. You wouldn't want the machines to be on different OS versions, or have a secondary server that can't handle the load that the primary server normally handles. It is also tempting to give the secondary server other work to do while it is just sitting there listening to the heartbeat of the primary server, but refrain from that. Its job is to be a backup server, pure and simple.

We need to set up two networks for the IP failover pair to join. One will probably be your existing network that your other machines use to connect to your server. The other network will be a private network that the pair will communicate over. Typically this will be IP over Firewire. You don't have to do it this way, but it does preserve your secondary Ethernet port on machines that have one and allows a private network on machines that don't have a second Ethernet port (i.e.: MacMini).

Let's set up our networking like this:

Primary Server

192.168.254.165 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.165 on fw0

255.255.0.0 Subnet Mask

Secondary Server

192.168.254.170 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.170 on fw0

255.255.0.0 Subnet Mask

Make sure that you have good DNS entries for both machines and test them. Do not enter DNS servers or gateway information in the Firewire interface.

Now, let's edit /etc/hostconfig on each server (using your favorite editor via sudo). Add the following lines:

Primary Server

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

Secondary Server

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"

FAILOVER_PEER_IP="10.0.0.165"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

So, what does all that mean?

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"-This identifies to the primary server the IP addresses of the network interfaces of the secondary server. You can either specify the IP's of the secondary server or use the broadcast addresses for the subnet (i.e.: 192.168.254.255, 10.0.0.255)

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"-This identifies the primary interface IP of the primary server. Note the syntax of "en0:" when creating your configuration.

FAILOVER_PEER_IP="10.0.0.165"-This identifies the secondary interface on the primary server. In this case it is the Firewire port (fw0).

FAILOVER_EMAIL_RECIPIENT=user@domain.com-This is the email address of the person who needs to know about failover actions. Make sure that your machine is configured to be able to send mail. You may need to configure SMTP services.

Hook up the servers to the Ethernet network and connect a Firewire cable between the two machines. Check that you can ping each machine on each interface from each machine. Both machines need to be able to see one another. Now restart the primary machine and then the secondary. This is important because if you start the secondary machine before the primary, it won't hear the heartbeat message from the primary and will try to failover immediately.

Ok, now that each server is up and running let's test it out. On a third machine, ping the primary server's public IP address. You should get a good solid return. Now open up Console on each machine and view the System log. Using tail on /var/log/system.log so you can see what is going on with each machine, alternately pull the Firewire cable and then Ethernet cable on the primary machine. You will notice that you stop getting ping responses from the primary server. Wait a few seconds and you should see the pings start to return again. This is the secondary machine reacting to the loss of the heartbeat message from the primary machine and initiating the ProcessFailover script to allow the secondary machine to acquire the IP of the primary machine. You have just gotten IP failover to work!

To failback, I suggest not just plugging the cables back into the primary machine. In a production environment you may have to shutdown the secondary server in a controlled manner, bring the primary back on line and then bring up the secondary. This is inconvenient as it would be great if you could just have everything failback to its original state, but practice has shown that this doesn't happen exactly the way you would want it to in every case.

Conclusion

So, it's great that we can failover from one server to another, but what good does this really do us? In the next article we will start making IP failover do some tricks for us that will be useful. Stay tuned!

References:

http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf

http://docs.info.apple.com/article.html?artnum=305066

man heartbeatd

man failoverd


Ben has worked Apple based technology integration projects from Maine to Japan while learning all the way. When not collecting frequent flyer miles he spends his favorite time with his wife and 2.5 year old daughter at their home outside of Philadelphia. He can be reached at magikben@mac.com.
 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Latest Forum Discussions

See All

Top Mobile Game Discounts
Every day, we pick out a curated list of the best mobile discounts on the App Store and post them here. This list won't be comprehensive, but it every game on it is recommended. Feel free to check out the coverage we did on them in the links... | Read more »
Price of Glory unleashes its 1.4 Alpha u...
As much as we all probably dislike Maths as a subject, we do have to hand it to geometry for giving us the good old Hexgrid, home of some of the best strategy games. One such example, Price of Glory, has dropped its 1.4 Alpha update, stocked full... | Read more »
The SLC 2025 kicks off this month to cro...
Ever since the Solo Leveling: Arise Championship 2025 was announced, I have been looking forward to it. The promotional clip they released a month or two back showed crowds going absolutely nuts for the previous competitions, so imagine the... | Read more »
Dive into some early Magicpunk fun as Cr...
Excellent news for fans of steampunk and magic; the Precursor Test for Magicpunk MMORPG Crystal of Atlan opens today. This rather fancy way of saying beta test will remain open until March 5th and is available for PC - boo - and Android devices -... | Read more »
Prepare to get your mind melted as Evang...
If you are a fan of sci-fi shooters and incredibly weird, mind-bending anime series, then you are in for a treat, as Goddess of Victory: Nikke is gearing up for its second collaboration with Evangelion. We were also treated to an upcoming... | Read more »
Square Enix gives with one hand and slap...
We have something of a mixed bag coming over from Square Enix HQ today. Two of their mobile games are revelling in life with new events keeping them alive, whilst another has been thrown onto the ever-growing discard pile Square is building. I... | Read more »
Let the world burn as you have some fest...
It is time to leave the world burning once again as you take a much-needed break from that whole “hero” lark and enjoy some celebrations in Genshin Impact. Version 5.4, Moonlight Amidst Dreams, will see you in Inazuma to attend the Mikawa Flower... | Read more »
Full Moon Over the Abyssal Sea lands on...
Aether Gazer has announced its latest major update, and it is one of the loveliest event names I have ever heard. Full Moon Over the Abyssal Sea is an amazing name, and it comes loaded with two side stories, a new S-grade Modifier, and some fancy... | Read more »
Open your own eatery for all the forest...
Very important question; when you read the title Zoo Restaurant, do you also immediately think of running a restaurant in which you cook Zoo animals as the course? I will just assume yes. Anyway, come June 23rd we will all be able to start up our... | Read more »
Crystal of Atlan opens registration for...
Nuverse was prominently featured in the last month for all the wrong reasons with the USA TikTok debacle, but now it is putting all that behind it and preparing for the Crystal of Atlan beta test. Taking place between February 18th and March 5th,... | Read more »

Price Scanner via MacPrices.net

AT&T is offering a 65% discount on the ne...
AT&T is offering the new iPhone 16e for up to 65% off their monthly finance fee with 36-months of service. No trade-in is required. Discount is applied via monthly bill credits over the 36 month... Read more
Use this code to get a free iPhone 13 at Visi...
For a limited time, use code SWEETDEAL to get a free 128GB iPhone 13 Visible, Verizon’s low-cost wireless cell service, Visible. Deal is valid when you purchase the Visible+ annual plan. Free... Read more
M4 Mac minis on sale for $50-$80 off MSRP at...
B&H Photo has M4 Mac minis in stock and on sale right now for $50 to $80 off Apple’s MSRP, each including free 1-2 day shipping to most US addresses: – M4 Mac mini (16GB/256GB): $549, $50 off... Read more
Buy an iPhone 16 at Boost Mobile and get one...
Boost Mobile, an MVNO using AT&T and T-Mobile’s networks, is offering one year of free Unlimited service with the purchase of any iPhone 16. Purchase the iPhone at standard MSRP, and then choose... Read more
Get an iPhone 15 for only $299 at Boost Mobil...
Boost Mobile, an MVNO using AT&T and T-Mobile’s networks, is offering the 128GB iPhone 15 for $299.99 including service with their Unlimited Premium plan (50GB of premium data, $60/month), or $20... Read more
Unreal Mobile is offering $100 off any new iP...
Unreal Mobile, an MVNO using AT&T and T-Mobile’s networks, is offering a $100 discount on any new iPhone with service. This includes new iPhone 16 models as well as iPhone 15, 14, 13, and SE... Read more
Apple drops prices on clearance iPhone 14 mod...
With today’s introduction of the new iPhone 16e, Apple has discontinued the iPhone 14, 14 Pro, and SE. In response, Apple has dropped prices on unlocked, Certified Refurbished, iPhone 14 models to a... Read more
B&H has 16-inch M4 Max MacBook Pros on sa...
B&H Photo is offering a $360-$410 discount on new 16-inch MacBook Pros with M4 Max CPUs right now. B&H offers free 1-2 day shipping to most US addresses: – 16″ M4 Max MacBook Pro (36GB/1TB/... Read more
Amazon is offering a $100 discount on the M4...
Amazon has the M4 Pro Mac mini discounted $100 off MSRP right now. Shipping is free. Their price is the lowest currently available for this popular mini: – Mac mini M4 Pro (24GB/512GB): $1299, $100... Read more
B&H continues to offer $150-$220 discount...
B&H Photo has 14-inch M4 MacBook Pros on sale for $150-$220 off MSRP. B&H offers free 1-2 day shipping to most US addresses: – 14″ M4 MacBook Pro (16GB/512GB): $1449, $150 off MSRP – 14″ M4... Read more

Jobs Board

All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.