TweetFollow Us on Twitter

OSX Failover - Part 1

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Network Administration

OSX Failover - Part 1

A Beginner's Guide

By Ben Greisler

Introduction

OS X Server has the capability to provide IP failover, a high availability feature that allows a secondary backup server to take over for a failed primary server. It is a great feature and can be very handy keeping your services available, but it has its limitations and constraints. We will review the basics of IP failover in this article and then expand on the concept in later issues. This is aimed at getting the beginner up and running with a minimum of hassle.

IP Failover Concepts

There are two major parts to the failover process: The primary server sending out notification that it is up and running and the secondary server monitoring the signal from the primary server. Kind of like, "Can you hear me now?" but without the primary server repeating "Good" after each question. This process is done via two daemons, heartbeatd and failoverd. Both are available on OS X Server, but not on OS X client.

On the primary server, heartbeatd sends out a message every second via port 1694 on both of the network interfaces involved in the process. This is the signal to the other machine in the failover pair that the primary is still alive and well, or at least well enough to keep a heartbeat going.

On the secondary server, failoverd listens for the heartbeat message on port 1694 on both network interfaces. If it stops receiving the heartbeat message it will start the failover process.

Initial configuration of IP failover starts in /etc/hostconfig where you define what role each server will be. We'll get into the specifics in the next section. There is a startup item at /System/Library/StartupItems/IPFailover that checks for configuration specifications and starts either heartbeatd or failoverd located in /usr/sbin as appropriate.

When failoverd on the secondary server realizes that it isn't receiving a heartbeat message, it sets off a series of events based on scripts located in /usr/libexec. The script NotifyFailover grabs the email address of failover recipient from /etc/hostconfig and sends a message to that address. It then utilizes the ProcessFailover script which will make an IP alias on a network interface, allowing the secondary server to take the IP address of the primary server. Both of these scripts are available for examination and are pretty well commented.

Another purpose of the ProcessFailover script is to execute scripts located in the /Library/IPFailover/ folder. This folder does not exist in a standard install of OS X Server and has to be created if needed. Within that folder can be 4 subfolders: PreAcq, PostAcq, PreRel and PostRel. You can utilize these folders to perform certain actions. The names are self-explanatory and define when the content scripts will be used (i.e.: before IP acquisition or after the IP release, etc). This is where the power and flexibility of IP failover resides.

More information can be found in the High Availability Administration document http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf , but it does have some incorrect information as referenced in this Apple tech article: http://docs.info.apple.com/article.html?artnum=305066

Setting up IP Failover

In this article, we will set up the most basic IP failover configuration to show that it works. In general, IP failover can be done in three easy steps:

1. Set up OSX Server on two machines with appropriate network configurations.

2. Add the appropriate entries to /etc/hostconfig on both machines.

3. Reboot each machine and have a working IP failover pair.

Easy, huh? Ok, now to the steps needed to accommodate the above.

It is best that the two machines in the failover pair be as identical as possible. You wouldn't want the machines to be on different OS versions, or have a secondary server that can't handle the load that the primary server normally handles. It is also tempting to give the secondary server other work to do while it is just sitting there listening to the heartbeat of the primary server, but refrain from that. Its job is to be a backup server, pure and simple.

We need to set up two networks for the IP failover pair to join. One will probably be your existing network that your other machines use to connect to your server. The other network will be a private network that the pair will communicate over. Typically this will be IP over Firewire. You don't have to do it this way, but it does preserve your secondary Ethernet port on machines that have one and allows a private network on machines that don't have a second Ethernet port (i.e.: MacMini).

Let's set up our networking like this:

Primary Server

192.168.254.165 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.165 on fw0

255.255.0.0 Subnet Mask

Secondary Server

192.168.254.170 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.170 on fw0

255.255.0.0 Subnet Mask

Make sure that you have good DNS entries for both machines and test them. Do not enter DNS servers or gateway information in the Firewire interface.

Now, let's edit /etc/hostconfig on each server (using your favorite editor via sudo). Add the following lines:

Primary Server

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

Secondary Server

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"

FAILOVER_PEER_IP="10.0.0.165"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

So, what does all that mean?

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"-This identifies to the primary server the IP addresses of the network interfaces of the secondary server. You can either specify the IP's of the secondary server or use the broadcast addresses for the subnet (i.e.: 192.168.254.255, 10.0.0.255)

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"-This identifies the primary interface IP of the primary server. Note the syntax of "en0:" when creating your configuration.

FAILOVER_PEER_IP="10.0.0.165"-This identifies the secondary interface on the primary server. In this case it is the Firewire port (fw0).

FAILOVER_EMAIL_RECIPIENT=user@domain.com-This is the email address of the person who needs to know about failover actions. Make sure that your machine is configured to be able to send mail. You may need to configure SMTP services.

Hook up the servers to the Ethernet network and connect a Firewire cable between the two machines. Check that you can ping each machine on each interface from each machine. Both machines need to be able to see one another. Now restart the primary machine and then the secondary. This is important because if you start the secondary machine before the primary, it won't hear the heartbeat message from the primary and will try to failover immediately.

Ok, now that each server is up and running let's test it out. On a third machine, ping the primary server's public IP address. You should get a good solid return. Now open up Console on each machine and view the System log. Using tail on /var/log/system.log so you can see what is going on with each machine, alternately pull the Firewire cable and then Ethernet cable on the primary machine. You will notice that you stop getting ping responses from the primary server. Wait a few seconds and you should see the pings start to return again. This is the secondary machine reacting to the loss of the heartbeat message from the primary machine and initiating the ProcessFailover script to allow the secondary machine to acquire the IP of the primary machine. You have just gotten IP failover to work!

To failback, I suggest not just plugging the cables back into the primary machine. In a production environment you may have to shutdown the secondary server in a controlled manner, bring the primary back on line and then bring up the secondary. This is inconvenient as it would be great if you could just have everything failback to its original state, but practice has shown that this doesn't happen exactly the way you would want it to in every case.

Conclusion

So, it's great that we can failover from one server to another, but what good does this really do us? In the next article we will start making IP failover do some tricks for us that will be useful. Stay tuned!

References:

http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf

http://docs.info.apple.com/article.html?artnum=305066

man heartbeatd

man failoverd


Ben has worked Apple based technology integration projects from Maine to Japan while learning all the way. When not collecting frequent flyer miles he spends his favorite time with his wife and 2.5 year old daughter at their home outside of Philadelphia. He can be reached at magikben@mac.com.
 

Community Search:
MacTech Search:

Software Updates via MacUpdate

VMware Fusion 11.5.6 - Run Windows apps...
VMware Fusion and Fusion Pro - virtualization software for running Windows, Linux, and other systems on a Mac without rebooting. The latest version includes full support for Windows 10, macOS Mojave... Read more
Alfred 4.1 - Quick launcher for apps and...
Alfred is an award-winning productivity application for OS X. Alfred saves you time when you search for files online or on your Mac. Be more productive with hotkeys, keywords, and file actions at... Read more
Dashlane 6.2032.0 - Password manager and...
Dashlane is an award-winning service that revolutionizes the online experience by replacing the drudgery of everyday transactional processes with convenient, automated simplicity - in other words,... Read more
Skype 8.63.0.76 - Voice-over-internet ph...
Skype is a telecommunications app that provides HD video calls, instant messaging, calling to any phone number or landline, and Skype for Business for productive cooperation on the projects. This... Read more
Mellel 5.0.3 - The word processor for sc...
Mellel is the leading word processor for OS X and has been widely considered the industry standard for long form documents since its inception. Mellel focuses on writers and scholars for technical... Read more
A Better Finder Rename 11.20 - File, pho...
A Better Finder Rename is the most complete renaming solution available on the market today. That's why, since 1996, tens of thousands of hobbyists, professionals and businesses depend on A Better... Read more
TunnelBear 3.9.10 - Subscription-based p...
TunnelBear is a subscription-based virtual private network (VPN) service and companion app, enabling you to browse the internet privately and securely. Features Browse privately - Secure your data... Read more
Dropbox 103.4.383 - Cloud backup and syn...
Dropbox for Mac is a file hosting service that provides cloud storage, file synchronization, personal cloud, and client software. It is a modern workspace that allows you to get to all of your files... Read more
Daylite 2020.29.1 - Dynamic business org...
Daylite helps businesses organize themselves with tools such as shared calendars, contacts, tasks, projects, notes, and more. Enable easy collaboration with features such as task and project... Read more
HoudahSpot 5.1.5 - Advanced file-search...
HoudahSpot is a versatile desktop search tool. Use HoudahSpot to locate hard-to-find files and keep frequently used files within reach. HoudahSpot will immediately feel familiar. It works just the... Read more

Latest Forum Discussions

See All

Motorball is a car football game from No...
A few years back Noodlecake Studios announced that they would be dipping in the multiplayer gaming realm with two different games. The first of those, Golf Blitz, released a while back and has proven to be very popular. Now, the second has arrived... | Read more »
SINoALICE's latest update introduce...
SINoALICE's latest update has now arrived, adding several fan-favourite characters from popular RPG series NieR. Young Nier, Kaine, and Young Emil are available in-game as part of a limited-time crossover event set to run until August 20th. [Read... | Read more »
Rocat Jumpurr is an intense roguelite pl...
Rocat Jumpurr is a roguelite platformer from developer Mousetrap Games. You might already be familiar with it if you follow the Big Indie Pitch, where it won first place during this year's Pocket Gamer Connects London competition. Following its... | Read more »
PUBG Mobile's Play As One campaign...
Back in mid-July, we reported that PUGB Mobile had teamed up with Direct Relief to help raise money for the charity's COVID-19 response project. It focused on an in-game running challenge for players, which lead to the PUBG Mobile donating $2... | Read more »
Marvel Contest of Champions' latest...
Marvel Contest of Champions' latest motion comic has arrived, and it shows off new fighters Air-Walker and Dragon Man. Both characters are set to arrive in-game this month. [Read more] | Read more »
Clash Royale: The Road to Legendary Aren...
Supercell recently celebrated its 10th anniversary and their best title, Clash Royale, is as good as it's ever been. Even for lapsed players, returning to the game is as easy as can be. If you want to join us in picking the game back up, we've put... | Read more »
Global Spy is an intriguing 2D spy sim f...
Developer Yuyosoft Innovations' Global Spy launched last month for iOS and Android, though if you missed it at the time, we're here to tell you why it's well worth a go. This one's all about international espionage, tracking down elusive spies,... | Read more »
Distract Yourself With These Great Mobil...
There’s a lot going on right now, and I don’t really feel like trying to write some kind of pithy intro for it. All I’ll say is lots of people have been coming together and helping each other in small ways, and I’m choosing to focus on that as I... | Read more »
Hyena Squad is sci-fi turn-based strateg...
Wave Light Games has just revealed its latest release, Hyena Squad, a turn-based RPG set in a space station infested by gross aliens and the living dead. The announcement was first reported on by Touch Arcade. [Read more] | Read more »
Idle Guardians: Never Die is a pixel art...
SuperPlanet has been fairly prolific with game releases so far this year with both Evil Hunter Tycoon and Lucid Adventure releasing earlier this year. Now, they've released another idle RPG called Idle Guardians: Never Die, which you can download... | Read more »

Price Scanner via MacPrices.net

Apple restocks refurbished 2020 13″ MacBook A...
Apple has restocked Certified Refurbished 2020 13″ MacBook Airs starting at only $849 and up to $200 off the cost of new Airs. Each MacBook features a new outer case, comes with a standard Apple one-... Read more
Apple restocks clearance 2019 13″ 2.4GHz MacB...
Apple has restocked Certified Refurbished 2019 13″ 2.4GHz 4-Core Touch Bar MacBook Pros starting at $1359 and up to $560 off original MSRP. Apple’s one-year warranty is included, shipping is free,... Read more
Apple restocks refurbished iPhone XR models s...
Apple has restocked Certified Refurbished, unlocked, iPhone XR models in the refurbished section of their online store starting at $539. Each iPhone comes with Apple’s standard one-year warranty,... Read more
Price drops! $100-$200 off clearance 27″ 5K i...
B&H Photo has dropped prices on clearance, previous-generation 27″ 5K iMacs by up to $200 off Apple’s original MSRP: – 27″ 3.0GHz 6-Core 5K iMac: $1699 $100 off original MSRP – 27″ 3.1GHz 6-Core... Read more
Woot offers Apple Watch and iPhone models fro...
Amazon-owned Woot has refurbished Apple Watch and iPhone models available from $99-$749 through August 6th. According to Woot, the items may show some wear, but they have all been fully tested and... Read more
Apple’s Phil Schiller Steps Down As SVP OF Wo...
NEWS: 08.05.20 – Former Apple senior Vice President of worldwide marketing, Phil Schiller, is stepping down from his long time role at the company in order to focus on spending more time with family... Read more
Expercom offers $320 discount on the 6-core 1...
Apple reseller Expercom has the Silver 16″ 6-core MacBook Pro on sale for a limited time for $2079 shipped. Their price is $320 off Apple’s MSRP for this model, and it’s the cheapest price currently... Read more
Apple announces Education pricing for new 202...
Purchase a new 2020 iMac or iMac Pro at Apple using Apple’s Education discount, and take up to $400 off MSRP. All teachers, students, and staff of any educational institution with a .edu email... Read more
Apple reseller Expercom offers $256 discount...
Expercom has Apple’s new 2020 10-core iMac Pro available for order and on sale for $4743 shipped. Their price is $256 off Apple’s MSRP for this new model, and it’s the cheapest price we’ve seen so... Read more
Apple releases refreshed 2020 27″ iMacs with...
Apple today released updated versions of their 27″ iMacs featuring 10th generation Intel processors, SSDs across the board, a better 5K display, and improvements to the camera, speakers, and mic.... Read more

Jobs Board

Executive Team Leader GM Sales (Assistant Man...
…(Assistant Manager General Merchandise and Operations) - Apple Valley, CaliforniaApply NowJob ID:R0000082364job family:Store Managementschedule:Full Read more
Cub Foods - *Apple* Valley - Now Hiring Par...
Cub Foods - Apple Valley - Now Hiring Part Time! United States of America, Minnesota, Apple Valley New Retail Post Date 2 days ago Requisition # 122305 Sign Up Read more
Part-time Geek Squad *Apple* Consultation P...
**770829BR** **Job Title:** Part-time Geek Squad Apple Consultation Professional-Store 384(Ithaca) **Job Category:** Store Associates **Store Number or Department:** Read more
Product Manager, *Apple* Commercial Sales -...
Product Manager, Apple Commercial Sales Austin, TX, US Requisition Number:77652 As an Apple Product Manager for the Commercial Sales team at Insight, you Read more
Cub Foods - *Apple* Valley - Now Hiring Par...
Cub Foods - Apple Valley - Now Hiring Part Time! United States of America, Minnesota, Apple Valley New Retail Post Date 1 day ago Requisition # 122305 Sign Up Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.