TweetFollow Us on Twitter

OSX Failover - Part 1

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Network Administration

OSX Failover - Part 1

A Beginner's Guide

By Ben Greisler

Introduction

OS X Server has the capability to provide IP failover, a high availability feature that allows a secondary backup server to take over for a failed primary server. It is a great feature and can be very handy keeping your services available, but it has its limitations and constraints. We will review the basics of IP failover in this article and then expand on the concept in later issues. This is aimed at getting the beginner up and running with a minimum of hassle.

IP Failover Concepts

There are two major parts to the failover process: The primary server sending out notification that it is up and running and the secondary server monitoring the signal from the primary server. Kind of like, "Can you hear me now?" but without the primary server repeating "Good" after each question. This process is done via two daemons, heartbeatd and failoverd. Both are available on OS X Server, but not on OS X client.

On the primary server, heartbeatd sends out a message every second via port 1694 on both of the network interfaces involved in the process. This is the signal to the other machine in the failover pair that the primary is still alive and well, or at least well enough to keep a heartbeat going.

On the secondary server, failoverd listens for the heartbeat message on port 1694 on both network interfaces. If it stops receiving the heartbeat message it will start the failover process.

Initial configuration of IP failover starts in /etc/hostconfig where you define what role each server will be. We'll get into the specifics in the next section. There is a startup item at /System/Library/StartupItems/IPFailover that checks for configuration specifications and starts either heartbeatd or failoverd located in /usr/sbin as appropriate.

When failoverd on the secondary server realizes that it isn't receiving a heartbeat message, it sets off a series of events based on scripts located in /usr/libexec. The script NotifyFailover grabs the email address of failover recipient from /etc/hostconfig and sends a message to that address. It then utilizes the ProcessFailover script which will make an IP alias on a network interface, allowing the secondary server to take the IP address of the primary server. Both of these scripts are available for examination and are pretty well commented.

Another purpose of the ProcessFailover script is to execute scripts located in the /Library/IPFailover/ folder. This folder does not exist in a standard install of OS X Server and has to be created if needed. Within that folder can be 4 subfolders: PreAcq, PostAcq, PreRel and PostRel. You can utilize these folders to perform certain actions. The names are self-explanatory and define when the content scripts will be used (i.e.: before IP acquisition or after the IP release, etc). This is where the power and flexibility of IP failover resides.

More information can be found in the High Availability Administration document http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf , but it does have some incorrect information as referenced in this Apple tech article: http://docs.info.apple.com/article.html?artnum=305066

Setting up IP Failover

In this article, we will set up the most basic IP failover configuration to show that it works. In general, IP failover can be done in three easy steps:

1. Set up OSX Server on two machines with appropriate network configurations.

2. Add the appropriate entries to /etc/hostconfig on both machines.

3. Reboot each machine and have a working IP failover pair.

Easy, huh? Ok, now to the steps needed to accommodate the above.

It is best that the two machines in the failover pair be as identical as possible. You wouldn't want the machines to be on different OS versions, or have a secondary server that can't handle the load that the primary server normally handles. It is also tempting to give the secondary server other work to do while it is just sitting there listening to the heartbeat of the primary server, but refrain from that. Its job is to be a backup server, pure and simple.

We need to set up two networks for the IP failover pair to join. One will probably be your existing network that your other machines use to connect to your server. The other network will be a private network that the pair will communicate over. Typically this will be IP over Firewire. You don't have to do it this way, but it does preserve your secondary Ethernet port on machines that have one and allows a private network on machines that don't have a second Ethernet port (i.e.: MacMini).

Let's set up our networking like this:

Primary Server

192.168.254.165 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.165 on fw0

255.255.0.0 Subnet Mask

Secondary Server

192.168.254.170 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.170 on fw0

255.255.0.0 Subnet Mask

Make sure that you have good DNS entries for both machines and test them. Do not enter DNS servers or gateway information in the Firewire interface.

Now, let's edit /etc/hostconfig on each server (using your favorite editor via sudo). Add the following lines:

Primary Server

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

Secondary Server

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"

FAILOVER_PEER_IP="10.0.0.165"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

So, what does all that mean?

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"-This identifies to the primary server the IP addresses of the network interfaces of the secondary server. You can either specify the IP's of the secondary server or use the broadcast addresses for the subnet (i.e.: 192.168.254.255, 10.0.0.255)

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"-This identifies the primary interface IP of the primary server. Note the syntax of "en0:" when creating your configuration.

FAILOVER_PEER_IP="10.0.0.165"-This identifies the secondary interface on the primary server. In this case it is the Firewire port (fw0).

FAILOVER_EMAIL_RECIPIENT=user@domain.com-This is the email address of the person who needs to know about failover actions. Make sure that your machine is configured to be able to send mail. You may need to configure SMTP services.

Hook up the servers to the Ethernet network and connect a Firewire cable between the two machines. Check that you can ping each machine on each interface from each machine. Both machines need to be able to see one another. Now restart the primary machine and then the secondary. This is important because if you start the secondary machine before the primary, it won't hear the heartbeat message from the primary and will try to failover immediately.

Ok, now that each server is up and running let's test it out. On a third machine, ping the primary server's public IP address. You should get a good solid return. Now open up Console on each machine and view the System log. Using tail on /var/log/system.log so you can see what is going on with each machine, alternately pull the Firewire cable and then Ethernet cable on the primary machine. You will notice that you stop getting ping responses from the primary server. Wait a few seconds and you should see the pings start to return again. This is the secondary machine reacting to the loss of the heartbeat message from the primary machine and initiating the ProcessFailover script to allow the secondary machine to acquire the IP of the primary machine. You have just gotten IP failover to work!

To failback, I suggest not just plugging the cables back into the primary machine. In a production environment you may have to shutdown the secondary server in a controlled manner, bring the primary back on line and then bring up the secondary. This is inconvenient as it would be great if you could just have everything failback to its original state, but practice has shown that this doesn't happen exactly the way you would want it to in every case.

Conclusion

So, it's great that we can failover from one server to another, but what good does this really do us? In the next article we will start making IP failover do some tricks for us that will be useful. Stay tuned!

References:

http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf

http://docs.info.apple.com/article.html?artnum=305066

man heartbeatd

man failoverd


Ben has worked Apple based technology integration projects from Maine to Japan while learning all the way. When not collecting frequent flyer miles he spends his favorite time with his wife and 2.5 year old daughter at their home outside of Philadelphia. He can be reached at magikben@mac.com.
 
AAPL
$528.59
Apple Inc.
-3.11
MSFT
$39.78
Microsoft Corpora
-0.21
GOOG
$530.81
Google Inc.
-4.00

MacTech Search:
Community Search:

Software Updates via MacUpdate

TeamViewer 9.0.28116 - Establish remote...
TeamViewer gives you remote control of any computer or Mac over the Internet within seconds, or can be used for online meetings. Find out why more than 200 million users trust TeamViewer! Free for... Read more
Viber 4.1.0 - Send messages and make cal...
Viber lets you send free messages and make free calls to other Viber users, on any device and network, in any country! Viber syncs your contacts, messages and call history with your mobile device,... Read more
Apple iOS 7.1.1 - The latest version of...
The latest version of iOS can be downloaded through iTunes. Apple iOS 7 brings an all-new design and all-new features. Simplicity Simplicity is often equated with minimalism. Yet true simplicity is... Read more
1Password 4.3 - Powerful password manage...
1Password is a password manager that uniquely brings you both security and convenience. It is the only program that provides anti-phishing protection and goes beyond password management by adding Web... Read more
Lens Blur 1.3.0 - True out-of-focus boke...
Let Lens Blur transform your existing photo into true SLR-quality out-of-focus bokeh effect! Everyone needs a gorgeous personalized background for a social profile, blog, Web/UI design, presentation... Read more
VMware Fusion 6.0.3 - Run Windows apps a...
VMware Fusion allows you to create a Virtual Machine on your Mac and run Windows (including Windows 8.1) and Windows software on your Mac. Run your favorite Windows applications alongside Mac... Read more
Tweetbot 1.5.1 - Popular iOS twitter cli...
Tweetbot is a full-featured OS X Twitter client with a lot of personality. Whether it's the meticulously-crafted interface, sounds and animation, or features like multiple timelines and column views... Read more
Mac DVDRipper Pro 4.1.7 - Copy, backup,...
Mac DVDRipper Pro is the DVD backup solution that lets you protect your DVDs from scratches, save your batteries by reading your movies from your hard disk, manage your collection with just a few... Read more
PDFpenPro 6.2 - Advanced PDF toolkit for...
PDFpenPro allows users to edit PDF's easily. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Create... Read more
PDFpen 6.2 - Edit and annotate PDFs with...
PDFpen allows users to easily edit PDF's. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Features... Read more

Latest Forum Discussions

See All

Zeebox is Now Beamly TV – Still Offers t...
Zeebox is Now Beamly TV – Still Offers the Same Social Networking TV Fun, Plus a New Look Posted by Rob Rich on April 22nd, 2014 [ permalink ] | Read more »
Bandai Namco Unveils Upcoming Slate of M...
At their recent Global Gamers’ Day event, Bandai Namco was largely focused on their console and PC offerings for the upcoming year. However mobile still had a small presence, with some upcoming titles revealed by the company – though few were in a... | Read more »
The Story Behind Grammar Girl’s Grammar...
Learning is so much more rewarding when it’s also fun. That’s part of the reason why recent educational app, Grammar Pop HD, garnered such a positive review from us last week. The brainchild of Mignon Fogarty, otherwise known as Grammar Girl, we... | Read more »
Pinnacle Studio for iPhone Review
Pinnacle Studio for iPhone Review By Jennifer Allen on April 22nd, 2014 Our Rating: :: SIMPLY POWERFULiPhone App - Designed for the iPhone, compatible with the iPad Video editing while on the move has never been so convenient.   | Read more »
Taste Savant Review
Taste Savant Review By Jennifer Allen on April 22nd, 2014 Our Rating: :: TASTY RECOMMENDATIONSiPhone App - Designed for the iPhone, compatible with the iPad In need of some new restaurant recommendations? This app will be ideal,... | Read more »
Where’s My Water? Featuring XYY Adds Mor...
Where’s My Water? Featuring XYY Adds More Than Just Sheep to the Liquid Physics Puzzler Posted by Rob Rich on April 22nd, 2014 [ permalink ] | Read more »
FarmVille 2: Country Escape Review
FarmVille 2: Country Escape Review By Jennifer Allen on April 22nd, 2014 Our Rating: :: STEADY FARMINGUniversal App - Designed for iPhone and iPad FarmVille is bigger and brighter than ever before, but don’t expect it to sway you... | Read more »
Doctor Who: Legacy receives big update,...
Doctor Who: Legacy receives big update, adds Perk System and more Posted by Tre Lawrence on April 22nd, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Strongarm Universal Mount Hardware Revie...
Made by: Viatek Price: $14.99 for 2 Hardware/iOS Integration Rating: 3.5 out of 5 stars Usability Rating: 4 out of 5 stars Reuse Value Rating: 3.75 out of 5 stars Build Quality Rating: 3.5 out of 5 stars Overall Rating: 3.69 out of 5 stars | Read more »
Boxer Review
Boxer Review By Campbell Bird on April 21st, 2014 Our Rating: :: KO YOUR INBOXUniversal App - Designed for iPhone and iPad Turn your inbox into a task list or tune it to your own needs with this highly customizable, user friendly... | Read more »

Price Scanner via MacPrices.net

Updated iPad Price Trackers
We’ve updated our iPad Price Tracker and our iPad mini Price Tracker with the latest information on prices and availability from Apple and other resellers. Using a mobile device? We’ve also updated... Read more
Everything You Wanted To Know And Probably Mo...
Macworld UK’s Lou Hattersley takes a look inside Apple’s A7 System On Chip (SoC) , noting that its processor module is much more powerful than other smartphone chipsets. He notes that the A7 was a... Read more
Mavericks Now Runs Half Of Internet Connected...
Computerworld’s Greg Keizer reports that half of all Macs that went online in March were running OS X Mavericks with Version 10.9 accounting for the largest adoption percentage of any individual OS X... Read more
WinZip Mac 3 Simplifies Cloud Sharing And Pri...
WinZip has announced WinZip Mac 3, a new file sharing app that makes it simple to manage, protect and share files across email and the cloud. With direct support for Dropbox and Google Drive,... Read more
2.5GHz Mac mini, Apple refurbished, available...
The Apple Store has Apple Certified Refurbished 2.5GHz Mac minis available for $509, $90 off MSRP. Apple’s one-year warranty is included, and shipping is free. Read more
13-inch MacBook Pro, Apple refurbished, avail...
The Apple Store continues to offer Apple Certified Refurbished 13″ 2.5GHz MacBook Pros (4GB RAM/500GB HD) for $999 which is $200 off MSRP. Apple’s one-year warranty is included, and shipping is free. Read more
15-inch MacBook Pros (refurbished) available...
The Apple Store has Apple Certified Refurbished October 2013 15″ Retina MacBook Pros available starting at $1699, with models up to $400 off MSRP. Apple’s one-year warranty is standard, and shipping... Read more
$100 off cellular iPad minis plus free $20 gi...
Best Buy is discounting cellular iPad minis with Retina Displays by $100 on their online store for a limited time. They’ll also throw in a free $20 Best Buy gift card. Choose free shipping or free... Read more
Updated Price Trackers
We’ve updated our Mac Price Trackers with the latest information on prices, bundles, and availability on systems from Apple’s authorized internet/catalog resellers: - 15″ MacBook Pros - 13″ MacBook... Read more
Apple refurbished MacBook Airs available for...
The Apple Store has Apple Certified Refurbished 2013 MacBook Airs available for up to $200 off the cost of new models. An Apple one-year warranty is included with each MacBook, and shipping is free... Read more

Jobs Board

*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
Position Opening at *Apple* - Apple (United...
…customers purchase our products, you're the one who helps them get more out of their new Apple technology. Your day in the Apple Store is filled with a range of Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Inc. Research Data Specialist - Appl...
…of Worldwide Market Research & Intelligence. The team is responsible for conducting Apple branded consumer market research. It is also responsible for analyzing data Read more
*Apple* Automotive Parts Department position...
Apple Automotive is one of the fastest growing dealer…and it shows. Consider making the switch to the Apple Automotive Group today! At Apple Automotive, we Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.