TweetFollow Us on Twitter

OSX Failover - Part 1

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Network Administration

OSX Failover - Part 1

A Beginner's Guide

By Ben Greisler

Introduction

OS X Server has the capability to provide IP failover, a high availability feature that allows a secondary backup server to take over for a failed primary server. It is a great feature and can be very handy keeping your services available, but it has its limitations and constraints. We will review the basics of IP failover in this article and then expand on the concept in later issues. This is aimed at getting the beginner up and running with a minimum of hassle.

IP Failover Concepts

There are two major parts to the failover process: The primary server sending out notification that it is up and running and the secondary server monitoring the signal from the primary server. Kind of like, "Can you hear me now?" but without the primary server repeating "Good" after each question. This process is done via two daemons, heartbeatd and failoverd. Both are available on OS X Server, but not on OS X client.

On the primary server, heartbeatd sends out a message every second via port 1694 on both of the network interfaces involved in the process. This is the signal to the other machine in the failover pair that the primary is still alive and well, or at least well enough to keep a heartbeat going.

On the secondary server, failoverd listens for the heartbeat message on port 1694 on both network interfaces. If it stops receiving the heartbeat message it will start the failover process.

Initial configuration of IP failover starts in /etc/hostconfig where you define what role each server will be. We'll get into the specifics in the next section. There is a startup item at /System/Library/StartupItems/IPFailover that checks for configuration specifications and starts either heartbeatd or failoverd located in /usr/sbin as appropriate.

When failoverd on the secondary server realizes that it isn't receiving a heartbeat message, it sets off a series of events based on scripts located in /usr/libexec. The script NotifyFailover grabs the email address of failover recipient from /etc/hostconfig and sends a message to that address. It then utilizes the ProcessFailover script which will make an IP alias on a network interface, allowing the secondary server to take the IP address of the primary server. Both of these scripts are available for examination and are pretty well commented.

Another purpose of the ProcessFailover script is to execute scripts located in the /Library/IPFailover/ folder. This folder does not exist in a standard install of OS X Server and has to be created if needed. Within that folder can be 4 subfolders: PreAcq, PostAcq, PreRel and PostRel. You can utilize these folders to perform certain actions. The names are self-explanatory and define when the content scripts will be used (i.e.: before IP acquisition or after the IP release, etc). This is where the power and flexibility of IP failover resides.

More information can be found in the High Availability Administration document http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf , but it does have some incorrect information as referenced in this Apple tech article: http://docs.info.apple.com/article.html?artnum=305066

Setting up IP Failover

In this article, we will set up the most basic IP failover configuration to show that it works. In general, IP failover can be done in three easy steps:

1. Set up OSX Server on two machines with appropriate network configurations.

2. Add the appropriate entries to /etc/hostconfig on both machines.

3. Reboot each machine and have a working IP failover pair.

Easy, huh? Ok, now to the steps needed to accommodate the above.

It is best that the two machines in the failover pair be as identical as possible. You wouldn't want the machines to be on different OS versions, or have a secondary server that can't handle the load that the primary server normally handles. It is also tempting to give the secondary server other work to do while it is just sitting there listening to the heartbeat of the primary server, but refrain from that. Its job is to be a backup server, pure and simple.

We need to set up two networks for the IP failover pair to join. One will probably be your existing network that your other machines use to connect to your server. The other network will be a private network that the pair will communicate over. Typically this will be IP over Firewire. You don't have to do it this way, but it does preserve your secondary Ethernet port on machines that have one and allows a private network on machines that don't have a second Ethernet port (i.e.: MacMini).

Let's set up our networking like this:

Primary Server

192.168.254.165 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.165 on fw0

255.255.0.0 Subnet Mask

Secondary Server

192.168.254.170 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.170 on fw0

255.255.0.0 Subnet Mask

Make sure that you have good DNS entries for both machines and test them. Do not enter DNS servers or gateway information in the Firewire interface.

Now, let's edit /etc/hostconfig on each server (using your favorite editor via sudo). Add the following lines:

Primary Server

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

Secondary Server

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"

FAILOVER_PEER_IP="10.0.0.165"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

So, what does all that mean?

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"-This identifies to the primary server the IP addresses of the network interfaces of the secondary server. You can either specify the IP's of the secondary server or use the broadcast addresses for the subnet (i.e.: 192.168.254.255, 10.0.0.255)

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"-This identifies the primary interface IP of the primary server. Note the syntax of "en0:" when creating your configuration.

FAILOVER_PEER_IP="10.0.0.165"-This identifies the secondary interface on the primary server. In this case it is the Firewire port (fw0).

FAILOVER_EMAIL_RECIPIENT=user@domain.com-This is the email address of the person who needs to know about failover actions. Make sure that your machine is configured to be able to send mail. You may need to configure SMTP services.

Hook up the servers to the Ethernet network and connect a Firewire cable between the two machines. Check that you can ping each machine on each interface from each machine. Both machines need to be able to see one another. Now restart the primary machine and then the secondary. This is important because if you start the secondary machine before the primary, it won't hear the heartbeat message from the primary and will try to failover immediately.

Ok, now that each server is up and running let's test it out. On a third machine, ping the primary server's public IP address. You should get a good solid return. Now open up Console on each machine and view the System log. Using tail on /var/log/system.log so you can see what is going on with each machine, alternately pull the Firewire cable and then Ethernet cable on the primary machine. You will notice that you stop getting ping responses from the primary server. Wait a few seconds and you should see the pings start to return again. This is the secondary machine reacting to the loss of the heartbeat message from the primary machine and initiating the ProcessFailover script to allow the secondary machine to acquire the IP of the primary machine. You have just gotten IP failover to work!

To failback, I suggest not just plugging the cables back into the primary machine. In a production environment you may have to shutdown the secondary server in a controlled manner, bring the primary back on line and then bring up the secondary. This is inconvenient as it would be great if you could just have everything failback to its original state, but practice has shown that this doesn't happen exactly the way you would want it to in every case.

Conclusion

So, it's great that we can failover from one server to another, but what good does this really do us? In the next article we will start making IP failover do some tricks for us that will be useful. Stay tuned!

References:

http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf

http://docs.info.apple.com/article.html?artnum=305066

man heartbeatd

man failoverd


Ben has worked Apple based technology integration projects from Maine to Japan while learning all the way. When not collecting frequent flyer miles he spends his favorite time with his wife and 2.5 year old daughter at their home outside of Philadelphia. He can be reached at magikben@mac.com.
 

Community Search:
MacTech Search:

Software Updates via MacUpdate

WhatsApp 0.2.5862 - Desktop client for W...
WhatsApp is the desktop client for WhatsApp Messenger, a cross-platform mobile messaging app which allows you to exchange messages without having to pay for SMS. WhatsApp Messenger is available for... Read more
Things 3.1.3 - Elegant personal task man...
Things is a task management solution that helps to organize your tasks in an elegant and intuitive way. Things combines powerful features with simplicity through the use of tags and its intelligent... Read more
BetterTouchTool 2.292 - Customize Multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
Things 3.1.3 - Elegant personal task man...
Things is a task management solution that helps to organize your tasks in an elegant and intuitive way. Things combines powerful features with simplicity through the use of tags and its intelligent... Read more
BetterTouchTool 2.292 - Customize Multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
Bookends 12.8.3 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more
Mellel 3.5.5 - The word processor for sc...
Mellel is the leading word processor for OS X and has been widely considered the industry standard since its inception. Mellel focuses on writers and scholars for technical writing and multilingual... Read more
Mellel 3.5.5 - The word processor for sc...
Mellel is the leading word processor for OS X and has been widely considered the industry standard since its inception. Mellel focuses on writers and scholars for technical writing and multilingual... Read more
Bookends 12.8.3 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more
Carbon Copy Cloner 4.1.18 - Easy-to-use...
Carbon Copy Cloner backups are better than ordinary backups. Suppose the unthinkable happens while you're under deadline to finish a project: your Mac is unresponsive and all you hear is an ominous,... Read more

Little Red Lie (Games)
Little Red Lie 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: ARE YOU MORE AFRAID OF POVERTY THAN DEATH? Little Red Lie is a narrative-focused, interactive fiction experience that reduces... | Read more »
You can now apply to be Clash of Clans...
Earlier this month, word got out that the Builder, the trusty handiman who tirelessly built every single building inevery singleClash of Clansbase had called it quits. Sick of seeing his work destroyed endless, the Builder has set out for our world... | Read more »
Meshi Quest beginner's guide - how...
Meshi Quest is Square Enix's newest free-to-play release, and it's a real charmer. You start off as the head of a sushi restaurant, upgrading your food and equipment as you serve visitors heaping helpings of your delicious meals. As you progress,... | Read more »
BUST-A-MOVE JOURNEY (Games)
BUST-A-MOVE JOURNEY 1.0.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.0 (iTunes) Description: BUST-A-MOVE Features:- Shoot bubbles and match 3 or more bubbles of the same color to make them pop!- Complete your... | Read more »
The best card combos in Clash Royale
Clash Royale is all about building a deck of units that synergise well. To help you get off to a flying start, we've put together a list of unit combinations that are incredibly effective. Looking for some choice 2v2 combos? Check out our guide. [... | Read more »
The best 2v2 card combos in Clash Royale
2v2 is making it's grand return toClash Royalequite soon. 2v2 has quickly become one of the game's most popular gameplay modes, though they still have yet to make it a permanent fixture in the game. 2v2 is exciting and adds some new flavor to... | Read more »
The best games we played this week - Aug...
Another busy week has come to a close. We played a lot of excellent games this week and now it's time to look back and reflect on some our favorites. Here are our picks for the week of August 18. [Read more] | Read more »
War Wings beginner's guide - how to...
War Wings is the newest project from well-established game maker Miniclip. It's a World War II aerial dogfighting game with loads of different airplane models to unlock and battle. The game offers plenty of single player and multiplayer action. We... | Read more »
How to win every 2v2 battle in Clash Roy...
2v2 is coming back to Clash Royale in a big way. Although it's only been available for temporary periods of time, 2v2 has seen a hugely positive fan response, with players clamoring for more team-based gameplay. Soon we'll get yet another taste of... | Read more »
Roll to Win with Game of Dice’s new upda...
Joycity’s hit Game of Dice gets a big new update this week, introducing new maps, mechanics, and even costumes. The update sets players loose on an exciting new map, The Cursed Tower, that allows folks to use special Runes mid-match. If you feel... | Read more »

Price Scanner via MacPrices.net

2016 15-inch 2.6GHz Touch Bar MacBook Pro ava...
B&H Photo has clearance 2016 15″ 2.6GHz MacBook Pros in stock today and on sale for $500 off original MSRP. Shipping is free, and B&H charges NY & NJ sales tax only: – 15″ 2.6GHz Touch... Read more
21-inch 2.3GHz iMac on sale for $999, save $1...
Amazon has the new 2017 21″ 2.3GHz iMac (MMQA2LL/A) in stock and on sale for $999.99 including free shipping. Their price is $100 off MSRP, and it’s the lowest price available for this model. Read more
Free Instant Translator 2.0 App For iOS Relea...
Mobile application development company, Neoappz has announced the release and immediate availability of Instant Translator 2.0 for iOS devices. Instant Translator is a user-friendly application which... Read more
2017 15-inch MacBook Pros on sale for $200 of...
Amazon has 2017 15″ MacBook Pros on sale for $200 off MSRP. Shipping is free: – 15″ 2.8GHz MacBook Pro Space Gray: $2199.99, $200 off MSRP – 15″ 2.8GHz MacBook Pro Silver: $2296, $103 off MSRP – 15″... Read more
Apple’s 2017 Back to School Promotion: Free B...
Purchase a new Mac using Apple’s Education discount, and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free. As part... Read more
Clearance 2016 12-inch Retina MacBooks, Apple...
Apple has Certified Refurbished 2016 12″ Retina MacBooks available starting at $1019. Apple will include a standard one-year warranty with each MacBook, and shipping is free. The following... Read more
15-inch 2.2GHz Retina MacBook Pro, Apple refu...
Apple has Certified Refurbished 2015 15″ 2.2GHz Retina MacBook Pros available for $1699. That’s $300 off MSRP, and it’s the lowest price available for a 15″ MacBook Pro. An Apple one-year warranty is... Read more
Apple refurbished Mac minis available startin...
Apple has Certified Refurbished Mac minis available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: – 1.4GHz Mac mini: $419 $80 off MSRP – 2.6GHz Mac... Read more
Apple refurbished iPad Pros available startin...
Apple has Certified Refurbished 2016 12″ WiFi iPad Pros available starting at $589. An Apple one-year warranty is included with each model, and shipping is free: – 32GB 12″ iPad Pro WiFi: $589... Read more
Weekend sale: 13-inch MacBook Pros for up to...
Amazon has new 2017 13″ MacBook Pros on sale today for up to $200 off MSRP, each including free shipping: – 13″ 3.1GHz/256GB Space Gray MacBook Pro (MPXV2LL/A): $1599.99 $200 off MSRP – 13″ 3.1GHz/... Read more

Jobs Board

Development Operations and Site Reliability E...
Development Operations and Site Reliability Engineer, Apple Payment Gateway Job Number: 57572631 Santa Clara Valley, California, United States Posted: Jul. 27, 2017 Read more
Frameworks Engineering Manager, *Apple* Wat...
Frameworks Engineering Manager, Apple Watch Job Number: 41632321 Santa Clara Valley, California, United States Posted: Jun. 15, 2017 Weekly Hours: 40.00 Job Summary Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Development Operations and Site Reliability E...
Development Operations and Site Reliability Engineer, Apple Payment Gateway Job Number: 57572631 Santa Clara Valley, California, United States Posted: Jul. 27, 2017 Read more
Frameworks Engineering Manager, *Apple* Wat...
Frameworks Engineering Manager, Apple Watch Job Number: 41632321 Santa Clara Valley, California, United States Posted: Jun. 15, 2017 Weekly Hours: 40.00 Job Summary Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.