TweetFollow Us on Twitter

OSX Failover - Part 1

Volume Number: 23 (2007)
Issue Number: 03
Column Tag: Network Administration

OSX Failover - Part 1

A Beginner's Guide

By Ben Greisler

Introduction

OS X Server has the capability to provide IP failover, a high availability feature that allows a secondary backup server to take over for a failed primary server. It is a great feature and can be very handy keeping your services available, but it has its limitations and constraints. We will review the basics of IP failover in this article and then expand on the concept in later issues. This is aimed at getting the beginner up and running with a minimum of hassle.

IP Failover Concepts

There are two major parts to the failover process: The primary server sending out notification that it is up and running and the secondary server monitoring the signal from the primary server. Kind of like, "Can you hear me now?" but without the primary server repeating "Good" after each question. This process is done via two daemons, heartbeatd and failoverd. Both are available on OS X Server, but not on OS X client.

On the primary server, heartbeatd sends out a message every second via port 1694 on both of the network interfaces involved in the process. This is the signal to the other machine in the failover pair that the primary is still alive and well, or at least well enough to keep a heartbeat going.

On the secondary server, failoverd listens for the heartbeat message on port 1694 on both network interfaces. If it stops receiving the heartbeat message it will start the failover process.

Initial configuration of IP failover starts in /etc/hostconfig where you define what role each server will be. We'll get into the specifics in the next section. There is a startup item at /System/Library/StartupItems/IPFailover that checks for configuration specifications and starts either heartbeatd or failoverd located in /usr/sbin as appropriate.

When failoverd on the secondary server realizes that it isn't receiving a heartbeat message, it sets off a series of events based on scripts located in /usr/libexec. The script NotifyFailover grabs the email address of failover recipient from /etc/hostconfig and sends a message to that address. It then utilizes the ProcessFailover script which will make an IP alias on a network interface, allowing the secondary server to take the IP address of the primary server. Both of these scripts are available for examination and are pretty well commented.

Another purpose of the ProcessFailover script is to execute scripts located in the /Library/IPFailover/ folder. This folder does not exist in a standard install of OS X Server and has to be created if needed. Within that folder can be 4 subfolders: PreAcq, PostAcq, PreRel and PostRel. You can utilize these folders to perform certain actions. The names are self-explanatory and define when the content scripts will be used (i.e.: before IP acquisition or after the IP release, etc). This is where the power and flexibility of IP failover resides.

More information can be found in the High Availability Administration document http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf , but it does have some incorrect information as referenced in this Apple tech article: http://docs.info.apple.com/article.html?artnum=305066

Setting up IP Failover

In this article, we will set up the most basic IP failover configuration to show that it works. In general, IP failover can be done in three easy steps:

1. Set up OSX Server on two machines with appropriate network configurations.

2. Add the appropriate entries to /etc/hostconfig on both machines.

3. Reboot each machine and have a working IP failover pair.

Easy, huh? Ok, now to the steps needed to accommodate the above.

It is best that the two machines in the failover pair be as identical as possible. You wouldn't want the machines to be on different OS versions, or have a secondary server that can't handle the load that the primary server normally handles. It is also tempting to give the secondary server other work to do while it is just sitting there listening to the heartbeat of the primary server, but refrain from that. Its job is to be a backup server, pure and simple.

We need to set up two networks for the IP failover pair to join. One will probably be your existing network that your other machines use to connect to your server. The other network will be a private network that the pair will communicate over. Typically this will be IP over Firewire. You don't have to do it this way, but it does preserve your secondary Ethernet port on machines that have one and allows a private network on machines that don't have a second Ethernet port (i.e.: MacMini).

Let's set up our networking like this:

Primary Server

192.168.254.165 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.165 on fw0

255.255.0.0 Subnet Mask

Secondary Server

192.168.254.170 on en0

255.255.255.0 Subnet Mask

192.168.254.1 Gateway

10.0.0.170 on fw0

255.255.0.0 Subnet Mask

Make sure that you have good DNS entries for both machines and test them. Do not enter DNS servers or gateway information in the Firewire interface.

Now, let's edit /etc/hostconfig on each server (using your favorite editor via sudo). Add the following lines:

Primary Server

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

Secondary Server

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"

FAILOVER_PEER_IP="10.0.0.165"

FAILOVER_EMAIL_RECIPIENT=user@domain.com

So, what does all that mean?

FAILOVER_BCAST_IPS="192.168.254.170 10.0.0.170"-This identifies to the primary server the IP addresses of the network interfaces of the secondary server. You can either specify the IP's of the secondary server or use the broadcast addresses for the subnet (i.e.: 192.168.254.255, 10.0.0.255)

FAILOVER_PEER_IP_PAIRS="en0:192.168.254.165"-This identifies the primary interface IP of the primary server. Note the syntax of "en0:" when creating your configuration.

FAILOVER_PEER_IP="10.0.0.165"-This identifies the secondary interface on the primary server. In this case it is the Firewire port (fw0).

FAILOVER_EMAIL_RECIPIENT=user@domain.com-This is the email address of the person who needs to know about failover actions. Make sure that your machine is configured to be able to send mail. You may need to configure SMTP services.

Hook up the servers to the Ethernet network and connect a Firewire cable between the two machines. Check that you can ping each machine on each interface from each machine. Both machines need to be able to see one another. Now restart the primary machine and then the secondary. This is important because if you start the secondary machine before the primary, it won't hear the heartbeat message from the primary and will try to failover immediately.

Ok, now that each server is up and running let's test it out. On a third machine, ping the primary server's public IP address. You should get a good solid return. Now open up Console on each machine and view the System log. Using tail on /var/log/system.log so you can see what is going on with each machine, alternately pull the Firewire cable and then Ethernet cable on the primary machine. You will notice that you stop getting ping responses from the primary server. Wait a few seconds and you should see the pings start to return again. This is the secondary machine reacting to the loss of the heartbeat message from the primary machine and initiating the ProcessFailover script to allow the secondary machine to acquire the IP of the primary machine. You have just gotten IP failover to work!

To failback, I suggest not just plugging the cables back into the primary machine. In a production environment you may have to shutdown the secondary server in a controlled manner, bring the primary back on line and then bring up the secondary. This is inconvenient as it would be great if you could just have everything failback to its original state, but practice has shown that this doesn't happen exactly the way you would want it to in every case.

Conclusion

So, it's great that we can failover from one server to another, but what good does this really do us? In the next article we will start making IP failover do some tricks for us that will be useful. Stay tuned!

References:

http://images.apple.com/server/pdfs/High_Availability_Admin_v10.4.pdf

http://docs.info.apple.com/article.html?artnum=305066

man heartbeatd

man failoverd


Ben has worked Apple based technology integration projects from Maine to Japan while learning all the way. When not collecting frequent flyer miles he spends his favorite time with his wife and 2.5 year old daughter at their home outside of Philadelphia. He can be reached at magikben@mac.com.
 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Latest Forum Discussions

See All

Can PokeMatch help you find love with Po...
The unofficial Pokemon GO companion app space has exploded almost as fast as the game itself over the last few weeks. Aspiring app developers, many of them working solo, have given us apps that locate Pokemon, keep track of the server status, and... | Read more »
How to get started with Prisma
If there's one thing people like to do more than taking pictures with their smartphones, it's tinkering with those photos in some way. Numerous apps have sprung up over the last several years that allow you to use filters and special effects to... | Read more »
6 Pokemon GO updates you can expect, acc...
Pokemon GO had a scheduled appearance at this year's San Diego Comic-Con for a while, but it was only relatively close to the show that it was upgraded to a spot in Hall H. That's the biggest venue at SDCC, one usually reserved for the largest... | Read more »
How to evolve Eevee in Pokemon GO
By now, almost everyone should be hip to how to evolve Pokemon in Pokemon GO (and if not, there's a guide for that). Just gather enough candy of the appropriate type, feed them all to the Pokemon, and evolution happens. It's a miracle that would... | Read more »
CSR Racing 2: Guide to all game modes
It might not seem like there are all that many ways to go fast in a straight line, but CSR Racing 2 begs to differ. [Read more] | Read more »
Bulb Boy (Games)
Bulb Boy 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: Multi-award winning 2D point & click horror adventure about a boy with a glowing head. | Read more »
5 top free emoji keyboard apps
If we're not at peak emoji yet as a society, it feels like we definitely should be. The emoji concept has gone far beyond what anyone in Japan could have envisioned when the people there unleashed it on an unsuspecting world, but the West has... | Read more »
How to unlock more characters in Disney...
One of the big charms of Disney Emoji Blitz is seeing a wide variety of beloved Disney and Pixar characters transformed into smiling emojis. Even someone like the sneaky Randall from Monsters Inc., who probably never cracked a smile on film, is... | Read more »
Cubway (Games)
Cubway 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Cubway is a journey with an abstract story of lifecycle of rebirth, called Samsara. Guide the cube through the long way full of dangers... | Read more »
Colorcube (Games)
Colorcube 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Turn pieces and blend colours in this minimal yet visually stunning puzzler.Over 200 handcrafted and challenging levels. Features... | Read more »

Price Scanner via MacPrices.net

External Keyboard Innovations For iPad Pro (1...
I’m an input device aficionado. With non-touchscreen computers, which includes all Macs, the keyboard and mouse or trackpad are the tactile points of interface between user and machine, and the... Read more
GSK Rheumatoid Arthritis Study Leverages iPho...
Global healthcare products company GlaxoSmithKline (GSK) says that since 2014 they have begun transforming the way they conduct research, by leveraging state-of-the-art digital technologies — a... Read more
Clearance 12-inch Retina MacBooks, Apple refu...
Apple has Certified Refurbished 2015 12″ Retina MacBooks available starting at $929. Apple will include a standard one-year warranty with each MacBook, and shipping is free. The following... Read more
13-inch Retina MacBook Pros on sale for up to...
B&H Photo has 13″ Retina MacBook Pros on sale for up to $150 off MSRP. Shipping is free, and B&H charges NY tax only: - 13″ 2.7GHz/128GB Retina MacBook Pro: $1199 $100 off MSRP - 13″ 2.7GHz/... Read more
13-inch 1.6GHz/128GB MacBook Air on sale for...
Amazon has the 13″ 1.6GHz/128GB MacBook Air on sale for $200 off MSRP for a limited time. Shipping is free: - 13″ 1.6GHz/128GB MacBook Air (sku MMGF2LL/A): $799.99 $200 off MSRP Their price is the... Read more
13-inch 1.6GHz/256GB MacBook Air on sale for...
Amazon has the 13″ 1.6GHz/256GB MacBook Air on sale for $200 off MSRP for a limited time. Shipping is free: - 13″ 1.6GHz/256GB MacBook Air (sku MMGG2LL/A): $999.99 $200 off MSRP Their price is the... Read more
Free iOS Business App notably* Helps Service...
PayStudio Inc. has introduced their new business app notably* 1.0, developed for iPhone and iPod touch. notably* was specifically developed to help service and trade professionals go digital and... Read more
27-inch iMacs on sale for $200 off MSRP
Amazon has 27″ iMacs on sale for $200 off MSRP including free shipping: - 27″ 3.3GHz iMac 5K: $2099 $200 off MSRP - 27″ 3.2GHz/1TB Fusion iMac 5K: $1799.99 $200 off MSRP - 27″ 3.2GHz/1TB HD iMac 5K... Read more
Mac Pros on sale for $200 off MSRP
B&H Photo has Mac Pros on sale for $200 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3799, $200... Read more
Save up to $600 with Apple refurbished Mac Pr...
Apple has Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The following... Read more

Jobs Board

*Apple* Retail - Multiple Positions, Willow...
Job Description:SalesSpecialist - Retail Customer Service and SalesTransform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Evangelist - JAMF Software (United S...
The Apple Evangelist is responsible for building and cultivating strategic relationships with Apple 's small and mid-market business development field teams. This Read more
*Apple* Solutions Consultant - APPLE (United...
Job Summary As an Apple Solutions Consultant, you'll be the link between our future customers and our products. You'll showcase your entrepreneurial spirit as you Read more
*Apple* Professional Learning Specialist - A...
Job Summary The Apple Professional Learning Specialist is a full-time position for one year with Apple in the Phoenix, AZ area. This position requires a high Read more
*Apple* Picker - Apple Hill Orchard (United...
Apple Hill Orchard, Co. Rte. 21,Whitehall, NY 9/7/16-10/228/16. Pick fresh market or processing apples Productivity of 60 boxes and 80 boxes processing fruit per Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.