Volume Number: 25
Issue Number: 05
Column Tag: MacEnterprise

MacEnterprise: Pain Management

Dealing with Apple firmware updates in an enterprise environment

By Greg Neagle, MacEnterprise.org

You've made fantastic progress on your systems management infrastructure for your organization's Macs. You have a method to create standard images. Maybe you are even using InstaDMG. (If you haven't heard of InstaDMG, fire up a browser and do a search now!) You have an imaging server to quickly get these images on new Macs - maybe NetInstall, NetRestore or Deploy Studio, or one of the commercial solutions. You've implemented software to install and update applications and OS patches - perhaps you are using the open-source radmind tools, or Apple Remote Desktop, or a commercial package like the Casper Suite or LANrev. You've even gotten good at using the command-line and ssh to remotely connect to a machine and troubleshoot and fix issues. Your environment is predictable, structured, and managed. Life is good.

Now that so much of your job is automated and streamlined, you are sitting in your office, browsing the web and looking at Mac tech sites. Suddenly, you see something that just ruins your whole day. Apple has just released a firmware update.

So why should a firmware update cause you heartburn? Can't you just use your fancy software distribution tool to push it out to the affected machines? Well, you can push the firmware updater, but in most cases, you can't actually install the firmware remotely or in an automated fashion! Worse, for EFI firmware updates, a person must physically press the power button long enough to start the update after a shutdown. What a pain. All your centralized automation goes out the window, and once again, you are doomed to a life of running from machine to machine - installing updates.

So now you are reading with bated breath (look it up - it's not "baited breath" - which makes me think of halitosis...) and hoping that your humble author will make all the pain go away. I am truly sorry - that is not within my power. I can't make your pain go away - I can only help you manage your pain.

There are two major challenges posed by Apple firmware updates. The first is finding out about them. Since firmware updates are generally for specific hardware models, unless you have one of everything Apple's ever shipped in your office, you won't find out about every firmware update simply by looking at Software Update on your machine. You'll need a way to find out about firmware updates that apply to all the various hardware in your organization.

The second challenge, as implied above, is deploying those updates. You cannot deploy these like you do OS updates or security updates. You'll need another method.

Let's tackle the first challenge. You have a few options. The first and most obvious is to check Mac tech websites or Apple's support downloads page at http://support.apple.com/downloads/ on a regular basis, checking for firmware updates. That's tedious, and since Apple firmware updates are relatively uncommon, something you're not likely to remember to do consistently.

Another possibility, if you have a machine running OS X Server, is to turn on the Software Update Server and monitor what it downloads. A Software Update Server will download all Apple updates without regard to which hardware it applies. You can then look through the list of downloaded updates periodically. A tip: sort the updates by Post Date, and filter the list by typing "Firmware" into the search field. This will give you a list of firmware updates (at least those for which Apple has included the word "Firmware" in its name) and puts the most recent at the top. See Figure 1 for an example.

Figure 1. Software Update Server

Again, this can be tedious, and you'll certainly find listings for updates that apply to hardware your organization does not own. Still, this works as a method of discovering what firmware updates Apple has made available.

Another possibility is to farm out the discovery: since firmware updates apply only to specific models of hardware, you could have a script that runs on every machine that you manage that runs the command-line version of Software Update (/usr/sbin/softwareupdate) and looks for available firmware updates. Here's a one-liner:

/usr/sbin/softwareupdate -l | /usr/bin/grep -i firmware

This will print available firmware updates for the current machine. You could expand on this - perhaps emailing you, or posting this info into a database. In fact, later on in this column we will expand on this technique.

Now you have a method of finding out that firmware updates are available. Next, you must find a way to actually deploy these updates. In my organization, I tried a couple of approaches. The first was simply verbal. I asked all the technical support people to check Software Update for available firmware updates whenever they had to visit a user and if there were any, to apply them. Not surprisingly, that didn't really get the job done.

The next approach I tried was to try as hard as I could to automate the application of firmware updates. I was seduced down this path by Apple's SMC firmware updates for the first generation of Intel Macs. These SMC updates were the ones that enabled Boot Camp, among other things, so I was quite anxious to get them out there. It turned out that you could automate SMC firmware updates, and I wrote some scripts to do just that. You can see them here:

http://managingosx.wordpress.com/2006/10/27/automating-firmware-updates/

and here:

http://managingosx.wordpress.com/2006/11/02/automating-intel-firmware-updates-ii-revenge-of-the-smc/

Ultimately, this was not a sustainable solution. It only handled SMC firmware updates, not any of the other types of firmware updates, and since the path to the actual firmware updater was different for each machine type, and since you had to keep track of the SMC firmware versions each updater was for, it quickly became a house of cards. You needed to continually update the scripts and the firmware installers as Apple released new hardware and new SMC updates. And there were still the EFI updates, and the Graphics Firmware updates, and the Trackpad and Keyboard Firmware updates and more to deal with.

Back to the drawing board. Approach number three was a script that ran at startup, ran softwareupdate -l and emailed me (or rather, my group of administrators) if a machine needed a firmware update. I'd then add that task to our helpdesk ticketing system for a technician to do. Needless to say, the technicians didn't like this solution very much, but it worked, to a point. Still, we were missing ideal opportunities to update the firmware when a tech visited a machine for other reasons.

If these updates could not be automated, what we wanted was something that:

Notified the user and us that a firmware update was needed for the machine. Since most of our users are not administrators, they could not actually perform the update, but at least they could notify the help desk for assistance.

If a tech was working on a machine and was logged in as an administrator, upon login they would be notified there were available firmware updates so they could install them right then.

With those desires in mind, I wrote a script to be run at each login that checked the output of softwareupdate -l for available firmware updates, and if any were found, it would display an alert informing the user to contact the help desk if they were not an administrator, or urging them to install the update if they had admin rights.

A version of the script follows. Yes, it's written in Perl, and I know all the cool kids are now using Python. Perl's support for regular expressions and straightforward support for calling system command-line tools make it a good fit for this type of script. Feel free to rewrite it in bash, Python, Ruby, or whatever makes you happy. Let's take a look.

#!/usr/bin/perl -w
use strict;
# change 'com.myorg' to your organization name
my $prefs = "com.myorg.firmwareupdatecheck";

As a nicety, we're going to set things up so that this only runs once per day - that is, if a user logs in and out several times in a single day, he or she will see a firmware update alert (if one is needed) only once a day. If you want to be more insistent, you could remove this check. If you do, and a firmware update is available, the user will be notified at every login. Alternately, you could increase the interval so that not every machine hits the Software Update server (either your local one, or Apple's via your Internet connection) every single day.

# check the last time we ran on this machine; 
# exit if we already ran today so we don't annoy too much
my $now = time;
my $lastChecked = 
   `defaults -currentHost read $prefs lastChecked 2>/dev/null`;
chomp $lastChecked;
if ($lastChecked ne "") {
  my $daysSinceLastChecked = 
     int(($now-$lastChecked)/(60*60*24));
  exit if ($daysSinceLastChecked < 1);
}

Next, we'll get a list of updates from softwareupdate and look through them to find firmware updates.

# get list of available updates from softwareupdate
my $allupdates = `softwareupdate -l | grep '^   \\* '`;
chomp $allupdates;
my @updates = split /\n/, $allupdates;
my $firmwareupdates = "";
my $firmwarelist = "";
my $otherupdates = "";
# walk through the list looking for firmware updates
for my $update (@updates) {
  $update = substr($update,5);
  if (($update =~ /[F|f]irmware/) ||
      ($update =~ /EFI/) ||
      ($update =~ /SMC/)) {
    $firmwareupdates .= "$update ";
    $firmwarelist .= "   $update\n";
  } else {
    $otherupdates .= "$update ";
  }
}

We're looking for "Firmware", "firmware", "EFI", or "SMC" in the name of each available update. Every firmware update I've seen with the exception of a 2005 "iMac G5 Sleep Light Update" has the word "Firmware" in the name, so this should get the job done.

Next, we record when we did this and what we found using defaults. This is the same file we read when we start to see if we already ran today.

# record when we checked and what we found

system "defaults -currentHost write $prefs lastChecked -int $now";
system "defaults -currentHost write $prefs availableUpdates '$firmwareupdates'";

And now the exciting part! If there are available firmware updates, we're going to notify the user. But before we do, we tell Software Update to hide all the available updates that aren't firmware updates. I do this in my environment because normally we install all the other updates with our software distribution mechanism and I didn't want to get a lot of questions about additional updates available in Software Update. You may want to handle this differently.

if ($firmwareupdates) {
  # there are available firmware updates
  if ($otherupdates) {
    # hide the non-firmware updates since I don't want 
    # users tempted to install them
    system "softwareupdate --ignore $otherupdates >/dev/null 2>&1";
  }

Admin users get one version of the alert. This version opens the Software Update application if he or she clicks Install. We embed some AppleScript and use osascript to call it. Note the "giving up after 120" clause to display alert - this causes AppleScript to close the alert after two minutes if the user ignores it.

  
  # are we running under an admin account?
  my $checkAdmin = `dseditgroup -o checkmember admin`;
  if ($checkAdmin =~ /^yes/) {
     # user is an admin, prompt them to install
           my $result = `osascript<<EOFA
try
  tell application "System Events"
    activate
    display alert "Firmware updates available" message 
"There are firmware updates available for this Mac:" & return & "$firmwarelist" as warning buttons {"Later", "Install"} default button "Install" cancel button "Later" giving up after 120
  end tell
  if button returned of the result is "Install" then
    tell application "Software Update" to activate
  end if
end try
EOFA`;

If the user is not an administrator, the user gets an alert directing him or her to call the help desk - again we use embedded AppleScript to display the alert.

  } else {
    # user is not an admin, tell them to call help desk
    my $result = `osascript<<EOFB
try
  tell application "System Events"
    activate
    display alert "Firmware updates available" message 
"There are firmware updates available for this Mac:" & return & "$firmwarelist" & return & "Please call Tech Support at 555-1212 for help in installing these updates." 
as warning buttons 
{"OK"} default button "OK" cancel button "OK" giving up after 120
  end tell
end try
EOFB`;
  }
}

If the script runs in the context of a user without admin rights and firmware updates are available for this machine, after a few seconds, the user will see an alert like the one in Figure 2.

Figure 2. Firmware alert for non-admins

If it runs under an admin user's login, the user should see a slightly different alert, as shown in Figure 3.

Figure 3. Firmware alert for admin users.

If the admin user clicks Install, Software Update will be launched and the admin user can install the firmware update immediately. Non-firmware updates should be hidden, but can be shown again by selecting Reset Ignored Updates from the Software Update menu.

If you don't want to retype this script (and who would?) you can find a copy here:

http://managingosx.wordpress.com/2009/01/06/firmware-updates/

To get the script to run at login for every user, we'll use a launchd agent. Make a directory called "Management" in the root Library directory (if you have another directory for this sort of thing, feel free to use that instead):

mkdir /Library/Management

Save the script as "firmwarecheck.pl" in that directory. Make sure it has execute rights:

chmod 755 /Library/Management/firmwarecheck.pl

You can check if you did things correctly by trying to run it:

/Library/Management/firmwarecheck.pl

Now we can create a launchd item to run it at login for every user. Create a plain text file at /Library/LaunchAgents/com.myorg.firmwarecheck.plist with this content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
      <key>Label</key>
      <string>com.myorg.firmwarecheck</string>
      <key>LimitLoadToSessionType</key>
      <string>Aqua</string>
      <key>Program</key>
      <string>/Library/Management/firmwarecheck.pl</string>         
      <key>RunAtLoad</key>
      <true/>
</dict>
</plist>

Some of these launchd keys are new in Leopard, but this launchd item will also work in Tiger. Check to make sure your launchd plist is a valid plist file:

cd /Library/LaunchAgents
plutil -lint com.myorg.firmwarecheck.plist
com.myorg.firmwarecheck.plist: OK

If you don't get an "OK" message, double-check your typing. Ensure your new launchd item is owned by root:wheel and has mode 644:

cd /Library/LaunchAgents
sudo chown root:wheel com.myorg.firmwarecheck.plist
sudo chmod 644 com.myorg.firmwarecheck.plist

If you made no mistakes, /Library/Management/firmwarecheck.pl will now be launched whenever someone logs in. All the pieces are in place! Package up the two files and push them out using your favorite software distribution mechanism.

In this month's column, we dealt with the pain of firmware updates in an enterprise environment by pushing the pain throughout the organization. A script that runs at login checks to see if firmware updates are available for the machine. If firmware updates are found, the user is notified. Even if the user chooses to ignore these notifications, the next time an admin user logs into the machine they will also be notified. If you are lucky, you'll have users who will call the help desk, and then you can use Apple Remote Desktop, Leopard Screen Sharing, or another remote access solution to kick off the firmware update, and tell the user to do the "human-needed" part of holding down the power button for those updates that require it. If you're not so lucky, at least you'll know when machines need updates, and the next time you visit the machine and log in as an administrator, you'll be reminded to apply the updates.

Even if this solution doesn't make sense for you or your organization, the techniques presented here could be used to craft other solutions - for this problem and others. As an OS X system administrator, you'll want to be familiar with as many tools as possible. This column touched on softwareupdate, Perl scripting, AppleScript, and launchd, each important tools in the Mac sysadmin's toolbox!

Greg Neagle is a member of the steering committee of the Mac OS X Enterprise Project (macenterprise.org) and is a senior systems engineer at a large animation studio. Greg has been working with the Mac since 1984, and with OS X since its release. He can be reached at gregneagle@mac.com.