TweetFollow Us on Twitter

MacEnterprise: Sparseimage Conversion

Volume Number: 26
Issue Number: 04
Column Tag: MacEnterprise

MacEnterprise: Sparseimage Conversion

Scripting the conversion of a Filevault sparseimage to a sparsebundle

By Greg Neagle, MacEnterprise.org

Introduction

Back in 2008, I wrote a series of MacEnterprise columns on managing FileVault in an enterprise environment. We covered preparation steps, including setting up an enterprise-wide FileVault Master Password, using MCX to automate the creation of FileVault-protected home directories for new mobile and local accounts, and how to recover from lost user passwords. In the last of three columns, I discussed how to move a FileVault-protected account and home directory from one machine to another.

These past columns can be found in the MacTech archives:

http://www.mactech.com/articles/mactech/Vol.24/24.07/2407MacEnterprise-FileVaultintheEnterprisePart1/index.html

http://www.mactech.com/articles/mactech/Vol.24/24.09/2409MacEnterprise-FileVaultintheEnterprisePart2/index.html

http://www.mactech.com/articles/mactech/Vol.24/24.10/2410MacEnterprise-MigratingFileVault/index.html

This month, we're going to look at another aspect of managing FileVault-encrypted home directories. The format of such directories in Tiger was an encrypted sparsebundle disk image. In this type of image, the entire contents of the disk image are stored in a single file. If the file became corrupted, the entire disk image could be lost. This is why I recommend that you do not implement FileVault unless you either have a backup strategy or are using portable home directories—where the local home directory is kept in sync with a network copy. But if you are backing up the disk image, any change requires backing up the entire disk image file. Another downside of the sparseimage format is as items are changed in the mounted disk image, the disk image file grows in size, even if you delete items within the image. To shrink the disk image file, on logout users are asked if they want to reclaim unused space. This process can take a long time if a lot of changes have been made since the last recovery operation, as the entire disk image file may need to be rewritten.

To address some of these issues, Apple changed the FileVault disk image format in Leopard. Starting with OS X 10.5, newly created FileVault home directories use the "sparse bundle" format. With this format, instead of storing the disk image contents in a single file, the contents are divided among files inside a directory. Think of the enclosing directory as a physical hard drive, and the files within (called "bands") correspond to sectors on a physical disk. This solves several problems. First, the corruption/failure of a single file within the sparsebundle won't render the entire disk image useless; as with a physical disk, only the data in a specific part of the virtual disk is affected. Backups can be faster since only the changed "band" files need to be backed up, instead of the entire image. Finally, recovering unused space in the image is much, much faster, as the OS can just remove the unused band files.

FileVault migration issues

So a big problem with my technique for migrating FileVault-protected accounts from one machine to another is that if you migrate a FileVault-protected home directory from a Tiger machine to a Leopard (or Snow Leopard) machine, the disk image says in the old sparseimage format. This is not a showstopper, as Leopard and Snow Leopard will still happily use sparseimage FileVault home directories. As we moved users from Tiger to Leopard, we saw this issue, but we thought we'd just go back and manually convert the images to the new format. Apple's recommended technique for the conversion? Turn FileVault off, wait for the image to decrypt, then turn FileVault back on and re-encrypt the image. This can take a long time.

A few years passed, and we have started moving users to Snow Leopard. To our chagrin, we're encountering a not-insignificant number of users that still have sparseimage FileVault homes! We could just take the same approach as before, but I knew that if after two years on Leopard we haven't manually converted most of the sparseimages, we're not going to do it now.

So I decided to script the conversion, with the initial goal of streamlining and standardizing the conversion process, and the eventual goal of automating it entirely. I think I've accomplished the first goal, and I will share my solution with you.

Language choices

If you are going to script a system administration task, the first choice you need to make is what language to use as the scripting language. Normally I am a big fan of Python, but I decided against it this time around. Why? It's really overkill for the simple series of operations needed. I developed the procedures I needed by using the shell, so a simple shell script seemed to fit the bill. It was extremely convenient to test a little bit by just typing stuff into the shell and making sure it worked as expected, then adding that to the shell script. So even if the specific task at hand is not relevant to you, you may learn something about shell scripting by following along.

The script, bit by bit

The script is very linear - there are no functions or subroutines or even any loops! It just starts at the beginning and works its way to the end, bailing at various spots if it runs into an error. This makes it easy to present the script in bite-sized chunks. I'll talk about what each bit does, then move to the next. Here goes!

#!/bin/bash
# convertsparseimage
#
# Copyright 2010 by Greg Neagle
# converts FileVault sparseimages to sparsebundles
# free to use, but no suitability for any purpose implied.
# use at your own risk! data loss is a possibility.
WHOAMI=`/usr/bin/id -u`
if [ "$WHOAMI" -ne 0 ]; then
    echo "You must run this as root!"
    exit -1
fi

We start with the familiar "she-bang" line: #!/bin/bash. This tells the OS to use the bash interpreter for this script. Then follow a few comments absolving me of any responsibility if this script causes you or your users to lose data. I don't think it will, but it never hurts to be careful. Test this script carefully before using it on actual user data!

First we check to make sure the effective user ID is that of the root user. In order to convert a user's data we need to run with root privileges. Running this script with 'sudo' is fine. If the current user doesn't have root privileges, we print an error and exit with an error code.

USER=$1
if [ "$USER" == "" ]; then
    echo "Need to specify a user to convert:"
    echo "$0 shortusername"
    exit -1
fi

When we run this script, we need to pass the short name of a user to convert, like so:

/path/to/convertsparseimage gneagle

The username is made available to the script in the $1 variable, and we copy it to a new variable named "$USER". If $USER is empty, we print an error and exit with an error code.

USERLOGGEDIN=`/usr/bin/who | 
              /usr/bin/grep console | 
              /usr/bin/grep $USER`
if [ "$USERLOGGEDIN" != "" ]; then
    echo "$USER is currently logged in!"
    exit -1
fi

Next, we check to see if the user we want to convert is logged in via to GUI. This is done by piping the output of /usr/bin/who to /usr/bin/grep, which will output only lines containing "console", then piping these lines to grep again, this time looking only for lines with our user's name. If $USER is logged in via the console, $USERLOGGEDIN will have something in it; otherwise it will be empty. If it's not empty, we print an error message and exit with an error code.

Note that this simple code could return a false positive. If the user you are looking for has a shortname of "sjohn" and "sjohnson" is logged in, grep will return a match. You can do more sophisticated matches with grep to avoid this issue.

SPARSEIMAGE="/Users/$USER/$USER.sparseimage"
if [ ! -e "$SPARSEIMAGE" ]; then
    echo "Sparseimage $SPARSEIMAGE does not exist!"
    exit -1
fi

Now we check to make sure there actually is a sparseimage to convert. Again, this code is a little simplistic, because the user's home directory and sparseimage name do not technically have to match the user shortname. In 99% or more of cases, though, they do, so this code will work. If you have users with home directories in places other than "/Users" or with alternate names, you'll have to get more clever; perhaps by reading the output of

 `dscl . read /Users/$USER HomeDirectory`.
SPARSEBUNDLE="/Users/$USER/$USER.sparsebundle"
if [ -e "$SPARSEBUNDLE" ]; then
    echo "Sparsebundle $SPARSEBUNDLE already exists!"
    exit -1
fi

This time we check to make sure a sparsebundle doesn't exist. We probably don't want to write over an existing sparsebundle in the user's home directory path.

# get the size of the sparseimage in megabytes
IMAGESIZE=`/usr/bin/du -m "$SPARSEIMAGE" |
           /usr/bin/cut -f1`
# add some breathing room; 100 MB
SPACENEEDED=$((IMAGESIZE+100))
# get the free space on the disk in megabytes
DISKSPACE=`/bin/df -m / | /usr/bin/awk 'NR>1 { print $4 }'`
if [ $DISKSPACE -lt $SPACENEEDED ]; then
    echo "Not enough free space on disk:" 
    echo "$DISKSPACE MB available; want $SPACENEEDED MB."
    exit -1
fi
echo "Need $SPACENEEDED MB to convert; found $DISKSPACE MB"

For our next subtask, we figure out the size of the sparseimage in megabytes and add another 100MB to guess at how much space we'll need to do the conversion. We then figure out the amount of free space on the startup disk, also in megabytes. If the free space on the disk is less than the space we need to convert the image, we print an error message and exit with an error code.

This bit makes use of the du (disk usage) and df (disk freespace) commands. To parse the output of these commands, we use cut and awk. awk is somewhat arcane - in fact, it is another language all to itself, but very useful when it comes to parsing text output from another shell command.

echo "Unlocking FileVaultMaster keychain..."
FVMASTER="/Library/Keychains/FileVaultMaster.keychain"
# /usr/bin/security will prompt the for the password
/usr/bin/security unlock-keychain "$FVMASTER"
if [ $? -ne 0 ]; then
    echo "Error unlocking FileVaultMaster keychain"
    exit -2
fi

All the preliminary checks are done. We can finally start with the actual task. Since this script is eventually destined for automation, we can't count on having the user's password. Fortunately, we've created all our FileVault-protected home directories along with a FileVault Master Password, and so we can use the FileVaultMaster keychain to decrypt the sparseimage. (If, in the name of security, you've moved (or removed!) the FileVaultMater keychain, you'll need to modify the script to use the new location or retrieve the keychain from wherever you've stashed it. If you aren't using a FileVault Master password, then this script would have to be modified to require and use the user's login password.

We're using the security command to unlock the FileVaultMaster keychain. It will prompt for a password for this keychain. That's a pretty significant barrier to completely automating this process. The security command does take a -p parameter, which allows you to pass the keychain password, but including the password in a script like this immediately compromises the security of all the FileVault-protected accounts in your organization, so I do not recommend it. For now, we're just going to have to type in the password.

echo "Mounting sparseimage $SPARSEIMAGE..."
/usr/bin/hdiutil attach "$SPARSEIMAGE" -recover "$FVMASTER" -nobrowse -mountpoint /tmp/fvimage
if [ $? -ne 0 ]; then
    echo "Error mounting FileVault-protected sparseimage"
    exit -3
fi

(In the above code block, the /usr/bin/hdiutil line should all be one single line through "/tmp/fvimage")

Now we use the FileVaultMaster keychain to mount the encrypted disk image even though we don't have the user's password. We also specify the -nobrowse option so it does not appear in the Finder (which would be bad if another user was logged in). We also specify a custom mountpoint in /tmp, so the volume is not accessible via the normal /Volumes mountpoints. Both of these options improve security somewhat. If we get an error while trying to mount the image, we exit with an error.

echo "Creating encrypted sparsebundle with contents from"
echo "sparseimage..."
PASSWORD="LetMeIn!\\0000"
echo -e "$PASSWORD" | /usr/bin/hdiutil create -srcfolder /tmp/fvimage -format UDSB -o "$SPARSEBUNDLE" -volname $USER -encryption -stdinpass -certificate /Library/Keychains/FileVaultMaster.cer
if [ $? -ne 0 ]; then
    echo "Error creating sparsebundle"
    rm -r $SPARSEBUNDLE
    echo "Unmounting sparseimage"
    /usr/bin/hdiutil eject /tmp/fvimage
    exit -4
fi

(In the above code block, the section starting with 'echo -e "$PASSWORD" ' and ending with '/Library/Keychains/FileVaultMaster.cer ' is all one long line.)

The actual image conversion happens here at long last. Since we don't have the user's password, we have to create this sparsebundle with a default password, which we'll need to provide to the user to use on the first login. Change this password as you wish, but leave the "\\0000" at the end - this is a null-character, which is needed by hdiutil at the end of a password passed in via STDIN.

Let's tear apart the rather complicated line with hdiutil:

echo -e "$PASSWORD" | 

- send the password via STDIN to hdiutil

/usr/bin/hdiutil create -srcfolder /tmp/fvimage 

- create a new disk image from the contents of the folder /tmp/fvimage, which is where we've mounted the sparseimage.

-format UDSB 

- the new disk image should be formatted as a sparsebundle.

-o "$SPARSEBUNDLE" 

- write the new disk image to the path described by $SPARSEBUNDLE, which we've previously set to "/Users/$USER/$USER.sparesbundle".

-volname $USER 

- make sure the volume has the same name as the user

-encryption 

- use the standard encryption

-stdinpass 

- read the password from STDIN.

-certificate /Library/Keychains/FileVaultMaster.cer

- use the certificate given as the alternate unlock key. This allows us to decrypt an image with the FileVaultMaster keychain.

hdiutil create may take a while to do its job. The larger the original sparseimage, the longer creating the sparsebundle will take. It does print some progress info as it works. If an error is encountered, we delete the sparesbundle, eject the sparseimage, and exit with an error message.

echo "Unmounting sparseimage $SPARSEIMAGE"

/usr/bin/hdiutil eject /tmp/fvimage
echo "Moving $SPARSEIMAGE to /Users/Shared/..."
/bin/mv "$SPARSEIMAGE" "/Users/Shared/$USER.sparseimage"
echo "Updating DirectoryService HomeDirectory to point to" 
echo "new sparsebundle"
/usr/bin/dscl . -create /Users/$USER HomeDirectory "<home_dir><url>file://localhost$SPARSEBUNDLE</url></home_dir>"

Finally, some cleanup and housekeeping: We eject the sparseimage and move it to /Users/Shared for safekeeping—I'm still not confident enough to delete it outright. After a day or so, assuming the converted sparsebundle works as expected, you could then go back and delete the old sparseimage.

Lastly, we update the HomeDirectory attribute in the local directory service to point to the sparsebundle instead of the old sparseimage. Note that the section starting with "/usr/bin/dscl" and ending with "</home_dir>" should all be a single line. This of course assumes the account information is local, but for mobile account or local users, the account info will be in the local directory service.

Now what?

So you've implemented and tested this script on some test accounts and you want to try it on a user's data. It's important to remember that the new sparsebundle does not have the user's login password, since we don't have that password anywhere in the script. Instead, we've created the disk image with a generic password that we'll provide to the user.

The first time the user tries to login after the conversion, they will see a dialog like the one in Figure 1.


Figure 1 - FileVault password warning

What is happening here is that the user has successfully authenticated against the directory service, but the current password isn't working to unlock the encrypted disk image. Since this is typically caused by network users changing their password, Mac OS X helpfully suggests that the user try their previous password to unlock the FileVault image.

In this case, you don't want the user to try their previous password. Instead, you want them to use the generic password that was used to encrypt the sparsebundle during the conversion. Once they provide that password, the sparsebundle will be unlocked and its password updated to the current user password.

Conclusion

In this month's MacEnterprise column, we stepped through a BASH script that converts FileVault sparseimages into FileVault sparsebundles for better performance and compatibility with Leopard and Snow Leopard. If you haven't done much shell scripting, I hope the examples in this script inspire you to learn more. Good luck!


Greg Neagle is a member of the steering committee of the Mac OS X Enterprise Project (macenterprise.org) and is a senior systems engineer at a large animation studio. Greg has been working with the Mac since 1984, and with OS X since its release. He can be reached at gregneagle@mac.com.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Vienna 3.1.9 :e81515b: - RSS and Atom ne...
Vienna is a freeware and Open-Source RSS/Atom newsreader with article storage and management via a SQLite database, written in Objective-C and Cocoa, for the OS X operating system. It provides... Read more
iExplorer 4.0.12.0 - View and transfer f...
iExplorer is an iPhone browser for Mac lets you view the files on your iOS device. By using a drag and drop interface, you can quickly copy files and folders between your Mac and your iPhone or... Read more
OpenEmu 2.0.5 - Open Source game-emulati...
OpenEmu is about to change the world of video game emulation, one console at a time... For the first time, the 'It just works' philosophy now extends to open source video game emulation on the Mac.... Read more
TextSoap 8.3.3 - Automate tedious text d...
TextSoap can automatically remove unwanted characters, fix up messed up carriage returns, and do pretty much anything else that we can think of to text. Save time and effort. Be more productive. Stop... Read more
Apple iTunes 12.6 - Play Apple Music and...
Apple iTunes lets you organize and stream Apple Music, download and watch video and listen to Podcasts. It can automatically download new music, app, and book purchases across all your devices and... Read more
Airmail 3.2.4 - Powerful, minimal email...
Airmail is an mail client with fast performance and intuitive interaction. Support for iCloud, MS Exchange, Gmail, Google Apps, IMAP, POP3, Yahoo!, AOL, Outlook.com, Live.com. Airmail was designed... Read more
MacPilot 9.0.6 - Enable over 1,200 hidde...
MacPilot gives you the power of UNIX and the simplicity of Macintosh, which means a phenomenal amount of untapped power in your hands! Use MacPilot to unlock over 1,200 features, and access them all... Read more
Jamf Pro 9.98 - Powerful sysadmin/enterp...
Jamf Pro (formerly Casper Suite) is the EMM tool that delights IT pros and the users they support by delivering on the promise of unified endpoint management for Apple devices. At Jamf, connecting... Read more
PopChar 7.7 - $16.99 (51% off)
PopChar helps you get the most out of your font collection. With its crystal-clear interface, PopChar provides a frustration-free way to access any font's special characters. Features Expanded... Read more
RapidWeaver 7.3.1 - Create template-base...
RapidWeaver is a next-generation Web design application to help you easily create professional-looking Web sites in minutes. No knowledge of complex code is required, RapidWeaver will take care of... Read more

Failbetter Games details changes coming...
Sunless Sea, Failbetter Games' dark and gloomy sea explorer, sets sail for the iPad tomorrow. Ahead of the game's launch, Failbetter took to Twitter to discuss what will be different in the mobile version of the game. Many of the changes make... | Read more »
Splish, splash! The Pokémon GO Water Fes...
Niantic is back with a new festival for dedicated Pokémon GO collectors. The Water Festival officially kicks off today at 1 P.M. PDT and runs through March 29. Magikarp, Squirtle, Totodile, and their assorted evolved forms will be appearing at... | Read more »
Death Road to Canada (Games)
Death Road to Canada 1.0 Device: iOS Universal Category: Games Price: $7.99, Version: 1.0 (iTunes) Description: Get it now at the low launch price! Price will go up a dollar every major update. Update news at the bottom of this... | Read more »
Bean's Quest Beginner's Guide:...
Bean's Quest is a new take on both the classic platformer and the endless runner, and it's free on the App Store for the time being. Instead of running constantly, you can't stop jumping. That adds a surprising new level of challenge to the game... | Read more »
How to rake in the cash in Bit City
Our last Bit City guide covered the basics. Now it's time to get into some of the more advanced techniques. In the later cities, cash flow becomes much more difficult, so you'll want to develop some strategies if you want to complete each level.... | Read more »
PixelTerra (Games)
PixelTerra 1.1.1 Device: iOS Universal Category: Games Price: $.99, Version: 1.1.1 (iTunes) Description: The world of PixelTerra is quite dangerous so you need to build a shelter, find some food supply and get ready to protect... | Read more »
Tokaido™ (Games)
Tokaido™ 1.0 Device: iOS Universal Category: Games Price: $6.99, Version: 1.0 (iTunes) Description: Discover the digital adaptation of Tokaido, the boardgame phenomenon that has already sold more than 250,000 copies worldwide, and... | Read more »
Card Thief (Games)
Card Thief 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Card Thief is a solitaire style stealth game played with a deck of cards. In Card Thief you move through a deck of cards as a... | Read more »
Smilegate’s crafting battler Super Tank...
Super Tank Rumbleputs you in the seat of your very own, handcrafted tank. You can choose from over 100 different parts to create your Super Tank before taking it out to wreak havoc on your opponents in glorious PVP combat. Now, Smilegate is upping... | Read more »
The best games to play while you wait fo...
Mass Effect: Andromeda arrives this coming Tuesday and people are getting understandably antsy. There's a whole weekend standing between us and a brand new Bioware adventure. What's a geek to do? Well, you could certainly give these fine mobile... | Read more »

Price Scanner via MacPrices.net

13-inch Touch Bar MacBook Pros on sale for up...
B&H Photo has the Apple 13″ Touch Bar MacBook Pros in stock today and on sale for up to $150 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 13″ 2.9GHz/512GB Touch Bar... Read more
Today only! 15-inch 2.7GHz Space Gray Touch B...
B&H Photo has the new 2016 15″ 2.7GHz Space Gray Apple Touch Bar MacBook Pro in stock today and on sale for $300 off MSRP for today only. Shipping is free, and B&H charges NY sales tax only... Read more
New $329 iPad A Fabulous Value; 10.5-Inch iPa...
Part of the iPad upgrade/new model puzzle is now in place. Yesterday, as KGI Securities financial services group analyst Ming-Chi Kuo last summer predicted they would, Apple released a new low-cost 9... Read more
New 9.7-Inch iPad Features All Of The Fun...
Apple today updated its most popular-sized iPad, featuring a brighter 9.7-inch Retina display and best-in-class performance at its most affordable price ever, starting at $329 (US) with 32GB of... Read more
Apple Introduces iPhone 7 and iPhone 7 Plus (...
Apple today announced iPhone 7 and iPhone 7 Plus (PRODUCT)RED Special Edition in a vibrant red matte aluminum finish, in recognition of more than 10 years of partnership between Apple and (RED). This... Read more
Apple now offering Certified Refurbished 15-i...
Apple is now offering Certified Refurbished 2016 15″ Touch Bar MacBook Pros for $360-$420 off original MSRP. An Apple one-year warranty is included with each model, and shipping is free: - 15″ 2.6GHz... Read more
Apple Introduces Clips: A Free Innovative Way...
Apple today introduced Clips, a new app that makes it quick and fun for anyone to create expressive videos on iPhone and iPad. The app features a unique design for combining video clips, photos and... Read more
Urban Armor Gear Unveils Case For 4th Generat...
Orange County, California based Urban Armor Gear (UAG), designers of rugged, lightweight protective cases for phones, tablets and laptops, has released its latest drop-tested cases for Apple’s 4th... Read more
Most Users Continue To Prefer 5.0 to 5.3 Inch...
In the first half of 2016, US and UK smartphone users were most likely to be interested in purchasing a device with a 5.3″ or 5.0″ display. Findings in a new report from the User Experience... Read more
12-inch 1.2GHz Retina MacBooks on sale for up...
B&H has 12″ 1.2GHz Retina MacBooks on sale for up to $200 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.2GHz Space Gray Retina MacBook: $1439.99 $160 off MSRP - 12″ 1... Read more

Jobs Board

*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Fulltime aan de slag als shopmanager in een h...
Ben jij helemaal gek van Apple -producten en vind je het helemaal super om fulltime shopmanager te zijn in een jonge en hippe elektronicazaak? Wil jij werken in Read more
Starte Dein Karriere-Abenteuer in den Hauptst...
…mehrsprachigen Teams betreust Du Kunden von bekannten globale Marken wie Apple , Mercedes, Facebook, Expedia, und vielen anderen! Funktion Du wolltest schon Read more
Starte Dein Karriere-Abenteuer in den Hauptst...
…mehrsprachigen Teams betreust Du Kunden von bekannten globale Marken wie Apple , Mercedes, Facebook, Expedia, und vielen anderen! Funktion Du wolltest schon Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.