TweetFollow Us on Twitter

Do You Copy?

Volume Number: 22 (2006)
Issue Number: 8
Column Tag: Field Notes

Do You Copy?

Are you copying all the information about your files that you need?

by Dan Shoop

Introduction

The Macintosh environment has always been feature rich, and that extends to its files, file attributes, and file metadata the file system maintains for them. OS X has introduced Macintosh to a wider array of application environments which, aside from that of Classic, include beasts such as Carbon, Cocoa, BSD, X-Windows, Web Objects, Java, QuickTime, and now Rosetta. Additionally, OS X supports different file systems that support different metadata. All of these may see and consider files differently than others. This results in files being copied rather differently.

This article is a short attempt to attune application developers, scripters, systems programmers, and file system architects to considerations relating to file metadata stored by the file systems on Mac OS X.

Philosophic Questions

In many respects it is a philosophic question as to what each environment on the Mac should properly copy. Some environments may not normally or naturally have access to all file metadata that OS X maintains for any given file. Named Extended Attributes may not be accessible through Carbon, Cocoa and Core Foundations, while they may be accessible from BSD. The nature of the operation may color and determine what metadata is important and what makes sense for preserving as a copy or whether the metadata should be reflective of a whole new file.

Consider the subtle differences of what metadata, or even files, get maintained as part of the different functions of operations like copies vs archives vs backups vs clones vs sync vs snapshots, et cetera. Is a copied file a new file or is the copy merely a replica or clone? This isn't as easy of a question as it first seems. For instance, if you copy a file from one location to another, does it make sense to maintain any ACLs it may have had or shouldn't the new file have its own, new ACL? But now, what if that "copy" was a backup or clone and part of a larger set of files in its directory? Wouldn't you then expect the ACLs to be maintained? Likewise, if you're backing up a set of files would you expect metadata associated with the directory (like sort order, icon positions, etc) to be maintained for the files? Yet, these are stored in places like .DS_Store or the Desktop Database, not as metadata for the file itself.

What about when we copy files to non-HFS+ file systems? What should be maintained? What should be expected if a file is copied off, then gets copied back? The Macintosh has defined the Apple Double format to store a file non-natively. Is it right for a Windows sysadmin to delete ._file's or .DS_Store's found on their file systems; is this "clutter"? To the Mac user who's lost the CODE resource of an application file, icons or font resources, a file's creation date, or creator/type you may end up with torches and pitchforks at your door if you're inconsiderate, and confusing if your tools are inconsistent. But should an ACL get copied to a foreign file system? Maybe not, but other Extended Attributes clearly are in Apple Doubles.

Moreover, consider what file creation, copying, and modification may mean. For instance consider that cat > file is often seen as creating a new file, but that if 'file' already existed then this operation is in fact a "modification".

Lastly, in the future we may be limited as to what we may and may not copy through DRM. This may become a technical rather than semantic issue as "secure" file systems and files are deployed (as in secure for the owners of the content).

Mac OS X File Metadata and Attributes

Briefly, let's run down some of the most significant file metadata and attributes common to Mac OS X.

Data Streams

The data fork is well known, but all Macintosh files have at least two forks, a data fork and a resource fork, either of which may be of zero length. Implied is that both exist. Under HFS+ multiple named streams may exist, which may be named, but the data and resource streams may not be renamed. Until Tiger, this was merely semantics.

Traditional views of files are just a single byte stream of named data (like found on a tape) with beginning-of-file (BOFs) and end-of-file (EOFs), and little to no metadata. The metadata was just "accounting" information used by the file system, it wasn't really "the data". However, over the years additional information was tacked on by file systems. Users got used to this additional metadata for cataloging and sysadmins got used to preserving them on backups. Environments became more complex and file systems started offering better ways to store ancillary data. The Macintosh isn't unique in offering multiple streams per file. Even Windows' NTFS has such provisions. HFS+ defined an architecture for "unlimited" numbers of streams or forks using the Attributes B-tree structure on the volume.

Under OS X the mandatory data and resource forks of a Macintosh file are exposed on HFS+ volumes as filename and filename/..namedfork/rsrc to BSD environments and as filename and ._filename when "split" (as for use on foreign file systems.) It also introduced tools for splitting and recombining forks split into what are arguably Apple Doubles.

Mac OS X 10.4 Tiger introduced the concept of Extended Attributes which implemented and exposed the additional named forks or streams on HFS+ volumes and through the 'split' fork ._filename on foreign file systems. ACLs are in turn implemented as a privately named and protected Extended Attribute and hence are a third fork that files may have on file systems that have ACLs enabled. Additionally any number of additional "Extended Attributes" (forks) may now be added to or read from a file with setxattr(), getxattr(), listxattr(), and removexattr().

Finder Flags

See `man GetFileInfo` for details, but these include file creator and type, attributes bits (such as locked, stationary, invisible), creation date and modification date. Stored in the ._filename Apple Double half on file systems other than HFS and its derivatives.

POSIX

Files have ownerships (user and group) and permissions. Symbolic links also have these attributes. Unix files also traditionally carry dates such as atime, mtime, and ctime, namely, the time of last access, last time file modified, and change time (last time the file's inode was changed), respectively. Note that ctime changes after operations like chmod.

Creation Dates

The subject of creation dates on the Mac warrants detailed discussion in itself. The Mac has always maintained file "creation date" metadata, it's a Finder file attribute, copied along with the file by the finder, displayed in the Finder's Get Info dialog and important enough that it is supposed to be included as part of the Apple Double file information stored in a ._file. But, while Mac users are long familiar with it and often use it to categorize files it's a rather foreign concept to unix (mostly because it traditionally never existed there), and some unix pundits believe it's just woolly-thinking (see http://mail-index.netbsd.org/netbsd-users/2000/11/22/0000.html) or at best the product of the confused (see http://toadstool.se/journal/2006/01/11/the-fallacy-of-ctime). Other unices and file systems support a btime (birth time) with varying degrees of success.

However, Mac users are used to creation dates, and may get rather agitated if they're lost or, say replaced with the modification time (which seems to be the OS X BSD environment behavior). As such, applications should strongly consider properly handling this metadata or, if they do mangle it, do so in a manner that lends itself to notice (like beginning of epoch).

For a more detailed commentary of Mac file creation dates see http://blog.plasticsfuture.org/2006/06/27/mac-os-file-creation-dates/

BSD Flags

BSD has traditionally maintained a series of flags associated with properties of a file as manipulated by `chflags`. These include flags such as the 'system immutable flag' (schg) that prohibit a file from being modified, or the 'archive flag' (arch).

The classical Desktop Database and OS X .DS_Store files

Traditionally the Macintosh stored certain metadata regarding files in the Desktop Database. Under Mac OS X the .DS_Store file associated with a directory contains similar information such as directory background info, the position or order of files and their icons within a directory, and more. When copying a directory you should probably consider copying these.

Lastly there's one Finder-level piece of information stored, disassociated from the file, which brings us to...

Spotlight Metadata

Tiger introduced us to Spotlight and a huge amount of application specific metadata stored that is associated with a file based on its content. This data is indexed by the kernel through 'importers' as a part of normal operations on the file.

While most of this doesn't need to be preserved since it is indexed as needed by the kernel, one attribute exposed to the Finder and stored in .DS_Store is what traditionally was known as the file Comments, called under Tiger the Spotlight Comments, and is displayed through the Finder's Get Info dialog.

How to preserve file metadata

There a few primary methods by which file metadata get preserved:

  • copied natively
  • preserved in special containers or lists
  • preserved by images of the volumes that support them (e.g. .dmg's)
  • split as Apple Doubles

BSD copies files using copyfile() which splits to Apple Doubles as necessary either internally or externally (with varying results)

Note that, currently, copyfile() munges the modified date into the creation date field of its Apple Doubles.

What needs to be copied? What doesn't?

While mostly a philosophic issue, it's probably safe to say that traditional Macintosh attributes such as creator and type are becoming less important since OS X has various mechanisms for associating files with their applications (sometimes at the cost of additional metadata) while other metadata, like creation date, should arguably be maintained. Spotlight metadata looks like a good candidate for general exclusion since the kernel will automagically recreate this for installed application importers on the target machine, but all Mac OS X systems might not have the same importers installed. This may or may not matter. Old Desktop Database files probably aren't an issue either as Classic withers or is mooted by Intel Macs, but .DS_Stores should probably be maintained if you're backing up a volume. Mandatory file forks should always be copied, but while these are Extended Attributes you might consider not copying the other forks that may exist, such as ACLs, depending on if you're backing up or cloning, or, if you're just duplicating the file elsewhere on a volume. It's hard to tell currently if other Extended Attributes need copying since they're rarely seen in action.

What Copies What?

Let's take a look at what metadata, some common tools and their operations preserve. The following is not designed to be a "report card" and it's not my goal to rank or rate a tool for its handling of the data, but sysadmins should be aware of how operations they perform may affect their data.

Coverage of various third-party backup applications is beyond the scope of this article, but it seems they fare rather poorly with regard to not preserving Extended Attributes aside from resource forks. Creation dates are a mixed bag and may vary within the application based on operation (sometimes intentionally). For additional discussion, see http://blog.plasticsfutures.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/

In general, copyfile(), which most all BSD tools rely on, does not perform consistently compared to the operation of the Finder. Creation date is clobbered by the modification date despite that copyfile() attempts to use Apple Doubles 'internally' and externally to foreign file systems and the Apple Double format specifically calls for the creation date. The various tools that use copyfile() then quite often fail to properly maintain internal or synthetic files resulting in lost Extended Attributes and other data. In rare cases the tool may crash, as in the instance of trying to `rsync -aE` a file with both ACLs and an additional non-resource fork, named Extended Attributes.

Apple System Restore (asr) in device mode manages to copy all the tested data, but this is to be expected since it is essentially a complete copy of the device. In file mode, however, asr, Disk Utility, and hdiutil have digressed in their behavior with OS X 10.4.6. Where previously it maintained locks (the Finder "L" attribute and BSD uchg flags), Extended Attributes and ACLs; it no longer preserves them, though this may change in a future update or (although this makes packages and dmgs problematic) it may be the intended behavior.

`dd` fares pretty much as you'd expect. At least philosophically it's understandable.

Editor's Note: To download a copy of the folloing table for reference please visit the MacTech source code ftp and select this month's issue, 22.08 :

http://www.mactech.com/editorial/filearchives.html



Figure 1. The changes in application code structure


Dan Shoop is principal researcher at iWiring (www.iwiring.net) and a Systems and Networks Architect for US Technical Services (ustsvs.com), both of which provide service and support for the Macintosh and other computer systems. He may be reached at <shoop@iwiring.net>.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Affinity Designer 1.5.5 - Vector graphic...
Affinity Designer is an incredibly accurate vector illustrator that feels fast and at home in the hands of creative professionals. It intuitively combines rock solid and crisp vector art with... Read more
djay Pro 1.4.3 - Transform your Mac into...
djay Pro provides a complete toolkit for performing DJs. Its unique modern interface is built around a sophisticated integration with iTunes and Spotify, giving you instant access to millions of... Read more
Jamf Pro 9.98 - Powerful sysadmin/enterp...
Jamf Pro (formerly Casper Suite) is the EMM tool that delights IT pros and the users they support by delivering on the promise of unified endpoint management for Apple devices. At Jamf, connecting... Read more
Airmail 3.2.4 - Powerful, minimal email...
Airmail is an mail client with fast performance and intuitive interaction. Support for iCloud, MS Exchange, Gmail, Google Apps, IMAP, POP3, Yahoo!, AOL, Outlook.com, Live.com. Airmail was designed... Read more
RapidWeaver 7.3.1 - Create template-base...
RapidWeaver is a next-generation Web design application to help you easily create professional-looking Web sites in minutes. No knowledge of complex code is required, RapidWeaver will take care of... Read more
PopChar 7.7 - $16.99 (51% off)
PopChar helps you get the most out of your font collection. With its crystal-clear interface, PopChar provides a frustration-free way to access any font's special characters. Features Expanded... Read more
Apple iTunes 12.6 - Play Apple Music and...
Apple iTunes lets you organize and stream Apple Music, download and watch video and listen to Podcasts. It can automatically download new music, app, and book purchases across all your devices and... Read more
MacPilot 9.0.6 - Enable over 1,200 hidde...
MacPilot gives you the power of UNIX and the simplicity of Macintosh, which means a phenomenal amount of untapped power in your hands! Use MacPilot to unlock over 1,200 features, and access them all... Read more
Sparkle Pro 2.1.4 - $79.99
Sparkle Pro will change your mind if you thought building websites wasn't for you. Sparkle is the intuitive site builder that lets you create sites for your online portfolio, team or band pages, or... Read more
TextSoap 8.3.3 - Automate tedious text d...
TextSoap can automatically remove unwanted characters, fix up messed up carriage returns, and do pretty much anything else that we can think of to text. Save time and effort. Be more productive. Stop... Read more

Failbetter Games details changes coming...
Sunless Sea, Failbetter Games' dark and gloomy sea explorer, sets sail for the iPad tomorrow. Ahead of the game's launch, Failbetter took to Twitter to discuss what will be different in the mobile version of the game. Many of the changes make... | Read more »
Splish, splash! The Pokémon GO Water Fes...
Niantic is back with a new festival for dedicated Pokémon GO collectors. The Water Festival officially kicks off today at 1 P.M. PDT and runs through March 29. Magikarp, Squirtle, Totodile, and their assorted evolved forms will be appearing at... | Read more »
Death Road to Canada (Games)
Death Road to Canada 1.0 Device: iOS Universal Category: Games Price: $7.99, Version: 1.0 (iTunes) Description: Get it now at the low launch price! Price will go up a dollar every major update. Update news at the bottom of this... | Read more »
Bean's Quest Beginner's Guide:...
Bean's Quest is a new take on both the classic platformer and the endless runner, and it's free on the App Store for the time being. Instead of running constantly, you can't stop jumping. That adds a surprising new level of challenge to the game... | Read more »
How to rake in the cash in Bit City
Our last Bit City guide covered the basics. Now it's time to get into some of the more advanced techniques. In the later cities, cash flow becomes much more difficult, so you'll want to develop some strategies if you want to complete each level.... | Read more »
PixelTerra (Games)
PixelTerra 1.1.1 Device: iOS Universal Category: Games Price: $.99, Version: 1.1.1 (iTunes) Description: The world of PixelTerra is quite dangerous so you need to build a shelter, find some food supply and get ready to protect... | Read more »
Tokaido™ (Games)
Tokaido™ 1.0 Device: iOS Universal Category: Games Price: $6.99, Version: 1.0 (iTunes) Description: Discover the digital adaptation of Tokaido, the boardgame phenomenon that has already sold more than 250,000 copies worldwide, and... | Read more »
Card Thief (Games)
Card Thief 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Card Thief is a solitaire style stealth game played with a deck of cards. In Card Thief you move through a deck of cards as a... | Read more »
Smilegate’s crafting battler Super Tank...
Super Tank Rumbleputs you in the seat of your very own, handcrafted tank. You can choose from over 100 different parts to create your Super Tank before taking it out to wreak havoc on your opponents in glorious PVP combat. Now, Smilegate is upping... | Read more »
The best games to play while you wait fo...
Mass Effect: Andromeda arrives this coming Tuesday and people are getting understandably antsy. There's a whole weekend standing between us and a brand new Bioware adventure. What's a geek to do? Well, you could certainly give these fine mobile... | Read more »

Price Scanner via MacPrices.net

13-inch Touch Bar MacBook Pros on sale for up...
B&H Photo has the Apple 13″ Touch Bar MacBook Pros in stock today and on sale for up to $150 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 13″ 2.9GHz/512GB Touch Bar... Read more
Today only! 15-inch 2.7GHz Space Gray Touch B...
B&H Photo has the new 2016 15″ 2.7GHz Space Gray Apple Touch Bar MacBook Pro in stock today and on sale for $300 off MSRP for today only. Shipping is free, and B&H charges NY sales tax only... Read more
New $329 iPad A Fabulous Value; 10.5-Inch iPa...
Part of the iPad upgrade/new model puzzle is now in place. Yesterday, as KGI Securities financial services group analyst Ming-Chi Kuo last summer predicted they would, Apple released a new low-cost 9... Read more
New 9.7-Inch iPad Features All Of The Fun...
Apple today updated its most popular-sized iPad, featuring a brighter 9.7-inch Retina display and best-in-class performance at its most affordable price ever, starting at $329 (US) with 32GB of... Read more
Apple Introduces iPhone 7 and iPhone 7 Plus (...
Apple today announced iPhone 7 and iPhone 7 Plus (PRODUCT)RED Special Edition in a vibrant red matte aluminum finish, in recognition of more than 10 years of partnership between Apple and (RED). This... Read more
Apple now offering Certified Refurbished 15-i...
Apple is now offering Certified Refurbished 2016 15″ Touch Bar MacBook Pros for $360-$420 off original MSRP. An Apple one-year warranty is included with each model, and shipping is free: - 15″ 2.6GHz... Read more
Apple Introduces Clips: A Free Innovative Way...
Apple today introduced Clips, a new app that makes it quick and fun for anyone to create expressive videos on iPhone and iPad. The app features a unique design for combining video clips, photos and... Read more
Urban Armor Gear Unveils Case For 4th Generat...
Orange County, California based Urban Armor Gear (UAG), designers of rugged, lightweight protective cases for phones, tablets and laptops, has released its latest drop-tested cases for Apple’s 4th... Read more
Most Users Continue To Prefer 5.0 to 5.3 Inch...
In the first half of 2016, US and UK smartphone users were most likely to be interested in purchasing a device with a 5.3″ or 5.0″ display. Findings in a new report from the User Experience... Read more
12-inch 1.2GHz Retina MacBooks on sale for up...
B&H has 12″ 1.2GHz Retina MacBooks on sale for up to $200 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.2GHz Space Gray Retina MacBook: $1439.99 $160 off MSRP - 12″ 1... Read more

Jobs Board

*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Fulltime aan de slag als shopmanager in een h...
Ben jij helemaal gek van Apple -producten en vind je het helemaal super om fulltime shopmanager te zijn in een jonge en hippe elektronicazaak? Wil jij werken in Read more
Starte Dein Karriere-Abenteuer in den Hauptst...
…mehrsprachigen Teams betreust Du Kunden von bekannten globale Marken wie Apple , Mercedes, Facebook, Expedia, und vielen anderen! Funktion Du wolltest schon Read more
Starte Dein Karriere-Abenteuer in den Hauptst...
…mehrsprachigen Teams betreust Du Kunden von bekannten globale Marken wie Apple , Mercedes, Facebook, Expedia, und vielen anderen! Funktion Du wolltest schon Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.