TweetFollow Us on Twitter

Do You Copy?

Volume Number: 22 (2006)
Issue Number: 8
Column Tag: Field Notes

Do You Copy?

Are you copying all the information about your files that you need?

by Dan Shoop

Introduction

The Macintosh environment has always been feature rich, and that extends to its files, file attributes, and file metadata the file system maintains for them. OS X has introduced Macintosh to a wider array of application environments which, aside from that of Classic, include beasts such as Carbon, Cocoa, BSD, X-Windows, Web Objects, Java, QuickTime, and now Rosetta. Additionally, OS X supports different file systems that support different metadata. All of these may see and consider files differently than others. This results in files being copied rather differently.

This article is a short attempt to attune application developers, scripters, systems programmers, and file system architects to considerations relating to file metadata stored by the file systems on Mac OS X.

Philosophic Questions

In many respects it is a philosophic question as to what each environment on the Mac should properly copy. Some environments may not normally or naturally have access to all file metadata that OS X maintains for any given file. Named Extended Attributes may not be accessible through Carbon, Cocoa and Core Foundations, while they may be accessible from BSD. The nature of the operation may color and determine what metadata is important and what makes sense for preserving as a copy or whether the metadata should be reflective of a whole new file.

Consider the subtle differences of what metadata, or even files, get maintained as part of the different functions of operations like copies vs archives vs backups vs clones vs sync vs snapshots, et cetera. Is a copied file a new file or is the copy merely a replica or clone? This isn't as easy of a question as it first seems. For instance, if you copy a file from one location to another, does it make sense to maintain any ACLs it may have had or shouldn't the new file have its own, new ACL? But now, what if that "copy" was a backup or clone and part of a larger set of files in its directory? Wouldn't you then expect the ACLs to be maintained? Likewise, if you're backing up a set of files would you expect metadata associated with the directory (like sort order, icon positions, etc) to be maintained for the files? Yet, these are stored in places like .DS_Store or the Desktop Database, not as metadata for the file itself.

What about when we copy files to non-HFS+ file systems? What should be maintained? What should be expected if a file is copied off, then gets copied back? The Macintosh has defined the Apple Double format to store a file non-natively. Is it right for a Windows sysadmin to delete ._file's or .DS_Store's found on their file systems; is this "clutter"? To the Mac user who's lost the CODE resource of an application file, icons or font resources, a file's creation date, or creator/type you may end up with torches and pitchforks at your door if you're inconsiderate, and confusing if your tools are inconsistent. But should an ACL get copied to a foreign file system? Maybe not, but other Extended Attributes clearly are in Apple Doubles.

Moreover, consider what file creation, copying, and modification may mean. For instance consider that cat > file is often seen as creating a new file, but that if 'file' already existed then this operation is in fact a "modification".

Lastly, in the future we may be limited as to what we may and may not copy through DRM. This may become a technical rather than semantic issue as "secure" file systems and files are deployed (as in secure for the owners of the content).

Mac OS X File Metadata and Attributes

Briefly, let's run down some of the most significant file metadata and attributes common to Mac OS X.

Data Streams

The data fork is well known, but all Macintosh files have at least two forks, a data fork and a resource fork, either of which may be of zero length. Implied is that both exist. Under HFS+ multiple named streams may exist, which may be named, but the data and resource streams may not be renamed. Until Tiger, this was merely semantics.

Traditional views of files are just a single byte stream of named data (like found on a tape) with beginning-of-file (BOFs) and end-of-file (EOFs), and little to no metadata. The metadata was just "accounting" information used by the file system, it wasn't really "the data". However, over the years additional information was tacked on by file systems. Users got used to this additional metadata for cataloging and sysadmins got used to preserving them on backups. Environments became more complex and file systems started offering better ways to store ancillary data. The Macintosh isn't unique in offering multiple streams per file. Even Windows' NTFS has such provisions. HFS+ defined an architecture for "unlimited" numbers of streams or forks using the Attributes B-tree structure on the volume.

Under OS X the mandatory data and resource forks of a Macintosh file are exposed on HFS+ volumes as filename and filename/..namedfork/rsrc to BSD environments and as filename and ._filename when "split" (as for use on foreign file systems.) It also introduced tools for splitting and recombining forks split into what are arguably Apple Doubles.

Mac OS X 10.4 Tiger introduced the concept of Extended Attributes which implemented and exposed the additional named forks or streams on HFS+ volumes and through the 'split' fork ._filename on foreign file systems. ACLs are in turn implemented as a privately named and protected Extended Attribute and hence are a third fork that files may have on file systems that have ACLs enabled. Additionally any number of additional "Extended Attributes" (forks) may now be added to or read from a file with setxattr(), getxattr(), listxattr(), and removexattr().

Finder Flags

See `man GetFileInfo` for details, but these include file creator and type, attributes bits (such as locked, stationary, invisible), creation date and modification date. Stored in the ._filename Apple Double half on file systems other than HFS and its derivatives.

POSIX

Files have ownerships (user and group) and permissions. Symbolic links also have these attributes. Unix files also traditionally carry dates such as atime, mtime, and ctime, namely, the time of last access, last time file modified, and change time (last time the file's inode was changed), respectively. Note that ctime changes after operations like chmod.

Creation Dates

The subject of creation dates on the Mac warrants detailed discussion in itself. The Mac has always maintained file "creation date" metadata, it's a Finder file attribute, copied along with the file by the finder, displayed in the Finder's Get Info dialog and important enough that it is supposed to be included as part of the Apple Double file information stored in a ._file. But, while Mac users are long familiar with it and often use it to categorize files it's a rather foreign concept to unix (mostly because it traditionally never existed there), and some unix pundits believe it's just woolly-thinking (see http://mail-index.netbsd.org/netbsd-users/2000/11/22/0000.html) or at best the product of the confused (see http://toadstool.se/journal/2006/01/11/the-fallacy-of-ctime). Other unices and file systems support a btime (birth time) with varying degrees of success.

However, Mac users are used to creation dates, and may get rather agitated if they're lost or, say replaced with the modification time (which seems to be the OS X BSD environment behavior). As such, applications should strongly consider properly handling this metadata or, if they do mangle it, do so in a manner that lends itself to notice (like beginning of epoch).

For a more detailed commentary of Mac file creation dates see http://blog.plasticsfuture.org/2006/06/27/mac-os-file-creation-dates/

BSD Flags

BSD has traditionally maintained a series of flags associated with properties of a file as manipulated by `chflags`. These include flags such as the 'system immutable flag' (schg) that prohibit a file from being modified, or the 'archive flag' (arch).

The classical Desktop Database and OS X .DS_Store files

Traditionally the Macintosh stored certain metadata regarding files in the Desktop Database. Under Mac OS X the .DS_Store file associated with a directory contains similar information such as directory background info, the position or order of files and their icons within a directory, and more. When copying a directory you should probably consider copying these.

Lastly there's one Finder-level piece of information stored, disassociated from the file, which brings us to...

Spotlight Metadata

Tiger introduced us to Spotlight and a huge amount of application specific metadata stored that is associated with a file based on its content. This data is indexed by the kernel through 'importers' as a part of normal operations on the file.

While most of this doesn't need to be preserved since it is indexed as needed by the kernel, one attribute exposed to the Finder and stored in .DS_Store is what traditionally was known as the file Comments, called under Tiger the Spotlight Comments, and is displayed through the Finder's Get Info dialog.

How to preserve file metadata

There a few primary methods by which file metadata get preserved:

  • copied natively
  • preserved in special containers or lists
  • preserved by images of the volumes that support them (e.g. .dmg's)
  • split as Apple Doubles

BSD copies files using copyfile() which splits to Apple Doubles as necessary either internally or externally (with varying results)

Note that, currently, copyfile() munges the modified date into the creation date field of its Apple Doubles.

What needs to be copied? What doesn't?

While mostly a philosophic issue, it's probably safe to say that traditional Macintosh attributes such as creator and type are becoming less important since OS X has various mechanisms for associating files with their applications (sometimes at the cost of additional metadata) while other metadata, like creation date, should arguably be maintained. Spotlight metadata looks like a good candidate for general exclusion since the kernel will automagically recreate this for installed application importers on the target machine, but all Mac OS X systems might not have the same importers installed. This may or may not matter. Old Desktop Database files probably aren't an issue either as Classic withers or is mooted by Intel Macs, but .DS_Stores should probably be maintained if you're backing up a volume. Mandatory file forks should always be copied, but while these are Extended Attributes you might consider not copying the other forks that may exist, such as ACLs, depending on if you're backing up or cloning, or, if you're just duplicating the file elsewhere on a volume. It's hard to tell currently if other Extended Attributes need copying since they're rarely seen in action.

What Copies What?

Let's take a look at what metadata, some common tools and their operations preserve. The following is not designed to be a "report card" and it's not my goal to rank or rate a tool for its handling of the data, but sysadmins should be aware of how operations they perform may affect their data.

Coverage of various third-party backup applications is beyond the scope of this article, but it seems they fare rather poorly with regard to not preserving Extended Attributes aside from resource forks. Creation dates are a mixed bag and may vary within the application based on operation (sometimes intentionally). For additional discussion, see http://blog.plasticsfutures.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/

In general, copyfile(), which most all BSD tools rely on, does not perform consistently compared to the operation of the Finder. Creation date is clobbered by the modification date despite that copyfile() attempts to use Apple Doubles 'internally' and externally to foreign file systems and the Apple Double format specifically calls for the creation date. The various tools that use copyfile() then quite often fail to properly maintain internal or synthetic files resulting in lost Extended Attributes and other data. In rare cases the tool may crash, as in the instance of trying to `rsync -aE` a file with both ACLs and an additional non-resource fork, named Extended Attributes.

Apple System Restore (asr) in device mode manages to copy all the tested data, but this is to be expected since it is essentially a complete copy of the device. In file mode, however, asr, Disk Utility, and hdiutil have digressed in their behavior with OS X 10.4.6. Where previously it maintained locks (the Finder "L" attribute and BSD uchg flags), Extended Attributes and ACLs; it no longer preserves them, though this may change in a future update or (although this makes packages and dmgs problematic) it may be the intended behavior.

`dd` fares pretty much as you'd expect. At least philosophically it's understandable.

Editor's Note: To download a copy of the folloing table for reference please visit the MacTech source code ftp and select this month's issue, 22.08 :

http://www.mactech.com/editorial/filearchives.html



Figure 1. The changes in application code structure


Dan Shoop is principal researcher at iWiring (www.iwiring.net) and a Systems and Networks Architect for US Technical Services (ustsvs.com), both of which provide service and support for the Macintosh and other computer systems. He may be reached at <shoop@iwiring.net>.

 
AAPL
$100.96
Apple Inc.
-0.83
MSFT
$47.52
Microsoft Corpora
+0.84
GOOG
$596.08
Google Inc.
+6.81

MacTech Search:
Community Search:

Software Updates via MacUpdate

Airfoil 4.8.9 - Send audio from any app...
Airfoil allows you to send any audio to AirPort Express units, Apple TVs, and even other Macs and PCs, all in sync! It's your audio - everywhere. With Airfoil you can take audio from any... Read more
WhatRoute 1.13.0 - Geographically trace...
WhatRoute is designed to find the names of all the routers an IP packet passes through on its way from your Mac to a destination host. It also measures the round-trip time from your Mac to the... Read more
Chromium 37.0.2062.122 - Fast and stable...
Chromium is an open-source browser project that aims to build a safer, faster, and more stable way for all Internet users to experience the web. FreeSMUG-Free OpenSource Mac User Group build is... Read more
Attachment Tamer 3.1.14b9 - Take control...
Attachment Tamer gives you control over attachment handling in Apple Mail. It fixes the most annoying Apple Mail flaws, ensures compatibility with other email software, and allows you to set up how... Read more
Duplicate Annihilator 5.0 - Find and del...
Duplicate Annihilator takes on the time-consuming task of comparing the images in your iPhoto library using effective algorithms to make sure that no duplicate escapes. Duplicate Annihilator detects... Read more
jAlbum Pro 12.2 - Organize your digital...
jAlbum Pro has all the features you love in jAlbum, but comes with a commercial license. With jAlbum, you can create gorgeous custom photo galleries for the Web without writing a line of code!... Read more
jAlbum 12.2 - Create custom photo galler...
With jAlbum, you can create gorgeous custom photo galleries for the Web without writing a line of code! Beginner-friendly, with pro results Simply drag and drop photos into groups, choose a design... Read more
Quicken 2015 2.0.4 - Complete personal f...
Quicken 2015 helps you manage all your personal finances in one place, so you can see where you're spending and where you can save. Quicken automatically categorizes your financial transactions,... Read more
iMazing 1.0 - Complete iOS device manage...
iMazing (formerly DiskAid) is the ultimate iOS device manager with capabilities far beyond what iTunes offers. With iMazing and your iOS device (iPhone, iPad, or iPod), you can: Copy music to and... Read more
Xcode 6.0.1 - Integrated development env...
Apple Xcode is Apple Computer's integrated development environment (IDE) for OS X. The full Xcode package is free to ADC members and includes all the tools you need to create, debug, and optimize... Read more

Latest Forum Discussions

See All

View Source – HTML, JavaScript and CSS...
View Source – HTML, JavaScript and CSS 1.0 Device: iOS Universal Category: Utilities Price: $.99, Version: 1.0 (iTunes) Description: View Source is an app plus an iOS 8 Safari extension that makes it easy to do one key web developer... | Read more »
Avenged Sevenfold’s Hail To The King: De...
Avenged Sevenfold’s Hail To The King: Deathbat is Coming to iOS on October 16th Posted by Jessica Fisher on September 19th, 2014 [ permalink ] Just in time for Halloween, on October 16 Avenged Sevenfold will be launching | Read more »
Talisman Has Gone Universal – Can Now be...
Talisman Has Gone Universal – Can Now be Played on the iPhone Posted by Jessica Fisher on September 19th, 2014 [ permalink ] | Read more »
Tap Army Review
Tap Army Review By Jennifer Allen on September 19th, 2014 Our Rating: :: SHOOT EM ALLUniversal App - Designed for iPhone and iPad Mindless but fun, Tap Army is a lane-based shooter that should help you relieve some stress.   | Read more »
Monsters! Volcanoes! Loot! Epic Island f...
Monsters! Volcanoes! Loot! | Read more »
Plunder Pirates: Tips, Tricks, Strategie...
Ahoy There, Seadogs: Interested in knowing our thoughts on all this plundering and pirating? Check out our Plunder Pirates Review! Have you just downloaded the rather enjoyable pirate-em-up Plunder Pirates and are in need of some assistance? Never... | Read more »
Goat Simulator Review
Goat Simulator Review By Lee Hamlet on September 19th, 2014 Our Rating: :: THE GRUFFEST OF BILLY GOATSUniversal App - Designed for iPhone and iPad Unleash chaos as a grumpy goat in this humorous but short-lived casual game.   | Read more »
A New and Improved Wunderlist is Here fo...
A New and Improved Wunderlist is Here for iOS 8 Posted by Jessica Fisher on September 19th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Evernote Update for iOS 8 Adds Web Clipp...
Evernote Update for iOS 8 Adds Web Clipping, Quick Notes, and More Posted by Ellis Spice on September 19th, 2014 [ permalink ] | Read more »
Apple Names Ultimate Productivity Bundl...
Apple Names Ultimate Productivity Bundle by Readdle as the Essential Bundle on the App Store Posted by Jessica Fisher on September 19th, 2014 [ permalink | Read more »

Price Scanner via MacPrices.net

Updated Price Trackers
We’ve updated our Mac Price Trackers with the latest information on prices, bundles, and availability on systems from Apple’s authorized internet/catalog resellers: - 15″ MacBook Pros - 13″ MacBook... Read more
Mac Pros available for up to $260 off MSRP
Adorama has Mac Pros on sale for up to $260 off MSRP. Shipping is free, and Adorama charges sales tax in NY & NJ only: - 4-core Mac Pro: $2839.99, $160 off MSRP - 6-core Mac Pro: $3739.99, $260... Read more
13-inch 2.6GHz/256GB Retina MacBook Pros avai...
B&H Photo has the 13″ 2.6GHz/256GB Retina MacBook Pro on sale for $1379 including free shipping plus NY sales tax only. Their price is $120 off MSRP. Read more
Previous-generation 15-inch 2.0GHz Retina Mac...
B&H Photo has leftover previous-generation 15″ 2.0GHz Retina MacBook Pros now available for $1599 including free shipping plus NY sales tax only. Their price is $400 off original MSRP. B&H... Read more
21″ 2.7GHz iMac available for $1179, save $12...
Adorama has 21″ 2.7GHz Hawell iMacs on sale for $1179.99 including free shipping. Their price is $120 off MSRP. NY and NJ sales tax only. Read more
iOS 8 Adoption Rate Slower than iOS 7, 6, Hit...
Apple began pushing out iOS 8 updates to eligible devices around 1pm ET on September 17, 2014. However, unlike with iOS 7, which boasted a wide variety of differences from its predecessor iOS 6, in... Read more
LIkely Final Definitive OS X 10.9.5 Mavericks...
Apple has released what will almost certainly be the last incremental version number update of OS X 10.9 Mavericks (save for futire security updates) before OS X 10.10 Yosemite is released next month... Read more
Fingerprints, Apple Pay and Identity Theft Wa...
On Sep 9th, CEO Tim Cook unveiled Apple Pay, along with the new iPhone 6 and iWatch. Apple Pay is a newly developed technology that utilizes a near field communication (NFC) to enable customer... Read more
Amazon Introduces Two All-New Kindles
Amazon on Thursday introduced the 7th generation of its Kindle dedicated e-reader device: Kindle Voyage, its top-of-the-line e-reader, and the new $79 Kindle, with a 20% faster processor, twice the... Read more
Save up to $300 on the price of a new Mac wit...
Purchase a new Mac or iPad at The Apple Store for Education and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free,... Read more

Jobs Board

*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Project Manager, *Apple* Financial Services...
**Job Summary** Apple Financial Services (AFS) offers consumers, businesses and educational institutions ways to finance Apple purchases. We work with national and Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.