TweetFollow Us on Twitter

Do You Copy?

Volume Number: 22 (2006)
Issue Number: 8
Column Tag: Field Notes

Do You Copy?

Are you copying all the information about your files that you need?

by Dan Shoop


The Macintosh environment has always been feature rich, and that extends to its files, file attributes, and file metadata the file system maintains for them. OS X has introduced Macintosh to a wider array of application environments which, aside from that of Classic, include beasts such as Carbon, Cocoa, BSD, X-Windows, Web Objects, Java, QuickTime, and now Rosetta. Additionally, OS X supports different file systems that support different metadata. All of these may see and consider files differently than others. This results in files being copied rather differently.

This article is a short attempt to attune application developers, scripters, systems programmers, and file system architects to considerations relating to file metadata stored by the file systems on Mac OS X.

Philosophic Questions

In many respects it is a philosophic question as to what each environment on the Mac should properly copy. Some environments may not normally or naturally have access to all file metadata that OS X maintains for any given file. Named Extended Attributes may not be accessible through Carbon, Cocoa and Core Foundations, while they may be accessible from BSD. The nature of the operation may color and determine what metadata is important and what makes sense for preserving as a copy or whether the metadata should be reflective of a whole new file.

Consider the subtle differences of what metadata, or even files, get maintained as part of the different functions of operations like copies vs archives vs backups vs clones vs sync vs snapshots, et cetera. Is a copied file a new file or is the copy merely a replica or clone? This isn't as easy of a question as it first seems. For instance, if you copy a file from one location to another, does it make sense to maintain any ACLs it may have had or shouldn't the new file have its own, new ACL? But now, what if that "copy" was a backup or clone and part of a larger set of files in its directory? Wouldn't you then expect the ACLs to be maintained? Likewise, if you're backing up a set of files would you expect metadata associated with the directory (like sort order, icon positions, etc) to be maintained for the files? Yet, these are stored in places like .DS_Store or the Desktop Database, not as metadata for the file itself.

What about when we copy files to non-HFS+ file systems? What should be maintained? What should be expected if a file is copied off, then gets copied back? The Macintosh has defined the Apple Double format to store a file non-natively. Is it right for a Windows sysadmin to delete ._file's or .DS_Store's found on their file systems; is this "clutter"? To the Mac user who's lost the CODE resource of an application file, icons or font resources, a file's creation date, or creator/type you may end up with torches and pitchforks at your door if you're inconsiderate, and confusing if your tools are inconsistent. But should an ACL get copied to a foreign file system? Maybe not, but other Extended Attributes clearly are in Apple Doubles.

Moreover, consider what file creation, copying, and modification may mean. For instance consider that cat > file is often seen as creating a new file, but that if 'file' already existed then this operation is in fact a "modification".

Lastly, in the future we may be limited as to what we may and may not copy through DRM. This may become a technical rather than semantic issue as "secure" file systems and files are deployed (as in secure for the owners of the content).

Mac OS X File Metadata and Attributes

Briefly, let's run down some of the most significant file metadata and attributes common to Mac OS X.

Data Streams

The data fork is well known, but all Macintosh files have at least two forks, a data fork and a resource fork, either of which may be of zero length. Implied is that both exist. Under HFS+ multiple named streams may exist, which may be named, but the data and resource streams may not be renamed. Until Tiger, this was merely semantics.

Traditional views of files are just a single byte stream of named data (like found on a tape) with beginning-of-file (BOFs) and end-of-file (EOFs), and little to no metadata. The metadata was just "accounting" information used by the file system, it wasn't really "the data". However, over the years additional information was tacked on by file systems. Users got used to this additional metadata for cataloging and sysadmins got used to preserving them on backups. Environments became more complex and file systems started offering better ways to store ancillary data. The Macintosh isn't unique in offering multiple streams per file. Even Windows' NTFS has such provisions. HFS+ defined an architecture for "unlimited" numbers of streams or forks using the Attributes B-tree structure on the volume.

Under OS X the mandatory data and resource forks of a Macintosh file are exposed on HFS+ volumes as filename and filename/..namedfork/rsrc to BSD environments and as filename and ._filename when "split" (as for use on foreign file systems.) It also introduced tools for splitting and recombining forks split into what are arguably Apple Doubles.

Mac OS X 10.4 Tiger introduced the concept of Extended Attributes which implemented and exposed the additional named forks or streams on HFS+ volumes and through the 'split' fork ._filename on foreign file systems. ACLs are in turn implemented as a privately named and protected Extended Attribute and hence are a third fork that files may have on file systems that have ACLs enabled. Additionally any number of additional "Extended Attributes" (forks) may now be added to or read from a file with setxattr(), getxattr(), listxattr(), and removexattr().

Finder Flags

See `man GetFileInfo` for details, but these include file creator and type, attributes bits (such as locked, stationary, invisible), creation date and modification date. Stored in the ._filename Apple Double half on file systems other than HFS and its derivatives.


Files have ownerships (user and group) and permissions. Symbolic links also have these attributes. Unix files also traditionally carry dates such as atime, mtime, and ctime, namely, the time of last access, last time file modified, and change time (last time the file's inode was changed), respectively. Note that ctime changes after operations like chmod.

Creation Dates

The subject of creation dates on the Mac warrants detailed discussion in itself. The Mac has always maintained file "creation date" metadata, it's a Finder file attribute, copied along with the file by the finder, displayed in the Finder's Get Info dialog and important enough that it is supposed to be included as part of the Apple Double file information stored in a ._file. But, while Mac users are long familiar with it and often use it to categorize files it's a rather foreign concept to unix (mostly because it traditionally never existed there), and some unix pundits believe it's just woolly-thinking (see or at best the product of the confused (see Other unices and file systems support a btime (birth time) with varying degrees of success.

However, Mac users are used to creation dates, and may get rather agitated if they're lost or, say replaced with the modification time (which seems to be the OS X BSD environment behavior). As such, applications should strongly consider properly handling this metadata or, if they do mangle it, do so in a manner that lends itself to notice (like beginning of epoch).

For a more detailed commentary of Mac file creation dates see

BSD Flags

BSD has traditionally maintained a series of flags associated with properties of a file as manipulated by `chflags`. These include flags such as the 'system immutable flag' (schg) that prohibit a file from being modified, or the 'archive flag' (arch).

The classical Desktop Database and OS X .DS_Store files

Traditionally the Macintosh stored certain metadata regarding files in the Desktop Database. Under Mac OS X the .DS_Store file associated with a directory contains similar information such as directory background info, the position or order of files and their icons within a directory, and more. When copying a directory you should probably consider copying these.

Lastly there's one Finder-level piece of information stored, disassociated from the file, which brings us to...

Spotlight Metadata

Tiger introduced us to Spotlight and a huge amount of application specific metadata stored that is associated with a file based on its content. This data is indexed by the kernel through 'importers' as a part of normal operations on the file.

While most of this doesn't need to be preserved since it is indexed as needed by the kernel, one attribute exposed to the Finder and stored in .DS_Store is what traditionally was known as the file Comments, called under Tiger the Spotlight Comments, and is displayed through the Finder's Get Info dialog.

How to preserve file metadata

There a few primary methods by which file metadata get preserved:

  • copied natively
  • preserved in special containers or lists
  • preserved by images of the volumes that support them (e.g. .dmg's)
  • split as Apple Doubles

BSD copies files using copyfile() which splits to Apple Doubles as necessary either internally or externally (with varying results)

Note that, currently, copyfile() munges the modified date into the creation date field of its Apple Doubles.

What needs to be copied? What doesn't?

While mostly a philosophic issue, it's probably safe to say that traditional Macintosh attributes such as creator and type are becoming less important since OS X has various mechanisms for associating files with their applications (sometimes at the cost of additional metadata) while other metadata, like creation date, should arguably be maintained. Spotlight metadata looks like a good candidate for general exclusion since the kernel will automagically recreate this for installed application importers on the target machine, but all Mac OS X systems might not have the same importers installed. This may or may not matter. Old Desktop Database files probably aren't an issue either as Classic withers or is mooted by Intel Macs, but .DS_Stores should probably be maintained if you're backing up a volume. Mandatory file forks should always be copied, but while these are Extended Attributes you might consider not copying the other forks that may exist, such as ACLs, depending on if you're backing up or cloning, or, if you're just duplicating the file elsewhere on a volume. It's hard to tell currently if other Extended Attributes need copying since they're rarely seen in action.

What Copies What?

Let's take a look at what metadata, some common tools and their operations preserve. The following is not designed to be a "report card" and it's not my goal to rank or rate a tool for its handling of the data, but sysadmins should be aware of how operations they perform may affect their data.

Coverage of various third-party backup applications is beyond the scope of this article, but it seems they fare rather poorly with regard to not preserving Extended Attributes aside from resource forks. Creation dates are a mixed bag and may vary within the application based on operation (sometimes intentionally). For additional discussion, see

In general, copyfile(), which most all BSD tools rely on, does not perform consistently compared to the operation of the Finder. Creation date is clobbered by the modification date despite that copyfile() attempts to use Apple Doubles 'internally' and externally to foreign file systems and the Apple Double format specifically calls for the creation date. The various tools that use copyfile() then quite often fail to properly maintain internal or synthetic files resulting in lost Extended Attributes and other data. In rare cases the tool may crash, as in the instance of trying to `rsync -aE` a file with both ACLs and an additional non-resource fork, named Extended Attributes.

Apple System Restore (asr) in device mode manages to copy all the tested data, but this is to be expected since it is essentially a complete copy of the device. In file mode, however, asr, Disk Utility, and hdiutil have digressed in their behavior with OS X 10.4.6. Where previously it maintained locks (the Finder "L" attribute and BSD uchg flags), Extended Attributes and ACLs; it no longer preserves them, though this may change in a future update or (although this makes packages and dmgs problematic) it may be the intended behavior.

`dd` fares pretty much as you'd expect. At least philosophically it's understandable.

Editor's Note: To download a copy of the folloing table for reference please visit the MacTech source code ftp and select this month's issue, 22.08 :

Figure 1. The changes in application code structure

Dan Shoop is principal researcher at iWiring ( and a Systems and Networks Architect for US Technical Services (, both of which provide service and support for the Macintosh and other computer systems. He may be reached at <>.


Community Search:
MacTech Search:

Software Updates via MacUpdate

AirPort Utility 6.3.6 - Set up and manag...
Note: Most recent release available only within OS X 10.11 El Capitan update. Use AirPort Utility to set up and manage your Wi-Fi network and AirPort base stations, including AirPort Express, AirPort... Read more
Quicksilver 1.3.1 - Application launcher...
Quicksilver is a light, fast and free Mac application that gives you the power to control your Mac with keystrokes alone. Quicksilver allows you to find what you need quickly and easily, then act... Read more
Tidy Up (Five Users) 4.1.5 - Find duplic...
Tidy Up is a complete duplicate finder and disk-tidiness utility. With Tidy Up you can search for duplicate files and packages by the owner application, content, type, creator, extension, time... Read more
Mellel 3.4.3 - The word processor of cho...
Mellel is the leading word processor for OS X and has been widely considered the industry standard since its inception. Mellel focuses on writers and scholars for technical writing and multilingual... Read more
Skype - Voice-over-internet p...
Skype allows you to talk to friends, family and co-workers across the Internet without the inconvenience of long distance telephone charges. Using peer-to-peer data transmission technology, Skype... Read more
Bookends 12.6.0 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Access the power of Bookends directly from Mellel, Nisus Writer Pro, or MS Word (... Read more
Apple iBooks Author 2.4 - Create and pub...
Apple iBooks Author helps you create and publish amazing Multi-Touch books for iPad. Now anyone can create stunning iBooks textbooks, cookbooks, history books, picture books, and more for iPad. All... Read more
Web Snapper 3.3.9 - Capture entire Web p...
Web Snapper lets you capture Web pages exactly as they appear in your browser. You can send them to a file as images or vector-based, multi-page PDFs. It captures the whole Web page - eliminating the... Read more
Tunnelblick 3.6beta10 - GUI for OpenVPN...
Tunnelblick is a free, open source graphic user interface for OpenVPN on OS X. It provides easy control of OpenVPN client and/or server connections. It comes as a ready-to-use application with all... Read more
EtreCheck 2.5.1 - For troubleshooting yo...
EtreCheck is a simple little app to display the important details of your system configuration and allow you to copy that information to the Clipboard. It is meant to be used with Apple Support... Read more

Camel Up (Games)
Camel Up 1.0.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.0 (iTunes) Description: | Read more »
The Martian: Bring Him Home (Games)
The Martian: Bring Him Home 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: Based on the best selling novel and critically acclaimed film, THE MARTIAN tells the story of Astronaut Mark... | Read more »
This Week at 148Apps: September 21-30, 2...
Leap Into Fall With 148Apps How do you know what apps are worth your time and money? Just look to the review team at 148Apps. We sort through the chaos and find the apps you're looking for. The ones we love become Editor’s Choice, standing out above... | Read more »
Tweetbot 4 for Twitter (Social Networki...
Tweetbot 4 for Twitter 4.0 Device: iOS Universal Category: Social Networking Price: $4.99, Version: 4.0 (iTunes) Description: *** 50% off for a limited time. *** | Read more »
Mori (Games)
Mori 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: Stop, rewind and unwind with Mori. Time is always running, take a moment to take control. Mori is an action puzzle game about infinitely... | Read more »
100 Years' War (Games)
100 Years' War 1.0 Device: iOS Universal Category: Games Price: $3.99, Version: 1.0 (iTunes) Description: | Read more »
Tower in the Sky (Games)
Tower in the Sky 0.0.60 Device: iOS Universal Category: Games Price: $1.99, Version: 0.0.60 (iTunes) Description: | Read more »
hocus. (Games)
hocus. 1.0.0 Device: iOS Universal Category: Games Price: $.99, Version: 1.0.0 (iTunes) Description: New, polished, mind-bending, minimal puzzle game with dozens of levels and extra-ordinary design Features:- Beautifully crafted... | Read more »
Mos Speedrun 2 (Games)
Mos Speedrun 2 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Mos is back, in her biggest and most exciting adventure ever! Wall-jump to victory through 30 mysterious, action packed levels... | Read more »
3D Touch could be a game-changer, but it...
Were you one of the lucky/financially secure enough ones to buy a new iPhone 6s or iPhone 6s Plus over the weekend? Yup, me too (I’m not convinced I was either of those two things, but let’s go with lucky for now), so I thought I’d delve into just... | Read more »

Price Scanner via

Bare Bones Software Releases TextWrangler 5.0...
Bare Bones Software has announced the release and immediate availability of TextWrangler 5.0, a major upgrade to its free, high performance, general purpose text editor for Mac OS X. Built on a new,... Read more
Apple refurbished iPad Air 2s available for u...
Apple has Certified Refurbished iPad Air 2s available for up to $140 off the price of new models. Apple’s one-year warranty is included with each model, and shipping is free: - 128GB Wi-Fi iPad Air 2... Read more
Save up to $100 on Mac AppleCare Protection P...
Adorama has 3-Year AppleCare Warranties on sale for up to $100 off MSRP including free shipping plus NY & NJ sales tax only: - Mac Laptops 15″ and Above: $249 $100 off MSRP - Mac Laptops 13″ and... Read more
Updated Mac Price Trackers
We’ve updated our Mac Price Trackers with the latest information on prices, bundles, and availability on systems from Apple’s authorized internet/catalog resellers: - 15″ MacBook Pros - 13″ MacBook... Read more
27-inch 3.5GHz 5K iMac on sale for $81 off MS...
Adorama has the 27″ 3.5GHz 5K iMac on sale for $2218.99, $81 off MSRP, including a free copy of Apple’s 3-Year AppleCare Protection Plan, plus a free external DVD/CD drive, and a copy of Corel... Read more
12-inch MacBooks in stock for up to $120 off,...
Adorama has 12″ Retina MacBooks in stock for up to $120 off MSRP including free shipping plus NY & NJ sales tax only. For a limited time, Adorama will include a free Apple USB-C to USB Adapter,... Read more
EasyDocScan Turns your iPhone or iPad into a...
Note-Ify Apps has announced the release and immediate availability of EasyDocScan 1.0, their new business app developed for iPhone, iPad and iPod touch devices. EasyDocScan is a simple application... Read more
15-inch 2.2GHz Retina MacBook Pro on sale for... has the 15″ 2.2GHz Retina MacBook Pro on sale for $1799 including free shipping. Their price is $200 off MSRP, and it’s the lowest price available for this model (except for Apple’s $1699... Read more
2.8GHz Mac mini available for $988, includes...
Adorama has the 2.8GHz Mac mini available for $988, $11 off MSRP, including a free copy of Apple’s 3-Year AppleCare Protection Plan. Shipping is free, and Adorama charges sales tax in NY & NJ... Read more
iPhone 6s and 6s Plus Feature Improved Durabi...
Upgraded components in the new iPhone 6s Plus cost $16 more than the components in the earlier iPhone 6 Plus according to a preliminary estimate from IHS Inc. The bill of materials (BOM) for an... Read more

Jobs Board

Senior Payments Security Manager - *Apple*...
**Job Summary** Apple , Inc. is looking for a highly motivated, innovative and hands-on senior payments security manager to join the Apple Pay security team. You will Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
*Apple* Site Security Manager - Apple (Unite...
**Job Summary** The Site Security Manager is a high-profile security position at Apple . The Site Manager is the face of Apple Global Security (GS) and primary point Read more
*Apple* Fulfillment Operations Execution Ana...
**Job Summary** The AMR Apple Fulfillment Operations Team is seeking a talented team player to drive the Apple Online Store (AOS) fulfillment performance to ensure a Read more
*Apple* Distinguished Educator (ADE) Communi...
**Job Summary** Apple is seeking candidates for a new position on the Apple Distinguished Educator (ADE) Program team as ADE Community Support Manager. Join a team Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.