TweetFollow Us on Twitter

Do You Copy?

Volume Number: 22 (2006)
Issue Number: 8
Column Tag: Field Notes

Do You Copy?

Are you copying all the information about your files that you need?

by Dan Shoop


The Macintosh environment has always been feature rich, and that extends to its files, file attributes, and file metadata the file system maintains for them. OS X has introduced Macintosh to a wider array of application environments which, aside from that of Classic, include beasts such as Carbon, Cocoa, BSD, X-Windows, Web Objects, Java, QuickTime, and now Rosetta. Additionally, OS X supports different file systems that support different metadata. All of these may see and consider files differently than others. This results in files being copied rather differently.

This article is a short attempt to attune application developers, scripters, systems programmers, and file system architects to considerations relating to file metadata stored by the file systems on Mac OS X.

Philosophic Questions

In many respects it is a philosophic question as to what each environment on the Mac should properly copy. Some environments may not normally or naturally have access to all file metadata that OS X maintains for any given file. Named Extended Attributes may not be accessible through Carbon, Cocoa and Core Foundations, while they may be accessible from BSD. The nature of the operation may color and determine what metadata is important and what makes sense for preserving as a copy or whether the metadata should be reflective of a whole new file.

Consider the subtle differences of what metadata, or even files, get maintained as part of the different functions of operations like copies vs archives vs backups vs clones vs sync vs snapshots, et cetera. Is a copied file a new file or is the copy merely a replica or clone? This isn't as easy of a question as it first seems. For instance, if you copy a file from one location to another, does it make sense to maintain any ACLs it may have had or shouldn't the new file have its own, new ACL? But now, what if that "copy" was a backup or clone and part of a larger set of files in its directory? Wouldn't you then expect the ACLs to be maintained? Likewise, if you're backing up a set of files would you expect metadata associated with the directory (like sort order, icon positions, etc) to be maintained for the files? Yet, these are stored in places like .DS_Store or the Desktop Database, not as metadata for the file itself.

What about when we copy files to non-HFS+ file systems? What should be maintained? What should be expected if a file is copied off, then gets copied back? The Macintosh has defined the Apple Double format to store a file non-natively. Is it right for a Windows sysadmin to delete ._file's or .DS_Store's found on their file systems; is this "clutter"? To the Mac user who's lost the CODE resource of an application file, icons or font resources, a file's creation date, or creator/type you may end up with torches and pitchforks at your door if you're inconsiderate, and confusing if your tools are inconsistent. But should an ACL get copied to a foreign file system? Maybe not, but other Extended Attributes clearly are in Apple Doubles.

Moreover, consider what file creation, copying, and modification may mean. For instance consider that cat > file is often seen as creating a new file, but that if 'file' already existed then this operation is in fact a "modification".

Lastly, in the future we may be limited as to what we may and may not copy through DRM. This may become a technical rather than semantic issue as "secure" file systems and files are deployed (as in secure for the owners of the content).

Mac OS X File Metadata and Attributes

Briefly, let's run down some of the most significant file metadata and attributes common to Mac OS X.

Data Streams

The data fork is well known, but all Macintosh files have at least two forks, a data fork and a resource fork, either of which may be of zero length. Implied is that both exist. Under HFS+ multiple named streams may exist, which may be named, but the data and resource streams may not be renamed. Until Tiger, this was merely semantics.

Traditional views of files are just a single byte stream of named data (like found on a tape) with beginning-of-file (BOFs) and end-of-file (EOFs), and little to no metadata. The metadata was just "accounting" information used by the file system, it wasn't really "the data". However, over the years additional information was tacked on by file systems. Users got used to this additional metadata for cataloging and sysadmins got used to preserving them on backups. Environments became more complex and file systems started offering better ways to store ancillary data. The Macintosh isn't unique in offering multiple streams per file. Even Windows' NTFS has such provisions. HFS+ defined an architecture for "unlimited" numbers of streams or forks using the Attributes B-tree structure on the volume.

Under OS X the mandatory data and resource forks of a Macintosh file are exposed on HFS+ volumes as filename and filename/..namedfork/rsrc to BSD environments and as filename and ._filename when "split" (as for use on foreign file systems.) It also introduced tools for splitting and recombining forks split into what are arguably Apple Doubles.

Mac OS X 10.4 Tiger introduced the concept of Extended Attributes which implemented and exposed the additional named forks or streams on HFS+ volumes and through the 'split' fork ._filename on foreign file systems. ACLs are in turn implemented as a privately named and protected Extended Attribute and hence are a third fork that files may have on file systems that have ACLs enabled. Additionally any number of additional "Extended Attributes" (forks) may now be added to or read from a file with setxattr(), getxattr(), listxattr(), and removexattr().

Finder Flags

See `man GetFileInfo` for details, but these include file creator and type, attributes bits (such as locked, stationary, invisible), creation date and modification date. Stored in the ._filename Apple Double half on file systems other than HFS and its derivatives.


Files have ownerships (user and group) and permissions. Symbolic links also have these attributes. Unix files also traditionally carry dates such as atime, mtime, and ctime, namely, the time of last access, last time file modified, and change time (last time the file's inode was changed), respectively. Note that ctime changes after operations like chmod.

Creation Dates

The subject of creation dates on the Mac warrants detailed discussion in itself. The Mac has always maintained file "creation date" metadata, it's a Finder file attribute, copied along with the file by the finder, displayed in the Finder's Get Info dialog and important enough that it is supposed to be included as part of the Apple Double file information stored in a ._file. But, while Mac users are long familiar with it and often use it to categorize files it's a rather foreign concept to unix (mostly because it traditionally never existed there), and some unix pundits believe it's just woolly-thinking (see or at best the product of the confused (see Other unices and file systems support a btime (birth time) with varying degrees of success.

However, Mac users are used to creation dates, and may get rather agitated if they're lost or, say replaced with the modification time (which seems to be the OS X BSD environment behavior). As such, applications should strongly consider properly handling this metadata or, if they do mangle it, do so in a manner that lends itself to notice (like beginning of epoch).

For a more detailed commentary of Mac file creation dates see

BSD Flags

BSD has traditionally maintained a series of flags associated with properties of a file as manipulated by `chflags`. These include flags such as the 'system immutable flag' (schg) that prohibit a file from being modified, or the 'archive flag' (arch).

The classical Desktop Database and OS X .DS_Store files

Traditionally the Macintosh stored certain metadata regarding files in the Desktop Database. Under Mac OS X the .DS_Store file associated with a directory contains similar information such as directory background info, the position or order of files and their icons within a directory, and more. When copying a directory you should probably consider copying these.

Lastly there's one Finder-level piece of information stored, disassociated from the file, which brings us to...

Spotlight Metadata

Tiger introduced us to Spotlight and a huge amount of application specific metadata stored that is associated with a file based on its content. This data is indexed by the kernel through 'importers' as a part of normal operations on the file.

While most of this doesn't need to be preserved since it is indexed as needed by the kernel, one attribute exposed to the Finder and stored in .DS_Store is what traditionally was known as the file Comments, called under Tiger the Spotlight Comments, and is displayed through the Finder's Get Info dialog.

How to preserve file metadata

There a few primary methods by which file metadata get preserved:

  • copied natively
  • preserved in special containers or lists
  • preserved by images of the volumes that support them (e.g. .dmg's)
  • split as Apple Doubles

BSD copies files using copyfile() which splits to Apple Doubles as necessary either internally or externally (with varying results)

Note that, currently, copyfile() munges the modified date into the creation date field of its Apple Doubles.

What needs to be copied? What doesn't?

While mostly a philosophic issue, it's probably safe to say that traditional Macintosh attributes such as creator and type are becoming less important since OS X has various mechanisms for associating files with their applications (sometimes at the cost of additional metadata) while other metadata, like creation date, should arguably be maintained. Spotlight metadata looks like a good candidate for general exclusion since the kernel will automagically recreate this for installed application importers on the target machine, but all Mac OS X systems might not have the same importers installed. This may or may not matter. Old Desktop Database files probably aren't an issue either as Classic withers or is mooted by Intel Macs, but .DS_Stores should probably be maintained if you're backing up a volume. Mandatory file forks should always be copied, but while these are Extended Attributes you might consider not copying the other forks that may exist, such as ACLs, depending on if you're backing up or cloning, or, if you're just duplicating the file elsewhere on a volume. It's hard to tell currently if other Extended Attributes need copying since they're rarely seen in action.

What Copies What?

Let's take a look at what metadata, some common tools and their operations preserve. The following is not designed to be a "report card" and it's not my goal to rank or rate a tool for its handling of the data, but sysadmins should be aware of how operations they perform may affect their data.

Coverage of various third-party backup applications is beyond the scope of this article, but it seems they fare rather poorly with regard to not preserving Extended Attributes aside from resource forks. Creation dates are a mixed bag and may vary within the application based on operation (sometimes intentionally). For additional discussion, see

In general, copyfile(), which most all BSD tools rely on, does not perform consistently compared to the operation of the Finder. Creation date is clobbered by the modification date despite that copyfile() attempts to use Apple Doubles 'internally' and externally to foreign file systems and the Apple Double format specifically calls for the creation date. The various tools that use copyfile() then quite often fail to properly maintain internal or synthetic files resulting in lost Extended Attributes and other data. In rare cases the tool may crash, as in the instance of trying to `rsync -aE` a file with both ACLs and an additional non-resource fork, named Extended Attributes.

Apple System Restore (asr) in device mode manages to copy all the tested data, but this is to be expected since it is essentially a complete copy of the device. In file mode, however, asr, Disk Utility, and hdiutil have digressed in their behavior with OS X 10.4.6. Where previously it maintained locks (the Finder "L" attribute and BSD uchg flags), Extended Attributes and ACLs; it no longer preserves them, though this may change in a future update or (although this makes packages and dmgs problematic) it may be the intended behavior.

`dd` fares pretty much as you'd expect. At least philosophically it's understandable.

Editor's Note: To download a copy of the folloing table for reference please visit the MacTech source code ftp and select this month's issue, 22.08 :

Figure 1. The changes in application code structure

Dan Shoop is principal researcher at iWiring ( and a Systems and Networks Architect for US Technical Services (, both of which provide service and support for the Macintosh and other computer systems. He may be reached at <>.


Community Search:
MacTech Search:

Software Updates via MacUpdate

World of Tanks Generals guide - Tips and...
World of Tanks Generals is a brand new card game by the developer behind the World of Tanks shooter franchise. It plays like a cross between chess and your typical card game. You have to keep in consideration where you place your tanks on the board... | Read more »
TruckSimulation 16 guide: How to succeed...
Remember those strangely enjoyable truck missions in Grand Theft Auto V whereit was a disturbing amount of fun to deliver cargo? TruckSimulation 16 is reminiscent of that, and has you play the role of a truck driver who has to deliver various... | Read more »
The best GIF making apps
Animated GIFs have exploded in popularity recently which is likely thanks to a combination of Tumblr, our shorter attention spans, and the simple fact they’re a lot of fun. [Read more] | Read more »
The best remote desktop apps for iOS
We've been sifting through the App Store to find the best ways to do computer tasks on a tablet. That gave us a thought - what if we could just do computer tasks from our tablets? Here's a list of the best remote desktop apps to help you use your... | Read more »
Warhammer 40,000: Freeblade guide - How...
Warhammer 40,000: Freebladejust launched in the App Store and it lets you live your childhood dream of blowing up and slashing a bunch of enemies as a massive, hulking Space Marine. It's not easy being a Space Marine though - and particularly if... | Read more »
Gopogo guide - How to bounce like the be...
Nitrome just launched a new game and, as to be expected, it's a lot of addictive fun. It's called Gopogo, and it challenges you to hoparound a bunch of platforms, avoiding enemies and picking up shiny stuff. It's not easy though - just like the... | Read more »
Sago Mini Superhero (Education)
Sago Mini Superhero 1.0 Device: iOS Universal Category: Education Price: $2.99, Version: 1.0 (iTunes) Description: KAPOW! Jack the rabbit bursts into the sky as the Sago Mini Superhero! Fly with Jack as he lifts impossible weights,... | Read more »
Star Wars: Galaxy of Heroes guide - How...
Star Wars: Galaxy of Heroes is all about collecting heroes, powering them up, and using them together to defeat your foes. It's pretty straightforward stuff for the most part, but increasing your characters' stats can be a bit confusing because it... | Read more »
The best cooking apps (just in time for...
It’s that time of year again, where you’ll be gathering around the dinner table with your family and a huge feast in front of you. [Read more] | Read more »
Square Rave guide - How to grab those te...
Square Rave is an awesome little music-oriented puzzle game that smacks of games like Lumines, but with its own unique sense of gameplay. To help wrap your head around the game, keep the following tips and tricks in mind. [Read more] | Read more »

Price Scanner via

Cyber Monday: Apple MacBooks on sale for up t...
Apple resellers have MacBook Pros, MacBook Airs, and MacBooks on sale for up to $250 off MSRP for Cyber Monday 2015. The following is a roundup of the lowest prices available for new models from any... Read more
Cyber Monday: 15% off Apple products, and sto...
Use code CYBER15 on Cyber Monday only to take 15% on Apple products at Target, and store-wide. Choose free shipping or free local store pickup (if available). Sale prices for online orders only, in-... Read more
iPad Air 2 And iPad mini Among Top Five Black...
Adobe has released its 2015 online shopping data for Black Friday and Thanksgiving Day. The five best selling electronic products on Black Friday were Samsung 4K TVs, Apple iPad Air 2, Microsoft Xbox... Read more
All-in-one PC Shipments Projected To Drop Ove...
Digitimes’ Aaron Lee and Joseph Tsai report that all-in-one (AIO) PC shipments may drop a double-digit percentage on-year in 2015 due to weaker-than-expected demand, although second-largest AIO make... Read more
Sprint Offers iPad Pro
Sprint now offers Apple’s new iPad Pro with Wi-Fi + Cellular, featuring a 12.9-inch Retina display with 5.6 million pixels. Customers can pick up iPad Pro at select Sprint retail locations. It can... Read more
Cyber Monday: Target offers 15% discount on A...
Target has discounted Apple Watches by 15% for Cyber Monday. Choose free shipping or free local store pickup (if available). Sale prices for online orders only, in-store prices may vary: - Apple... Read more
Sunday roundup of Holiday weekend Mac sales:...
Take up to $500 off MSRP on the price of a new Mac at B&H Photo today as part of their Black Friday/Holiday weekend sale. Shipping is free, and B&H charges NY tax only. These prices are... Read more
Holiday weekend: Apple Watch on sale for $50-...
B&H Photo has the Apple Watch on sale today for $50-$100 off MSRP. Shipping is free, and B&H charges NY sales tax only: - Apple Watch Sport: $50 off - Apple Watch: $50-$100 off Read more
Holiday weekend: iPad Air 2s on sale for up t...
B&H Photo has iPad Air 2s on sale for up to $80 off MSRP including free shipping plus NY sales tax only: - 16GB iPad Air 2 WiFi: $459 $40 off - 64GB iPad Air 2 WiFi: $569 $30 off - 128GB iPad Air... Read more
Holiday weekend Mac sales roundup: B&H Ph...
B&H Photo continues to have all new Macs on sale for up to $500 off MSRP as part of their Black Friday/Holiday weekend sale. Shipping is free, and B&H charges NY tax only: - 15″ 2.2GHz Retina... Read more

Jobs Board

Technical Program Manager, Strategic Merchant...
# Technical Program Manager, Strategic Merchants - Apple Pay Job Number: 44001177 Santa Clara Valley, Califo ia, United States Posted: Oct. 30, 2015 Weekly Hours: 40.00 Read more
Frameworks Engineer, *Apple* Watch - Apple...
# Frameworks Engineer, Apple Watch Job Number: 41403122 Santa Clara Valley, Califo ia, United States Posted: Jul. 1, 2015 Weekly Hours: 40.00 **Job Summary** Join the Read more
Software Engineer - *Apple* Pay - Apple (Un...
# Software Engineer - Apple Pay Job Number: 44003246 Santa Clara Valley, Califo ia, United States Posted: Nov. 16, 2015 Weekly Hours: 40.00 **Job Summary** Apple Pay Read more
Merchant Operations Manager: *Apple* Pay -...
# Merchant Operations Manager: Apple Pay Job Number: 43593822 Santa Clara Valley, Califo ia, United States Posted: Nov. 10, 2015 Weekly Hours: 40.00 **Job Summary** The Read more
Product Design Engineer - *Apple* Watch - A...
# Product Design Engineer - Apple Watch Job Number: 41727161 Santa Clara Valley, Califo ia, United States Posted: Jul. 22, 2015 Weekly Hours: 40.00 **Job Summary** Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.