TweetFollow Us on Twitter

Do You Copy?

Volume Number: 22 (2006)
Issue Number: 8
Column Tag: Field Notes

Do You Copy?

Are you copying all the information about your files that you need?

by Dan Shoop

Introduction

The Macintosh environment has always been feature rich, and that extends to its files, file attributes, and file metadata the file system maintains for them. OS X has introduced Macintosh to a wider array of application environments which, aside from that of Classic, include beasts such as Carbon, Cocoa, BSD, X-Windows, Web Objects, Java, QuickTime, and now Rosetta. Additionally, OS X supports different file systems that support different metadata. All of these may see and consider files differently than others. This results in files being copied rather differently.

This article is a short attempt to attune application developers, scripters, systems programmers, and file system architects to considerations relating to file metadata stored by the file systems on Mac OS X.

Philosophic Questions

In many respects it is a philosophic question as to what each environment on the Mac should properly copy. Some environments may not normally or naturally have access to all file metadata that OS X maintains for any given file. Named Extended Attributes may not be accessible through Carbon, Cocoa and Core Foundations, while they may be accessible from BSD. The nature of the operation may color and determine what metadata is important and what makes sense for preserving as a copy or whether the metadata should be reflective of a whole new file.

Consider the subtle differences of what metadata, or even files, get maintained as part of the different functions of operations like copies vs archives vs backups vs clones vs sync vs snapshots, et cetera. Is a copied file a new file or is the copy merely a replica or clone? This isn't as easy of a question as it first seems. For instance, if you copy a file from one location to another, does it make sense to maintain any ACLs it may have had or shouldn't the new file have its own, new ACL? But now, what if that "copy" was a backup or clone and part of a larger set of files in its directory? Wouldn't you then expect the ACLs to be maintained? Likewise, if you're backing up a set of files would you expect metadata associated with the directory (like sort order, icon positions, etc) to be maintained for the files? Yet, these are stored in places like .DS_Store or the Desktop Database, not as metadata for the file itself.

What about when we copy files to non-HFS+ file systems? What should be maintained? What should be expected if a file is copied off, then gets copied back? The Macintosh has defined the Apple Double format to store a file non-natively. Is it right for a Windows sysadmin to delete ._file's or .DS_Store's found on their file systems; is this "clutter"? To the Mac user who's lost the CODE resource of an application file, icons or font resources, a file's creation date, or creator/type you may end up with torches and pitchforks at your door if you're inconsiderate, and confusing if your tools are inconsistent. But should an ACL get copied to a foreign file system? Maybe not, but other Extended Attributes clearly are in Apple Doubles.

Moreover, consider what file creation, copying, and modification may mean. For instance consider that cat > file is often seen as creating a new file, but that if 'file' already existed then this operation is in fact a "modification".

Lastly, in the future we may be limited as to what we may and may not copy through DRM. This may become a technical rather than semantic issue as "secure" file systems and files are deployed (as in secure for the owners of the content).

Mac OS X File Metadata and Attributes

Briefly, let's run down some of the most significant file metadata and attributes common to Mac OS X.

Data Streams

The data fork is well known, but all Macintosh files have at least two forks, a data fork and a resource fork, either of which may be of zero length. Implied is that both exist. Under HFS+ multiple named streams may exist, which may be named, but the data and resource streams may not be renamed. Until Tiger, this was merely semantics.

Traditional views of files are just a single byte stream of named data (like found on a tape) with beginning-of-file (BOFs) and end-of-file (EOFs), and little to no metadata. The metadata was just "accounting" information used by the file system, it wasn't really "the data". However, over the years additional information was tacked on by file systems. Users got used to this additional metadata for cataloging and sysadmins got used to preserving them on backups. Environments became more complex and file systems started offering better ways to store ancillary data. The Macintosh isn't unique in offering multiple streams per file. Even Windows' NTFS has such provisions. HFS+ defined an architecture for "unlimited" numbers of streams or forks using the Attributes B-tree structure on the volume.

Under OS X the mandatory data and resource forks of a Macintosh file are exposed on HFS+ volumes as filename and filename/..namedfork/rsrc to BSD environments and as filename and ._filename when "split" (as for use on foreign file systems.) It also introduced tools for splitting and recombining forks split into what are arguably Apple Doubles.

Mac OS X 10.4 Tiger introduced the concept of Extended Attributes which implemented and exposed the additional named forks or streams on HFS+ volumes and through the 'split' fork ._filename on foreign file systems. ACLs are in turn implemented as a privately named and protected Extended Attribute and hence are a third fork that files may have on file systems that have ACLs enabled. Additionally any number of additional "Extended Attributes" (forks) may now be added to or read from a file with setxattr(), getxattr(), listxattr(), and removexattr().

Finder Flags

See `man GetFileInfo` for details, but these include file creator and type, attributes bits (such as locked, stationary, invisible), creation date and modification date. Stored in the ._filename Apple Double half on file systems other than HFS and its derivatives.

POSIX

Files have ownerships (user and group) and permissions. Symbolic links also have these attributes. Unix files also traditionally carry dates such as atime, mtime, and ctime, namely, the time of last access, last time file modified, and change time (last time the file's inode was changed), respectively. Note that ctime changes after operations like chmod.

Creation Dates

The subject of creation dates on the Mac warrants detailed discussion in itself. The Mac has always maintained file "creation date" metadata, it's a Finder file attribute, copied along with the file by the finder, displayed in the Finder's Get Info dialog and important enough that it is supposed to be included as part of the Apple Double file information stored in a ._file. But, while Mac users are long familiar with it and often use it to categorize files it's a rather foreign concept to unix (mostly because it traditionally never existed there), and some unix pundits believe it's just woolly-thinking (see http://mail-index.netbsd.org/netbsd-users/2000/11/22/0000.html) or at best the product of the confused (see http://toadstool.se/journal/2006/01/11/the-fallacy-of-ctime). Other unices and file systems support a btime (birth time) with varying degrees of success.

However, Mac users are used to creation dates, and may get rather agitated if they're lost or, say replaced with the modification time (which seems to be the OS X BSD environment behavior). As such, applications should strongly consider properly handling this metadata or, if they do mangle it, do so in a manner that lends itself to notice (like beginning of epoch).

For a more detailed commentary of Mac file creation dates see http://blog.plasticsfuture.org/2006/06/27/mac-os-file-creation-dates/

BSD Flags

BSD has traditionally maintained a series of flags associated with properties of a file as manipulated by `chflags`. These include flags such as the 'system immutable flag' (schg) that prohibit a file from being modified, or the 'archive flag' (arch).

The classical Desktop Database and OS X .DS_Store files

Traditionally the Macintosh stored certain metadata regarding files in the Desktop Database. Under Mac OS X the .DS_Store file associated with a directory contains similar information such as directory background info, the position or order of files and their icons within a directory, and more. When copying a directory you should probably consider copying these.

Lastly there's one Finder-level piece of information stored, disassociated from the file, which brings us to...

Spotlight Metadata

Tiger introduced us to Spotlight and a huge amount of application specific metadata stored that is associated with a file based on its content. This data is indexed by the kernel through 'importers' as a part of normal operations on the file.

While most of this doesn't need to be preserved since it is indexed as needed by the kernel, one attribute exposed to the Finder and stored in .DS_Store is what traditionally was known as the file Comments, called under Tiger the Spotlight Comments, and is displayed through the Finder's Get Info dialog.

How to preserve file metadata

There a few primary methods by which file metadata get preserved:

  • copied natively
  • preserved in special containers or lists
  • preserved by images of the volumes that support them (e.g. .dmg's)
  • split as Apple Doubles

BSD copies files using copyfile() which splits to Apple Doubles as necessary either internally or externally (with varying results)

Note that, currently, copyfile() munges the modified date into the creation date field of its Apple Doubles.

What needs to be copied? What doesn't?

While mostly a philosophic issue, it's probably safe to say that traditional Macintosh attributes such as creator and type are becoming less important since OS X has various mechanisms for associating files with their applications (sometimes at the cost of additional metadata) while other metadata, like creation date, should arguably be maintained. Spotlight metadata looks like a good candidate for general exclusion since the kernel will automagically recreate this for installed application importers on the target machine, but all Mac OS X systems might not have the same importers installed. This may or may not matter. Old Desktop Database files probably aren't an issue either as Classic withers or is mooted by Intel Macs, but .DS_Stores should probably be maintained if you're backing up a volume. Mandatory file forks should always be copied, but while these are Extended Attributes you might consider not copying the other forks that may exist, such as ACLs, depending on if you're backing up or cloning, or, if you're just duplicating the file elsewhere on a volume. It's hard to tell currently if other Extended Attributes need copying since they're rarely seen in action.

What Copies What?

Let's take a look at what metadata, some common tools and their operations preserve. The following is not designed to be a "report card" and it's not my goal to rank or rate a tool for its handling of the data, but sysadmins should be aware of how operations they perform may affect their data.

Coverage of various third-party backup applications is beyond the scope of this article, but it seems they fare rather poorly with regard to not preserving Extended Attributes aside from resource forks. Creation dates are a mixed bag and may vary within the application based on operation (sometimes intentionally). For additional discussion, see http://blog.plasticsfutures.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/

In general, copyfile(), which most all BSD tools rely on, does not perform consistently compared to the operation of the Finder. Creation date is clobbered by the modification date despite that copyfile() attempts to use Apple Doubles 'internally' and externally to foreign file systems and the Apple Double format specifically calls for the creation date. The various tools that use copyfile() then quite often fail to properly maintain internal or synthetic files resulting in lost Extended Attributes and other data. In rare cases the tool may crash, as in the instance of trying to `rsync -aE` a file with both ACLs and an additional non-resource fork, named Extended Attributes.

Apple System Restore (asr) in device mode manages to copy all the tested data, but this is to be expected since it is essentially a complete copy of the device. In file mode, however, asr, Disk Utility, and hdiutil have digressed in their behavior with OS X 10.4.6. Where previously it maintained locks (the Finder "L" attribute and BSD uchg flags), Extended Attributes and ACLs; it no longer preserves them, though this may change in a future update or (although this makes packages and dmgs problematic) it may be the intended behavior.

`dd` fares pretty much as you'd expect. At least philosophically it's understandable.

Editor's Note: To download a copy of the folloing table for reference please visit the MacTech source code ftp and select this month's issue, 22.08 :

http://www.mactech.com/editorial/filearchives.html



Figure 1. The changes in application code structure


Dan Shoop is principal researcher at iWiring (www.iwiring.net) and a Systems and Networks Architect for US Technical Services (ustsvs.com), both of which provide service and support for the Macintosh and other computer systems. He may be reached at <shoop@iwiring.net>.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

BBEdit 11.6.4 - Powerful text and HTML e...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more
beaTunes 4.6.12 - Organize your music co...
beaTunes is a full-featured music player and organizational tool for music collections. How well organized is your music Library? Are your artists always spelled the same way? Any R.E.M. vs REM?... Read more
Tinderbox 7.0.1 - Store and organize you...
Tinderbox is a personal content management assistant. It stores your notes, ideas, and plans. It can help you organize and understand them. And Tinderbox helps you share ideas through Web journals... Read more
FotoMagico 5.4 - Powerful slideshow crea...
FotoMagico lets you create professional slideshows from your photos and music with just a few, simple mouse clicks. It sports a very clean and intuitive yet powerful user interface. High image... Read more
Direct Mail 4.3.9 - Create and send grea...
Direct Mail is an easy-to-use, fully-featured email marketing app purpose-built for OS X. It lets you create and send great looking email campaigns. Start your newsletter by selecting from a gallery... Read more
Tinderbox 7.0.1 - Store and organize you...
Tinderbox is a personal content management assistant. It stores your notes, ideas, and plans. It can help you organize and understand them. And Tinderbox helps you share ideas through Web journals... Read more
Direct Mail 4.3.9 - Create and send grea...
Direct Mail is an easy-to-use, fully-featured email marketing app purpose-built for OS X. It lets you create and send great looking email campaigns. Start your newsletter by selecting from a gallery... Read more
FotoMagico 5.4 - Powerful slideshow crea...
FotoMagico lets you create professional slideshows from your photos and music with just a few, simple mouse clicks. It sports a very clean and intuitive yet powerful user interface. High image... Read more
beaTunes 4.6.12 - Organize your music co...
beaTunes is a full-featured music player and organizational tool for music collections. How well organized is your music Library? Are your artists always spelled the same way? Any R.E.M. vs REM?... Read more
Spotify 1.0.49.125. - Stream music, crea...
Spotify is a streaming music service that gives you on-demand access to millions of songs. Whether you like driving rock, silky R&B, or grandiose classical music, Spotify's massive catalogue puts... Read more

The best sales on the App Store this wee...
The App Store has quite an exciting lineup of discount games this week that range across a variety of genres. It's a great opportunity to catch up on some of the premium games you may have been holding off on -- and some you can even grab for free... | Read more »
The best new games we played this week
Ah, here we are again at the close of another busy week. Don't rest too easy, though. We had a lot of great new releases in mobile games this week, and now you're going to have to spend all weekend playing them. That shouldn't be too much of a... | Read more »
Rollercoaster Tycoon Touch Guide: How to...
| Read more »
Rabbids Crazy Rush Guide: How to unlock...
The Rabbids are back in a new endless running adventure, Rabbids Crazy Rush. It's more ridiculous cartoon craziness as you help the little furballs gather enough fuel (soda) to get to the moon. Sure, it's a silly idea, but everyone has dreams --... | Read more »
Tavern Guardians (Games)
Tavern Guardians 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: Tavern Guardians is a Hack-and-Slash action game played in the style of a match-three. You can experience high pace action... | Read more »
Slay your way to glory in idle RPG Endle...
It’s a golden age for idle games on the mobile market, and those addictive little clickers have a new best friend. South Korean developer Ekkorr released Endless Frontier last year, and players have been idling away the hours in the company of its... | Read more »
Tiny Striker: World Football Guide - How...
| Read more »
Good news everyone! Futurama: Worlds of...
Futurama is finding a new home on mobile in TinyCo and Fox Interactive's new game, Futurama: Worlds of Tomorrow. They're really doing it up, bringing on board Futurama creator Matt Groening along with the original cast and writers. TinyCo wants... | Read more »
MUL.MASH.TAB.BA.GAL.GAL (Games)
MUL.MASH.TAB.BA.GAL.GAL 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: ENDLESS UPGRADES. CONSTANT DANGER. ANCIENT WISDOM. BOUNCY BALLS. Launch Sale, 40% OFF for a very limited time!!! MUL.... | Read more »
Dungeon Rushers (Games)
Dungeon Rushers 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: Dungeon Rushers is a 2D tactical RPG combining dungeon crawler’s gameplay and turn based fights. Manage your team, loot dusty... | Read more »

Price Scanner via MacPrices.net

13-inch 2.0GHz Apple MacBook Pros on sale for...
B&H has the non-Touch Bar 13″ 2.0GHz MacBook Pros in stock today and on sale for $100 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 13″ 2.0GHz MacBook Pro Space Gray (... Read more
15-inch Touch Bar MacBook Pros on sale for up...
B&H Photo has the new 2016 15″ Apple Touch Bar MacBook Pros in stock today and on sale for up to $150 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.7GHz Touch Bar... Read more
12-inch Retina MacBooks on sale for $1150, $1...
B&H has 12″ 1.1GHz Retina MacBooks on sale for $150 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.1GHz Space Gray Retina MacBook: $1149 $150 off MSRP - 12″ 1.1GHz... Read more
Apple restocks refurbished 11-inch MacBook Ai...
Apple has Certified Refurbished 11″ MacBook Airs (the latest models recently discontinued by Apple), available for up to $170 off original MSRP. An Apple one-year warranty is included with each... Read more
Apple Park Opens to Employees in April With T...
Apple has announced that Apple Park, the company’s new 175-acre campus, will be ready for employees to begin occupying in April. The process of moving more than 12,000 people will take over six... Read more
Manhattan Neighbors for Safer Telecommunicati...
A new education and advocacy group focused on cell phone and wireless risks, Manhattan Neighbors for Safer Telecommunications, launched today at http://www.ManhattanNeighbors.org. Manhattan... Read more
Portable Dual DisplayPort Monitor Dock Enable...
IOGEAR has announced the launch of its USB-C Dual DisplayPort Monitor Portable Dock (GUC3CMST). The dock enables users to easily connect two DisplayPort monitors to a USB-C or Thunderbolt 3 laptop to... Read more
13-inch 2.7GHz Retina MacBook Pro on sale for...
Amazon.com has restocked the 13″ 2.7GHz/128GB Retina MacBook Pro (MF839LL/A) for $200 off MSRP including free shipping: - 13″ 2.7GHz/128GB Retina MacBook Pro: $1099 $200 off MSRP This model tends to... Read more
Apple’s New iPad Ads Don’t Address Pro Users’...
Apple launched a new tranche of iPad Pro TV ads last week addressing actual queries and challenges from the Twitterverse, albeit using actors for the visuals. That’s great. As an iPad fan and heavy... Read more
Free Verbum Catholic Bible Study App For iOS
The Verbum mobile app runs on Logos’ powerful Bible software and is an advanced resource for mobile Catholic study. The Verbum app surrounds the Bible with the Tradition. Verbum comes with 15 free... Read more

Jobs Board

*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Retail - Multiple Positions- Chicago...
SalesSpecialist - Retail Customer Service and SalesTransform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Manager *Apple* Systems Administration - Pu...
Req ID 3315BR Position Title Manager, Apple Systems Administration Job Description The Manager of Apple Systems Administration oversees the administration and Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Manager *Apple* Systems Administration - Pu...
Req ID 3315BR Position Title Manager, Apple Systems Administration Job Description The Manager of Apple Systems Administration oversees the administration and Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.