TweetFollow Us on Twitter

File-Based Dataflow

Volume Number: 19 (2003)
Issue Number: 3
Column Tag: Mac OS X

File-Based Dataflow

building robust systems without explicit file locking

by Rich Morin

Last month, I discussed "File Change Watcher", a Perl daemon that wakes up periodically, checks for file-system changes, copies files, and gathers metadata. To save space, I ignored a critical issue: how can another process know that a metadata file is "ready" to be read?

Race Conditions

This question comes up in the design of any system where one process is writing a file and another is accessing it. In this type of "race condition", the reading process can overtake the writing process, arriving prematurely at the end of the data (and ignoring any data the other process may write thereafter).

Alternatively, if two processes try to write the same file at the same time, assorted damage can ensue. Depending on the circumstances, changes might get lost, output could get intermingled, etc.

One way to deal with the problem is to use a "lock file". By common agreement, if the lock file is in place, the file it "locks" isn't available for access. An inquiring process can test for this by trying to create the lock file. If the process fails, it just waits for a while before trying again.

The reason that this test works is that the kernel won't let two processes open the same file with exclusive access (O_EXCL). So, the first attempt succeeds and all others fail (until the first process closes the file). The technique works quite well, as long as everyone plays nicely; the BSD side of OSX contains a number of lock files (e.g., /var/spool/lock).

In the general case, lock files (or some equivalent technique) are necessary. If, for instance, two users want to edit a file, you really don't want them doing it at the same time. So, some Unix editors (e.g., the BSD version of vi) implement file locking.

Unfortunately, lock files add complexity and room for error. All of the processes have to honor the lock files; what if an existing program doesn't want to play? Also, if the system crashes, the lock file has to be explicitly removed when things start up again. And, for all of this pain, they only solve one part (simultaneous access) of the file-based dataflow problem set.

Atomic Actions

Fortunately, there are a number of alternatives to lock files. Most of them are based on some kind of "atomic action"; that is, something that can only be done completely or not at all. BSD provides several atomic actions as system calls, including chmod(2), chown(2), link(2), open(2), and rename(2).

These same facilities are available from Perl, but its chmod and chown are only atomic for single file nodes. Corresponding shell commands are available, albeit with some caveats:

  • chgrp(1), chmod(1), chown(1) of a single file system node

  • link(1), but only for hard links

  • mv(1), to another name on the same file system

So, although you can't assume that a process will finish writing before some other process starts to read the output file, you can safely rename(2) an existing file (or mv(1) it to another name on the same file system) without worrying about race conditions.

Because any hard link to a file is simply a directory entry that points to a common inode(5), both the data and file system metadata are shared among a set of hard links. This lets a single atomic action (e.g., chmod) change the status of any number of links.

Finally, Cocoa's Application Kit Framework's NSFileWrapper class includes methods such as writeToFile:atomically:updateFilenames:. In short, a wealth of solutions is at hand.

Emitters and Consumers

The system I'm building is based on a data-flow model, similar to Unix pipelines, but it uses files instead of pipes. The method is far from new; mail transfer agents and print spoolers use variations of it in OSX. I am simply generalizing the idea into a framework for creating sets of file- and time-based tasks.

By specifying that only one program will ever write to a file, I can eliminate the issue of simultaneous writing. This means that I only need to prevent processes from opening or removing files prematurely and make sure that every file gets processed to completion.

The rules below allow the safe (i.e., no race conditions) use of files by any number of "emitters" and "consumers", without the need for lock files.

  • Files are created and written by emitters, read and deleted by consumers. No modification of files is allowed, including appending or read/write usage.

  • Files are created under temporary names (on the destination file system), then "published" (e.g., renamed) for use by consumers.

  • A file can only be used by one consumer, which removes it just before exiting.

    Note: This restriction applies only to consumers. Other programs (e.g., more) may read published files at any time. Also, the consumer is not required to read the file, just remove it.

  • If an emitter is creating files for multiple consumers, a separate link must be created for each consumer. All of the links must then be published in a single, atomic operation. One method uses a common directory:

    • Create a temporary directory.

    • For each file with N consumers, make N-1 links in the directory.

    • Rename the temporary file(s) into the directory, as the Nth file link(s).

    • Rename the temporary directory, publishing all of the links.

      Another method, which can only "protect" a single output file, has the advantage that the output links do not have to be in the same directory:

    • Turn off read access (using chmod) on the temporary file.

    • For a file with N consumers, make N-1 links.

    • Rename the temporary file as the Nth link.

    • Make any (and thereby, every) link readable.

Although these rules might complicate the life of programmers, the resulting programs aren't any more complicated. In fact, they tend to be quite simple: discover an input file, process it (writing any output to temporary files); when you're done, publish (e.g., rename) the output files and remove the input file.

Better yet, the file-discovery and -management details can be handed off to a scheduling daemon, allowing most of the "tasks" (working code) to be written as "filters": read from standard input; write to standard output.

A Data Collection System

Let's apply this to a data collection system and see how it plays out. A ps(1)-monitoring task is supposed to run once a minute, writing a report. Another task reads the report, writing a YAML (www.yaml.org) version. Other tasks produce hourly and daily summaries.

This is a fairly complex set of tasks, but it's only a small fraction of the workload for a full-scale operating system monitor. So, the amount of specification for each task should be as simple as possible. Here's a first cut at a configuration file:

# Every minute, collect raw ps(1) data.
{ ps_raw, type: cron, min: every,
  out: 'ps/raw/$time'
}
# Process the raw ps(1) data.
# Create two output links.
{ ps_rare, type: file, patt: 'ps/raw/*',
  out: ['ps/rare/1.$time',
        'ps/rare/2.$time' ]
}
# Every hour, create a summary.
{ ps_hour, type: cron, hour: every,
  out: 'ps/hour/$time'
}
# Every day, create a summary.
{ ps_day, type: cron, day: every,
  out: 'ps/day/$time'
}

This is fairly concise, but there's a lot of repetition. We could "boil it down" by taking advantage of the fact that it describes a tree of processes:

{ ps_raw, type: cron, min: every,
  out: 'ps/raw/$time',
  { ps_rare, type: file, patt: 'ps/raw/*',
    out: ['ps/rare/1.$time',
          'ps/rare/2.$time' ]
    { ps_hour, type: cron, hour: every,
      out: 'ps/hour/$time'
    },
    { ps_day, type: cron, day: every,
      out: 'ps/day/$time'
} } }

But that only kills off two lines (excluding comments), so it's not a huge win. Also, I'm not convinced that it's as easy to read, modify, etc. If we are willing to let the scheduling infrastructure generate file names for us, however, we can get away with "idioms" like:

{ ps_raw,      type: cron, min:  every,
  { ps_rare,   type: file,
    { ps_hour, type: cron, hour: every },
    { ps_day,  type: cron, day:  every }
} }

That brings the overhead back under control. And, if we need to access a file, we can follow standardized naming rules to find it. The output of ps_hour, for instance, would have a name of the form "ps_hour/1.<time>".

So much for theoretical hand-waving and speculative descriptions. Next month, I'll discuss some daemons that can actually make all of this work.


Rich Morin has been using computers since 1970, Unix since 1983, and Mac-based Unix since 1986 (when he helped Apple create A/UX 1.0). When he isn't writing this column, Rich runs Prime Time Freeware (www.ptf.com), a publisher of books and CD-ROMs for the Free and Open Source software community. Feel free to write to Rich at rdm@ptf.com.

 
AAPL
$102.25
Apple Inc.
+0.12
MSFT
$44.88
Microsoft Corpora
+0.01
GOOG
$569.20
Google Inc.
-1.80

MacTech Search:
Community Search:

Software Updates via MacUpdate

Cloud 3.0.0 - File sharing from your men...
Cloud is simple file sharing for the Mac. Drag a file from your Mac to the CloudApp icon in the menubar and we take care of the rest. A link to the file will automatically be copied to your clipboard... Read more
LibreOffice 4.3.1.2 - Free Open Source o...
LibreOffice is an office suite (word processor, spreadsheet, presentations, drawing tool) compatible with other major office suites. The Document Foundation is coordinating development and... Read more
SlingPlayer Plugin 3.3.20.505 - Browser...
SlingPlayer is the screen interface software that works hand-in-hand with the hardware inside the Slingbox to make your TV viewing experience just like that at home. It features an array of... Read more
Get Lyrical 3.8 - Auto-magically adds ly...
Get Lyrical auto-magically add lyrics to songs in iTunes. You can choose either a selection of tracks, or the current track. Or turn on "Active Tagging" to get lyrics for songs as you play them.... Read more
Viber 4.2.2 - Send messages and make cal...
Viber lets you send free messages and make free calls to other Viber users, on any device and network, in any country! Viber syncs your contacts, messages and call history with your mobile device,... Read more
Cocktail 7.6 - General maintenance and o...
Cocktail is a general purpose utility for OS X that lets you clean, repair and optimize your Mac. It is a powerful digital toolset that helps hundreds of thousands of Mac users around the world get... Read more
LaunchBar 6.1 - Powerful file/URL/email...
LaunchBar is an award-winning productivity utility that offers an amazingly intuitive and efficient way to search and access any kind of information stored on your computer or on the Web. It provides... Read more
Maya 2015 - Professional 3D modeling and...
Maya is an award-winning software and powerful, integrated 3D modeling, animation, visual effects, and rendering solution. Because Maya is based on an open architecture, all your work can be scripted... Read more
BBEdit 10.5.12 - Powerful text and HTML...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more
Microsoft Office 2011 14.4.4 - Popular p...
Microsoft Office 2011 helps you create professional documents and presentations. And since Office for Mac 2011 is compatible with Office for Windows, you can work on documents with virtually anyone... Read more

Latest Forum Discussions

See All

Hyperlapse Review
Hyperlapse Review By Jennifer Allen on August 28th, 2014 Our Rating: :: SPEEDY VIDEO SNAPSUniversal App - Designed for iPhone and iPad Want to make a great time-lapse video quickly? Hyperlapse is perfect for that.   | Read more »
Back To Bed Review
Back To Bed Review By Jennifer Allen on August 28th, 2014 Our Rating: :: STYLISH BUT LIMITEDUniversal App - Designed for iPhone and iPad It looks gorgeous, but Back to Bed is actually a fairly simple and uneventful puzzle game.   | Read more »
New Cars, New Locations, and a New Seaso...
New Cars, New Locations, and a New Season in Asphalt 8: Airborne Update Posted by Jessica Fisher on August 28th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
Award Winning Children’s Book Bedtime fo...
Bedtime for Sarah Sullivan is a children’s storybook that emphasizes the importance of going to bed, dreams, and those magical moments right before being tucked in. Now Kelly Paniagua, author of the award-winning children’s book, is planning to... | Read more »
Happy Cube Death Arena Review
Happy Cube Death Arena Review By Jordan Minor on August 28th, 2014 Our Rating: :: CUBEDUniversal App - Designed for iPhone and iPad Happy Cube Death Arena is adorably violent, but very, very shallow.   | Read more »
8bit Doves, the New Game from Icebreaker...
8bit Doves, the New Game from Icebreaker Developers Nitrome, is Now Available – and in Four Colours Posted by Ellis Spice on August 28th, 2014 [ | Read more »
Ace Ferrara and the Dino Menace Review
Ace Ferrara and the Dino Menace Review By Nadia Oxford on August 28th, 2014 Our Rating: :: DINO-MYTEUniversal App - Designed for iPhone and iPad Ace Ferrara and the Dino Menace combines space combat and weird humor into a fun game... | Read more »
Draw Stuff, Win Prizes. Glorkian Warrior...
Draw Stuff, Win Prizes. | Read more »
Lots of iOS Games Have Been Deeply Disco...
Labor Day is fast approaching, and so are the sales. Lots of sales, by the look of it. This list is already pretty sizable, and we haven’t even made it to the weekend yet. Naturally that means you can expect there to be plenty more price drops and... | Read more »
Letter Pix Review
Letter Pix Review By Jennifer Allen on August 28th, 2014 Our Rating: :: FLAWED WORDPLAYUniversal App - Designed for iPhone and iPad Create words to clear the board and guess a photo underlay correctly in this fun but flawed word... | Read more »

Price Scanner via MacPrices.net

Save up to $60 with Apple refurbished iPod to...
The Apple Store has Apple Certified Refurbished 5th generation iPod touches available starting at $149. Apple’s one-year warranty is included with each model, and shipping is free. Many, but not all... Read more
12-Inch MacBook Air Coming in 4Q14 or 2015 –...
Digitimes’ Aaron Lee and Joseph Tsai report that according to Taiwan-based upstream supply chain insiders, Apple plans to launch a thinner MacBook model either at year end 2014 or in 2015, and that... Read more
Sapphire Screen “Most Wanted” iPhone 6 New Fe...
According to the ‘uSell.com iPhone Most Wanted Survey’ — a representative survey of 1,000 U.S. smartphone users conducted by used iPhone marketplace uSell.com — close to half of all smartphone users... Read more
The iPad’s Real Competitive Challenger (Not S...
It’s been my contention for some time that the iPad is suffering from something of an identity crisis, and I suspect that may be a factor in slackening sales this year. Apple can’t seem to decide... Read more
13-inch 2.6GHz/256GB Retina MacBook Pro on sa...
B&H Photo has the 13″ 2.6GHz/256GB Retina MacBook Pro on sale for $1379 including free shipping plus NY sales tax only. Their price is $120 off MSRP. Read more
Life Inventory iOS Apps – Learn to Know Thyse...
James Hollender’s Life Inventory apps s are now on sale with 20% off thru Labor Day, 09/01/2014. This is a great opportunity to get started on that Moral Inventory you’ve been putting off doing for... Read more
Pocket Watch, LLC. Reveals Cloud Server For P...
Beaumont, Texas based Pocket Watch, LLC. has announced the availability of its new ActivePrint Cloud Server Powered by Raspberry Pi. With this small standalone box almost any USB printer or available... Read more
902it Simplifies Area Code Changes For Nova S...
The east coast Canadian provinces of Nova Scotia and Prince Edward Island are phasing in 10 digit telephone dialing, to be fully in place by November, in order to accommodate a second area code to... Read more
Boomerang iPad Stand Mounts Your iPad Anywher...
Boomerang, a Mountable Stand with Multiple Viewing Angles, is now available for iPad Air. Boomerang combines several functions that aim to expand your iPad’s potential in one, elegant product. The... Read more
Retina MacBook Pros available starting at $10...
The Apple Store has Apple Certified Refurbished 13″ and 15″ MacBook Pros available starting at $929. Apple’s one-year warranty is standard, and shipping is free: - 13″ 2.5GHz MacBook Pros (4GB RAM/... Read more

Jobs Board

*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Senior Event Manager, *Apple* Retail Market...
…This senior level position is responsible for leading and imagining the Apple Retail Team's global event strategy. Delivering an overarching brand story; in-store, Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
Project Manager / Business Analyst, WW *Appl...
…a senior project manager / business analyst to work within our Worldwide Apple Fulfillment Operations and the Business Process Re-engineering team. This role will work Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.