TweetFollow Us on Twitter

Text-based File Formats

Volume Number: 19 (2003)
Issue Number: 2
Column Tag: Section 7

Text-based File Formats

CSV, OML, XML, YAML...

by Rich Morin

BSD and OSX inherit a long tradition (stretching back into the earliest days of Unix) of using text files for data storage. Although there are some exceptions, most control, log, and other system data files are written in ASCII. This makes them easier to inspect, post-process, and even edit.

Apple, whose historic bent has been more toward binary file formats (e.g., the resource fork), seems to have adopted this idea wholeheartedly. In fact, they have gone a bit further, adopting XML (rather than line-oriented files) and Unicode (rather than ASCII). As a result, many OSX files are well structured, language-independent, and quite accessible to both humans and programs.

Many vendors (e.g., Microsoft) are also joining the XML caravan. Assuming that they document both the syntax and semantics of their interchange formats, we could see a dramatic change in possibilities for file interchange.

It is not clear, however, that XML is the Right Answer for all problems. Let's look at some of the alternatives, examining their strengths and weaknesses. Don't expect a comprehensive list; there are zillions of data formats in use. Here, in any event, are some that I would recommend.

CSV

Although CSV stands for "comma-separated values", commas are by no means essential to the idea. In fact, another term for this is "flat file format". Basically, the idea is that each line is a record and that some other delimiter (e.g., colons, commas, white space) is used to separate fields. Quotes or other devices are sometimes used to protect instances of the delimiter in the body of a field. Here are some examples:

/etc/crontab
  15 3 * * * root periodic daily
/etc/gettytab
  a|std.110|110-baud:\
        :np:nd#1:cd#1:uc:sp#110:
/var/log/netinfo.log
  Dec 21 17:59:33 cerberus netinfod ...
An_Excel_File.csv
  1,2,3
  "1,2,3","3,4,5"

Some files get a bit complex, adding syntax to support block structure, comments, line breaks, shell commands, etc. A highly-ornamented CSV file can begin to look like a "little language". Here is a fairly complex format, drawn from /var/named/named.local:

$TTL 86400
@ IN SOA localhost. root.localhost. (
    1997022700 ; Serial
    28800      ; Refresh
    14400      ; Retry
    3600000    ; Expire
    86400 )    ; Minimum
  IN NS  localhost.
1 IN PTR localhost.

Many BSD control files, logs, and reports use white space to delineate fields, making the assumption that the included data will not contain spaces. This was never a safe assumption for path names, but the advent of OSX has made the problem all too real.

Try running "ps -axww" and see how man path names (for both commands and arguments) contain spaces. Then, consider how you would code up a way to determine which spaces are field separators and which are not. Not simple...

Of course, ps has other problems. Run "vi 1 '2 3' 4" in one window and "ps" in another. In the ps output, the COMMAND field looks like "vi 1 2 3 4", dooming any effort to parse it into a command path and distinct command-line arguments. Some sort of quoting convention is desperately needed here.

Blanks in path names make it impossible to parse certain log files, have already broken one Apple installer script, and could well foul up many BSD control files. It's probably too late to get Apple to back off from their use of embedded blanks in system path names, so you can expect to see some problems of this nature coming along...

OML

Although CSV is attractively simple, it may be too simple for your needs. For instance, you may need to support optional data, hierarchical structures, etc. On the other hand, you may not be ready for the formality of XML (Extensible Markup Language) and unwilling to design your own markup language (and parser).

OML (Ostensible Markup Language :-) is a powerful, simple, and convenient solution to this dilemma:

# This is a sample of file-system metadata.
<snap>
  <file>ASCII text</>
  <flags>f,avbstclinmed,,</>
  <lstat>303507,33200,1,1000,20,914,8</>
  <md5>2e1240f444fc3f984186fc5a4fd28eb0</>
  <times>1040087752,230,1740993,10890145</>
</snap>

This looks quite a bit like XML, but there are some small peculiarities. That comment, for instance, isn't legal XML. Nor is "</>" a legal termination for a tag. Is that really CSV syntax in the middle of some fields? Finally, where are the header lines?

Though OML is seemingly designed to give hives to XML purists, it is also designed to work smoothly with existing XML tools. A couple of lines of Perl will strip out the comments and fill out the terminations. A quick pass through an XML parser extracts all of the named fields and attributes. Perl's split() operator, if need be, can break up the CSV data.

Since OML is pretty much a "roll your own" kind of thing, there isn't any real documentation. I'd suggest a look at "Doing it Simpler" (Leigh Dodds; www.xml.com) for some ideas, however.

XML

Despite any appearance to the contrary, I am quite a fan of XML. An enormous amount of meticulous and thoughtful effort (and some rather fancy computer science!) is going into creating "industrial strength" data formats and processing tools. In addition, the W3C (World Wide Web Consortium; www.w3.org) is being very careful to make sure that the official standards are open to all players.

For some projects, you really need to bring in power tools. The host of translators, validators, and other tools that XML provides can make otherwise impossible projects feasible, if not necessarily reasonable. Using XML also increases the chance that someone else's program will be able to parse your data. Given all of that, complaining about a bit of formality seems rather petty.

I won't try to cover XML here; there are shelves of books on the subject, with more coming out on a weekly basis. O'Reilly and Addison-Wesley have the broadest coverage; O'Reilly's XML web site (www.xml.com) is a good place to start your journey...

YAML

CSV has low overhead, is simple to read and edit, and handles lists and arrays well. OML and XML are a bit more bulky, but handle optional data and hierarchies smoothly. XML has an impressive suite of documentation, standards activities, and support software. Sometimes, however, you want low overhead, simplicity, and support for arbitrary data structures.

YAML (YAML Ain't Markup Language) fills this niche quite admirably. The syntax is simple and clean. The basic data structures are sequences (i.e., Perl arrays) and mappings (i.e., Perl hashes). YAML handles lists, arrays, and hierarchies easily; with a bit of extra work, it can handle arbitrary Perl data structures (e.g., cyclic graphs).

Here is the previous example, transliterated into YAML:

# This is a sample of file-system metadata.
snap:
  file:  'ASCII text'
  flags: 'f,avbstclinmed,,'
  lstat: '303507,33200,1,1000,20,914,8'
  md5:   '2e1240f444fc3f984186fc5a4fd28eb0'
  times: '1040087752,230,1740993,10890145'

A more idiomatic rendering, however, would look like:

# This is a sample of file-system metadata.
snap:
  file:  ASCII text
  flags: [ f, avbstclinmed, , ]
  lstat: [ 303507, 33200, 1, 1000, 20, 914, 8 ]
  md5:   2e1240f444fc3f984186fc5a4fd28eb0
  times: [ 1040087752, 230, 1740993, 10890145 ]

Aside from the fact that some spaces have been added after commas and the quotes have been eliminated (some turned into brackets), the second version looks very similar to the first. The resulting data structure is quite different, however; the bracketed lists have been turned into YAML sequences. This means that they don't have to be parsed in a follow-on step. Here is some access code, in Perl:

$file = $yaml{snap}{file};
$uid  = $yaml{snap}{lstat}[3];

YAML has several ways to write textual data. Here are some examples:

  - a simple text item
  - "double-quoted text\n "
  - 'single-quoted text'
- >
    This text
    is freeform.
- |
    This text
    isn't.

Although YAML has nowhere near the amount of documentation that XML has, there are some useful resources to recommend. The YAML web site (www.yaml.org) is the logical place to start; be sure to visit the YAML wiki. I'd also recommend a look at "Look Ma, No Tags" (Kendall Clark Grant; www.xml.com) for an informal introduction.


Rich Morin has been using computers since 1970, Unix since 1983, and Mac-based Unix since 1986 (when he helped Apple create A/UX 1.0). When he isn't writing this column, Rich runs Prime Time Freeware (www.ptf.com), a publisher of books and CD-ROMs for the Free and Open Source software community. Feel free to write to Rich at rdm@ptf.com.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Hopper Disassembler 4.2.5- - Binary disa...
Hopper Disassembler is a binary disassembler, decompiler, and debugger for 32-bit and 64-bit executables. It will let you disassemble any binary you want, and provide you all the information about... Read more
BetterTouchTool 2.261 - Customize Multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
Sketch 44.1 - Design app for UX/UI for i...
Sketch is an innovative and fresh look at vector drawing. Its intentionally minimalist design is based upon a drawing space of unlimited size and layers, free of palettes, panels, menus, windows, and... Read more
BetterTouchTool 2.260 - Customize Multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
Chromium 59.0.3071.115 - Fast and stable...
Chromium is an open-source browser project that aims to build a safer, faster, and more stable way for all Internet users to experience the web. Version 59.0.3071.115: This update has no Flash plug... Read more
SyncTwoFolders 2.2.3 - Syncs two user-sp...
SyncTwoFolders simply synchronizes two folders. It supports synchronization across mounted network drives and it is a possibility to run a simulation showing in a log what will be done. Please visit... Read more
Myriad 4.1 - $79.00
Myriad is, simply put, one of the best audio batch processors. Totally redesigned, it looks beautiful and delivers incredible performance. Let Myriad do the heavy lifting while you get back to doing... Read more
Suitcase Fusion 7 18.2.4 - Font manageme...
Suitcase Fusion 7 is the creative professional's font manager. Every professional font manager should deliver the basics: spectacular previews, powerful search tools, and efficient font organization... Read more
SoftRAID 5.6.1 - High-quality RAID manag...
SoftRAID allows you to create and manage disk arrays to increase performance and reliability. SoftRAID allows the user to create and manage RAID 4 and 5 volumes, RAID 1+0, and RAID 1 (Mirror) and... Read more
BetterTouchTool 2.25 - Customize Multi-T...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more

Latest Forum Discussions

See All

Missile Cards (Games)
Missile Cards 1.0.9 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0.9 (iTunes) Description: "Missile Command meets Solitaire...only with more doomlasers, death, and explosions." | Read more »
Collect mini assassins in 'Assassin...
Assassin's Creed is traveling back in time to the Spanish Inquisition for its latest mobile entry, Assassin's Creed Rebellion. The game is giving the series a look that's a huge departure from its past design, recreating classic characters in a... | Read more »
Animal Crossing is still coming to mobil...
Animal Crossing is still coming to mobile in 2017, according to aWaypointinterview with Nintendo. Announced in 2016, the game was delayed without a defined release window. However, fans of Nintendo's fantasy slice of life game won't have to wait... | Read more »
Ravenscroft 275 Piano (Music)
Ravenscroft 275 Piano 1.0.0 Device: iOS Universal Category: Music Price: $35.99, Version: 1.0.0 (iTunes) Description: Experience the splendor of a Ravenscroft Grand with the most realistic sounding piano ever created for iOS. Launch... | Read more »
This War of Mine gets a new ending and m...
This War of Mine just got a big new update, featuring free DLC that adds a new ending to the game, among other exciting changes. The update is celebrating the game's two-year release anniversary. Apart from the new ending, which will be quite... | Read more »
Summon eight new heroes in Fire Emblem H...
Nintendo keeps coming at us with Fire Emblem Heroes updates, and it doesn't look like that trend is stopping anytime soon. The folks behind the game have just announced the new War of the Clerics Voting Gauntlet, expected to start next Tuesday. [... | Read more »
The best deals on the App Store this wee...
iOS publishers are pulling out all the stops this week -- there's a huge number of seriously great games at discounted prices this week. Let's not waste any time and get right down to business. [Read more] | Read more »
The House of da Vinci (Games)
The House of da Vinci 1.0.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.0 (iTunes) Description: Enter The House of Da Vinci, a new must-try 3D puzzle adventure game. Solve mechanical puzzles, discover hidden... | Read more »
Solve the disappearance of history’s gre...
Blue Brain Games invites you to indulge in an immersive hands-on 3D puzzle adventure in similar vein to The Room series, with its debut release The House of Da Vinci. Set during the historic period of the Italian Renaissance (when Leonardo himself... | Read more »
Age of Rivals (Games)
Age of Rivals 3.3 Device: iOS Universal Category: Games Price: $.99, Version: 3.3 (iTunes) Description: Deep civilization-building strategy in a fast-paced card game! | Read more »

Price Scanner via MacPrices.net

Apple Releases iOS 11 Public Beta; How To Get...
The official release of Apple’s latest mobile operating system iOS 11 is vaguely slated for the fall, but as of June 26, ordinary users can download an iOS 11 public beta. To download the iOS 11... Read more
Extend Life of MacBook Pro Retina 2.0TB With...
MacSales.com/Other World Computing has announced availability of the new OWC 2.0TB Aura Pro Solid State Drive for mid-2012 to early 2013 Apple MacBook Pro with Retina display. One of the fastest... Read more
BBEdit SummerFest 2017 Discount Ends Friday,...
You can get 20% off BBEdit for a limited time in Bare Bones Software’s http://www.eastgate.com/Tinderbox/Specials/SummerFest.html?mc_cid=f2101ca260&mc_eid=[UNIQID]SummerFest 2017 sale and... Read more
Use Apple’s Education discount to save up to...
Purchase a new Mac using Apple’s Education discount, and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free: - 15″ 2... Read more
Clearance 27-inch 3.3GHz 5K iMac available fo...
Amazon clearance 27″ 3.3GHz 5K iMacs (MK482LL/A) available for $1799.90 including free shipping. Their price is $500 off original MSRP, and it’s the lowest price available for this model from any... Read more
13-inch 1.8GHz/256GB MacBook Air on sale for...
B&H Photo has the updated 2017 13″ 1.8GHz/256GB MacBook Air (MQD42LL/A) in stock and on sale for $1129 including free shipping plus NY & NJ tax only. Their price is $70 off MSRP. Read more
27-inch 3.4GHz iMac on sale for $1699, save $...
B&H Photo has the new 2017 27″ 3.4GHz iMac (MNE92LL/A) in stock and on sale for $1699 including free shipping plus NY & NJ sales tax only. Their price is $100 off MSRP. Read more
21-inch 2.3GHz iMac on sale for $1049, save $...
B&H Photo has the new 2017 21″ 2.3GHz iMac (MMQA2LL/A) in stock and on sale for $1049 including free shipping plus NY & NJ tax only. Their price is $50 off MSRP. Read more
ABBYY TextGrabber 6 for iOS Implements Instan...
ABBYY has announced the release of TextGrabber 6.0.0, an important feature update to the company’s productivity app developed for iOS and Android devices. TextGrabber 6.0 now offers Real-Time... Read more
vPhone, First Smartphone That Can’t Be Lost,...
Austin, Texas based Hypori has introduced the vPhone, a virtual smartphone that affords every business user the benefits of separate work and personal phones, conveniently delivered on a single... Read more

Jobs Board

*Apple* News Product Marketing Mgr., Publish...
…organizational consensus on strategy and vision for publisher tools, authoring, and Apple News Format.Carries this strategy and vision across the organization to Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Security Data Analyst - *Apple* Information...
…data sources need to be collected to allow Information Security to better protect Apple employees and customers from a wide range of threats.Act as the subject matter Read more
Lead *Apple* Solutions Consultant - Apple I...
…integrity, and trust.Success Metrics/Key Performance Indicators:Quantitative* Year over Year growth in Apple Product and Beyond the Box sales in the assigned Point of Read more
*Apple* Solutions Consultant till v%u00E5r...
…ethics, integrity, and trust.Success Metrics/Key Performance Indicators:QuantitativeYear over Year growth in Apple Product and Beyond the Box sales in the assigned Point Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.