TweetFollow Us on Twitter

What's Rattling Around in my sed

Volume Number: 21 (2005)
Issue Number: 12
Column Tag: Programming

Mac In The Shell

What's Rattling Around in my sed

by Edward Marczak

Automating edits.

OS X is Unix" - a statement that has been under a bit of debate since the OS shipped. It does look like a duck and does smell like a duck, but it doesn't really quack like a duck. While many, many projects will, unmodified, compile and run under OS X, there are enough that do not thanks to the acrobatics that OS X engages in underneath it all. That's why we all have to delight in any project that comes pre-installed for us - no compile necessary, and we know it'll be on all OS X boxes that we touch - and runs identically to the Solaris/IRIX/Linux/HP versions. There are two indispensable utilities that fall into this category: sed and awk. These will consume us the next few columns. With sed ("the stream editor") being the more simple of the two, that's where we'll start.

Cuckoo for Cocoa Puffs!

OK - it's not that kind of cereal! sed is a "serial editor" as in it will repeatedly make changes to an input stream in series. I was, not all that long ago, amongst a group of like-minded Mac techs and one person asked how he could take a single file and alter it many times over for specific destinations. I immediately offered sed as a solution. "What?" "sed," I repeated. Blank stares. sed and awk are too useful to not use!

sed takes input, via file name or standard input, processes the file and pumps the results back via standard out. However, instead of making edits interactively, like you might with vi or a word processor, sed allows you to script your edits, almost as if you're creating a macro. This script can then be applied to many (or, any) files. Just like your editor has commands that will insert a line, delete a line and so on, so does sed. Even more impressive, is sed's ability to alter a file, or stream through substitution and regular expressions. And there's the rub. We can 't talk about sed without talking about regular expressions (RE).

Lucky Charms

Regular expressions are fundamental to the Unix landscape. Understanding them makes your computing life better and easier on many levels. Even many GUI editors have support for RE built in. Notable examples are BBEdit and Dreamweaver. There are patterns that you can describe with RE that just can't be done cleanly otherwise. A full discourse on regular expressions is its own article or even book. Fortunately, there's plenty of material out there. The February 2005 issue of MacTech ran, "Matchmaking With Regular Expressions" by Paul Ammann, so if you've been a subscriber, or have that issue, you should start there. O'Reilly has an entire book on RE alone called "Mastering Regular Expressions." If you've already mastered RE, please skip ahead to the next section. Otherwise, I'm ready to present the world's shortest intro and tutorial on regular expressions.

Regular expressions are one of those things that can be a little daunting at first, because the syntax is a little out of the ordinary. However once you dive in, you'll start seeing patterns everywhere that are a great fit. I will admit that I'm going to gloss over some of the finer details regarding RE.

A regular expression is just that: an expression. It's not to be taken literally as it is by itself. This is because most patterns involve metacharacters. We all intrinsically now understand the wildcard metacharacter, "*". If we were to type into our shell, "ls -l *", we'd expect a listing of all files, as we're not literally looking for a file named "*". Similarly, sed is heavily dependant on metacharacters in RE. Also, as an expression, it is something that the interpreter evaluates, much like a mathematical expression. Since understanding how the interpreter evaluates each metacharacter is fundamental to your understanding of the concepts covered in this piece, that's where we'll start.

Non-metacharacters are matched verbatim. If you ask sed (or grep) to match "Cat", it will match a capital "C" immediately followed by a lower case "a", adjacent to a "t". No other permutation will match. Each character is its own RE that matches that single character. To match any single character, use the "." (dot) meta. So, a regexp of "ca." would match "cat" or "cab", but not "rat". To match a wider swath of "any" characters, use the wildcard "*" (asterisk). The asterisk behaves a little differently than you may think. It modifies the previous character, and matches zero or more characters. Again: it modifies the previous character. If you are looking for "car", a RE of "c*r" will not do what you expect. Yes, it will match "car", but will also match any word with "r" in it. Huh? "c*r" says, "match zero or more of the letter c, and then an r." That re-enforces the second point: it will match zero or more characters. To find words that begin with "c" and end with "r", you can use "c.*r", which will match "cr" (if that was a word), "car" and "cur", but also "choir", "czar", "carpetmonger", "calcaneoplantar"...and much more, as we'll see. This says, "match the letter 'c', zero or more of any character, and then an 'r'".

Well, if you actually ran the above on a dictionary file, you'd notice that you'd get a lot more that expected. How does "c.*r" match "buckaroo"? We need to learn two more metacharacters: beginning-of-line "^" (circumflex) and end-of-line "$" (dollar sign). So, to truly get words that begin with "c" and end with "r", you'd use a RE of "^c.*r$". That says, "at the beginning of the line, look for a 'c', then match zero or more characters, and finally match an 'r' at the end of the line."

Cheerios

Let's jump right in with an example, shall we? Probably the sed command (or, mnemonic) used most often is substitute - "s":

sed -e 's/Mike/Michael/' letter.txt

This tells sed to run through the file letter.txt and replace 'Mike' with 'Michael'. Note the use of the forward slash as a delimiter. However, if you read the preceding section, you may have come to expect that it's never quite that simple. The substitute command will only make the substitution for the first occurrence on a line, unless you add the global flag - 'g'. Simply, the command should be:

sed -e 's/Mike/Michael/g' letter.txt

That'll get all of them. You can even make multiple changes in a file:

sed -e 's/Dot/Dorothy/g' -e 's/Chris/Christopher/g' personnel.txt

This gives us an opportunity to talk about how sed applies edits, which is critical to understanding what's going on.

sed works with a line that it brings into its pattern space. It then applies all edits that you've asked for to this line, moves the updated line to stdout (if appropriate), and then brings the next line into the pattern space. To really understand this, a demonstration is in order. The following text is in a file called 'short_story.txt':

    Bill and Michael went to the store. Bill needed to buy some butter, eggs and flour. He and Michael were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Michael realized that they forgot cake icing.

Of course, Michael, being the older brother, feels he should precede Bill in the story. Well, you have sed and that's an easy task, right? Wherever you see 'Michael', change it to 'Bill', and wherever you see 'Bill', change it to 'Michael':

$ sed -e 's/Bill/Michael/g' -e 's/Michael/Bill/g' short_story.txt 

But when you run the command, you get this output:

    Bill and Bill went to the store. Bill needed to buy some butter, eggs and flour. He and Bill were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Bill realized that they forgot cake icing.

What happened? Let's trace:

    1. sed brought the first line into pattern space.

    2. sed found the pattern 'Bill' and substituted 'Michael', making the line:

    "Michael and Michael went to the store. Michael needed to buy"

    3. sed then applied the second edit we asked for, making 'Michael' into 'Bill', resulting in the output we just saw.

Be aware of that interaction. Later on (read: next month's column), I'll show how to deal with that. Using the simple substitution above is great static files, like a checklist:

Hello -=firstname=-!  Welcome to Acme and Associates.  There are some things 
   you'll need to know to get started with our network.  Please keep a record of these items:
Name: -=username=-
Phone: -=phonenum=-
IP address: -=ipaddr=-
Thanks!

You could write a sed script that alters this file appropriately before mailing it out. Of course, substitution gets much more interesting when combined with regular expressions. But first, we need to look at some of the other sed commands.

Rice Krispies

As mentioned, sed is an editor. What good would an editor be if it couldn't add and delete lines? The simpler of the two is delete. Delete erases the entire pattern space, and then continues on to the next line - there's nothing left to match after that.

sed -e '1d' short_story.txt

This prints out our short story, minus the first line - that gets deleted. Of course, delete is even more powerful when combined with regular expressions. How about one that gets rid of bash comments:

sed -e '/^#/d' bashscript.sh

To create more complex sed interactions, we can put all of our commands in a file to make a sed script. By placing your edits into a sed script, you gain the advantage of making your routine reusable, and being able to build it up slowly - these scripts can get complex very quickly, and it's not often that you get it 100% right on the first shot. If you were to save the following into a file named "editscript.sed":

1d
s/Bill/William/g
s/cake/pie/g

...you'd be able to run it against our short story, using the "-f" flag, and see each edit:

$ sed -f editscript.sed short_story.txt 
some butter, eggs and flour.  He and Michael were in a hurry
to bake a pie for their parent's Anniversary.  Once they got
home, William and Michael realized that they forgot pie icing.

First line deleted, "Bill" becomes "William" and "cake" becomes "pie". Using a sed script also brings some other advantages in terms of opening up other commands for our use. First and foremost, as opposed to delete, we can insert. Insert is one of the odd sed commands in that it expects its input to be broken up over more than one line. If we wanted to insert a title at the top of our story, we could use this:

1i\
The Wedding Cake

With insert, you must include a backslash after the command with the text to be inserted on the next line.

Also, within a sed script, we can use functions. Functions in sed? Well, perhaps they're not like traditional functions, but we can opt for a series of edits to take place on a sub-set of the document we're editing:

/^    Bill/ {
        s/Bill/William/
        s/    / /
}

(note here that the second edit is "s slash space space space space slash tab slash". Thank you for your patience.)

This tells sed, "only on lines that start with four spaces immediately followed by 'Bill' will you make the following edits: change Bill to William and then change four spaces to a tab character." Think of all the conditional ways you can program edits with this feature! (OK, did I get too excited over that?)

Again, the power of regular expressions increases sed's value exponentially. This is especially effective when dealing with any kind of mark-up. And to make this even more powerful, we need to introduce addressing. So far, our commands have not had an address, which makes sed apply edits to each line. Another possibility is to supply one address. This can come in the form of a number, or a pattern. If we only wanted to make substitutions on lines that end with a period, we could use something like this:

sed -e '/\.$/s/\./!/g' short_story.txt

(The address to affect is in bold)

This tells sed, "only on lines that end with a period ("\." - we have to escape the period, otherwise it'll match any single character), substitute an exclamation point for any period on the line." OK, it's a completely contrived example, but it should convey the power this brings. I used a not so contrived example the day I finished this article. I was working on a mailing for a client (not a spammer), that needed to retrieve their client e-mail addresses from a Crystal Reports CSV file. CR writes crummy CSV, only enclosing fields in quotes if they have an embedded comma. While that's valid, the software this client was using for mailing didn't like that format too much. sed to the rescue! Here's what I did:

$ sed '/^\"/s/,/ /1' clients.csv > clients2.csv

This told sed, "only on lines that begin with a quote (gotta escape the quote there), substitute a space for the first comma that you find." This output was then redirected to another file, which we had the mailing software (happily) read in.

Don't feel confuSED

Just learning the sed presented in this month's column will bring you incredible power on the command line and when editing files in batch or under the command of a shell script. Believe it or not, there's more! If this was your first exposure to sed, practice, practice, practice! Get a test file (or two or three) and see what sed does when you issue various commands. Next month, I'll cover a little more on the quirky interactions, other commands and more advanced usage.

While I normally say, 'see you next month,' and I mean 'in print,' I really do hope to see everyone next month in person: at MacWorld! I'll be there all week, and presenting the, "From the Chime to the Desktop" session on Wednesday in the IT track conference. Please say hello if you'll be in San Francisco! Of course, I'll still see everyone in print. Until then, let sed rip through some files for you!


Ed Marczak owns and operates Radiotope, a technology consulting company. Ed will be speaking and hanging out at MacWorld SF 2006 all week - hope to see you there! He also deposits tech thoughts on-line at http://www.radiotope.com

 
AAPL
$524.94
Apple Inc.
+5.93
MSFT
$40.01
Microsoft Corpora
-0.39
GOOG
$536.10
Google Inc.
-20.44

MacTech Search:
Community Search:

Software Updates via MacUpdate

Tweetbot 1.5.1 - Popular iOS twitter cli...
Tweetbot is a full-featured OS X Twitter client with a lot of personality. Whether it's the meticulously-crafted interface, sounds and animation, or features like multiple timelines and column views... Read more
Mac DVDRipper Pro 4.1.7 - Copy, backup,...
Mac DVDRipper Pro is the DVD backup solution that lets you protect your DVDs from scratches, save your batteries by reading your movies from your hard disk, manage your collection with just a few... Read more
PDFpenPro 6.2 - Advanced PDF toolkit for...
PDFpenPro allows users to edit PDF's easily. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Create... Read more
PDFpen 6.2 - Edit and annotate PDFs with...
PDFpen allows users to easily edit PDF's. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Features... Read more
Monolingual 1.5.9 - Remove unwanted OS X...
Monolingual is a program for removing unnecesary language resources from OS X, in order to reclaim several hundred megabytes of disk space. It requires a 64-bit capable Intel-based Mac and at least... Read more
Maya 2015 - Professional 3D modeling and...
Maya is an award-winning software and powerful, integrated 3D modeling, animation, visual effects, and rendering solution. Because Maya is based on an open architecture, all your work can be scripted... Read more
Starcraft II: Wings of Liberty 1.1.1.180...
Download the patch by launching the Starcraft II game and downloading it through the Battle.net connection within the app. Starcraft II: Wings of Liberty is a strategy game played in real-time. You... Read more
Sibelius 7.5.0 - Music notation solution...
Sibelius is the world's best-selling music notation software for Mac. It is as intuitive to use as a pen, yet so powerful that it does most things in less than the blink of an eye. The demo includes... Read more
Typinator 5.9 - Speedy and reliable text...
Typinator turbo-charges your typing productivity. Type a little. Typinator does the rest. We've all faced projects that require repetitive typing tasks. With Typinator, you can store commonly used... Read more
MYStuff Pro 2.0.16 - Create inventories...
MYStuff Pro is the most flexible way to create detail-rich inventories for your home or small business. Add items to MYStuff by dragging and dropping existing information, uploading new images, or... Read more

Latest Forum Discussions

See All

Have a Special Dead Trigger 2 Easter Bas...
Have a Special Dead Trigger 2 Easter Basket Full of Goodies, Courtesy of Madfinger Games Posted by Rob Rich on April 18th, 2014 [ permalink ] Dead Trigger 2 | Read more »
Zynga Launches Brand New Farmville Exper...
Zynga Launches Brand New Farmville Experience with Farmville 2: Country Escape Posted by Tre Lawrence on April 18th, 2014 [ permalink ] | Read more »
David. Review
David. Review By Cata Modorcea on April 18th, 2014 Our Rating: :: MINIMALISTIC IN A DIFFERENT WAYUniversal App - Designed for iPhone and iPad David is a minimalistic game wrapped inside of a soothing atmosphere in which the hero... | Read more »
Eyefi Unveils New Eyefi Cloud Service Th...
Eyefi Unveils New Eyefi Cloud Service That Allows Users to Share Media Across Personal Devices Posted by Tre Lawrence on April 18th, 2014 [ permalink ] | Read more »
Tales from the Dragon Mountain: The Lair...
Tales from the Dragon Mountain: The Lair Review By Jennifer Allen on April 18th, 2014 Our Rating: :: STEADY ADVENTURINGiPad Only App - Designed for the iPad Treading a safe path, Tales from the Dragon Mountain: The Lair is a... | Read more »
Yahoo Updates Flickr App with Advanced E...
Yahoo Updates Flickr App with Advanced Editing Features and More Posted by Tre Lawrence on April 18th, 2014 [ permalink ] | Read more »
My Incredible Body - A Kid's App to...
My Incredible Body - A Kid's App to Learn about the Human Body 1.1.00 Device: iOS Universal Category: Education Price: $2.99, Version: 1.1.00 (iTunes) Description: Wouldn’t it be cool to look inside yourself and see what was going on... | Read more »
Trials Frontier Review
Trials Frontier Review By Carter Dotson on April 18th, 2014 Our Rating: :: A ROUGH LANDINGUniversal App - Designed for iPhone and iPad Trials Frontier finally brings the famed stunt racing franchise to mobile, but how much does its... | Read more »
Evernote Business Notebook by Moleskin I...
Evernote Business Notebook by Moleskin Introduced – Support Available in Evernote for iOS Posted by Tre Lawrence on April 18th, 2014 [ permalink ] | Read more »
Sparkle Unleashed Review
Sparkle Unleashed Review By Jennifer Allen on April 18th, 2014 Our Rating: :: CLASSY MARBLE FLINGINGUniversal App - Designed for iPhone and iPad It’s a concept we’ve seen before, but Sparkle Unleashed is a solidly enjoyable orb... | Read more »

Price Scanner via MacPrices.net

iMacs on sale for up to $160 off MSRP this we...
Best Buy has iMacs on sale for up to $160 off MSRP for a limited time. Choose free home shipping or free instant local store pickup (if available). Prices are valid for online orders only, in-store... Read more
iPad Airs on sale this weekend for up to $100...
Best Buy has WiFi iPad Airs on sale for $50 off MSRP and WiFi + Cellular iPad Airs on sale for $100 off MSRP on their online store for a limited time, with prices now starting at $449. Choose free... Read more
Apple restocks refurbished Mac minis starting...
The Apple Store has restocked Apple Certified Refurbished Mac minis for up to $150 off the cost of new models. Apple’s one-year warranty is included with each mini, and shipping is free: - 2.5GHz Mac... Read more
Hyundai Brings Apple CarPlay To The 2015 Sona...
Hyundai Motor America has announced it will bring Apple CarPlay functionality to the 2015 Sonata. CarPlay is pitched as a smarter, safer and easier way to use iPhone in the car and gives iPhone users... Read more
Updated iPads Coming Sooner Than We Had Thoug...
MacRumors, cites KGI securities analyst Ming Chi Kuo, well-respected as an Apple product prognisticator, saying that Apple will introduce an upgraded iPad Air and iPad mini in 2014/Q3, meaning the... Read more
Toshiba Unveils New High And Low End Laptop M...
Toshiba has announced new laptop models covering both the high-end and low-end of the notebook computer spectrum. Toshiba 4K Ultra HD Laptop Toshiba’s new Satellite P55t features one of the world’s... Read more
Save up to $270 with Apple refurbished 13-inc...
The Apple Store has Apple Certified Refurbished October 2013 13″ Retina MacBook Pros available starting at $1099, with models up to $270 off MSRP. Apple’s one-year warranty is standard, and shipping... Read more
Apple now offering refurbished iPad mini with...
The Apple Store has Certified Refurbished 2nd generation iPad minis with Retina Displays now available starting at $339. Apple’s one-year warranty is included with each model, and shipping is free.... Read more
Microsoft Blinks – Drops Microsoft Office 365...
Microsoft has dropped the annual subscription fee for Microsoft Office 365 Personal – which is needed in order to create and edit documents in Microsoft Office for iPad. However, Apple’s iOS and OS X... Read more
New AVG Vault Apps for iOS and Android Help K...
AVG Technologies N.V. an online security company for 177 million active users, has announced the launch of its latest mobile application, AVG Vault. The free app introduces an innovative user... Read more

Jobs Board

*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Retail - Manager - Holyoke - Apple I...
Job Summary Keeping an Apple Store thriving requires a diverse set of leadership skills, and as a Manager, you’re a master of them all. In the store’s fast-paced, Read more
*Apple* Retail - Manager - Apple (United Sta...
Job SummaryKeeping an Apple Store thriving requires a diverse set of leadership skills, and as a Manager, you're a master of them all. In the store's fast-paced, dynamic Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Retail - Market Leader - Cincinnati...
…challenges of developing individuals, building teams, and affecting growth across Apple Stores. You demonstrate successful leadership ability - focusing on excellence Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.