TweetFollow Us on Twitter

What's Rattling Around in my sed

Volume Number: 21 (2005)
Issue Number: 12
Column Tag: Programming

Mac In The Shell

What's Rattling Around in my sed

by Edward Marczak

Automating edits.

OS X is Unix" - a statement that has been under a bit of debate since the OS shipped. It does look like a duck and does smell like a duck, but it doesn't really quack like a duck. While many, many projects will, unmodified, compile and run under OS X, there are enough that do not thanks to the acrobatics that OS X engages in underneath it all. That's why we all have to delight in any project that comes pre-installed for us - no compile necessary, and we know it'll be on all OS X boxes that we touch - and runs identically to the Solaris/IRIX/Linux/HP versions. There are two indispensable utilities that fall into this category: sed and awk. These will consume us the next few columns. With sed ("the stream editor") being the more simple of the two, that's where we'll start.

Cuckoo for Cocoa Puffs!

OK - it's not that kind of cereal! sed is a "serial editor" as in it will repeatedly make changes to an input stream in series. I was, not all that long ago, amongst a group of like-minded Mac techs and one person asked how he could take a single file and alter it many times over for specific destinations. I immediately offered sed as a solution. "What?" "sed," I repeated. Blank stares. sed and awk are too useful to not use!

sed takes input, via file name or standard input, processes the file and pumps the results back via standard out. However, instead of making edits interactively, like you might with vi or a word processor, sed allows you to script your edits, almost as if you're creating a macro. This script can then be applied to many (or, any) files. Just like your editor has commands that will insert a line, delete a line and so on, so does sed. Even more impressive, is sed's ability to alter a file, or stream through substitution and regular expressions. And there's the rub. We can 't talk about sed without talking about regular expressions (RE).

Lucky Charms

Regular expressions are fundamental to the Unix landscape. Understanding them makes your computing life better and easier on many levels. Even many GUI editors have support for RE built in. Notable examples are BBEdit and Dreamweaver. There are patterns that you can describe with RE that just can't be done cleanly otherwise. A full discourse on regular expressions is its own article or even book. Fortunately, there's plenty of material out there. The February 2005 issue of MacTech ran, "Matchmaking With Regular Expressions" by Paul Ammann, so if you've been a subscriber, or have that issue, you should start there. O'Reilly has an entire book on RE alone called "Mastering Regular Expressions." If you've already mastered RE, please skip ahead to the next section. Otherwise, I'm ready to present the world's shortest intro and tutorial on regular expressions.

Regular expressions are one of those things that can be a little daunting at first, because the syntax is a little out of the ordinary. However once you dive in, you'll start seeing patterns everywhere that are a great fit. I will admit that I'm going to gloss over some of the finer details regarding RE.

A regular expression is just that: an expression. It's not to be taken literally as it is by itself. This is because most patterns involve metacharacters. We all intrinsically now understand the wildcard metacharacter, "*". If we were to type into our shell, "ls -l *", we'd expect a listing of all files, as we're not literally looking for a file named "*". Similarly, sed is heavily dependant on metacharacters in RE. Also, as an expression, it is something that the interpreter evaluates, much like a mathematical expression. Since understanding how the interpreter evaluates each metacharacter is fundamental to your understanding of the concepts covered in this piece, that's where we'll start.

Non-metacharacters are matched verbatim. If you ask sed (or grep) to match "Cat", it will match a capital "C" immediately followed by a lower case "a", adjacent to a "t". No other permutation will match. Each character is its own RE that matches that single character. To match any single character, use the "." (dot) meta. So, a regexp of "ca." would match "cat" or "cab", but not "rat". To match a wider swath of "any" characters, use the wildcard "*" (asterisk). The asterisk behaves a little differently than you may think. It modifies the previous character, and matches zero or more characters. Again: it modifies the previous character. If you are looking for "car", a RE of "c*r" will not do what you expect. Yes, it will match "car", but will also match any word with "r" in it. Huh? "c*r" says, "match zero or more of the letter c, and then an r." That re-enforces the second point: it will match zero or more characters. To find words that begin with "c" and end with "r", you can use "c.*r", which will match "cr" (if that was a word), "car" and "cur", but also "choir", "czar", "carpetmonger", "calcaneoplantar"...and much more, as we'll see. This says, "match the letter 'c', zero or more of any character, and then an 'r'".

Well, if you actually ran the above on a dictionary file, you'd notice that you'd get a lot more that expected. How does "c.*r" match "buckaroo"? We need to learn two more metacharacters: beginning-of-line "^" (circumflex) and end-of-line "$" (dollar sign). So, to truly get words that begin with "c" and end with "r", you'd use a RE of "^c.*r$". That says, "at the beginning of the line, look for a 'c', then match zero or more characters, and finally match an 'r' at the end of the line."

Cheerios

Let's jump right in with an example, shall we? Probably the sed command (or, mnemonic) used most often is substitute - "s":

sed -e 's/Mike/Michael/' letter.txt

This tells sed to run through the file letter.txt and replace 'Mike' with 'Michael'. Note the use of the forward slash as a delimiter. However, if you read the preceding section, you may have come to expect that it's never quite that simple. The substitute command will only make the substitution for the first occurrence on a line, unless you add the global flag - 'g'. Simply, the command should be:

sed -e 's/Mike/Michael/g' letter.txt

That'll get all of them. You can even make multiple changes in a file:

sed -e 's/Dot/Dorothy/g' -e 's/Chris/Christopher/g' personnel.txt

This gives us an opportunity to talk about how sed applies edits, which is critical to understanding what's going on.

sed works with a line that it brings into its pattern space. It then applies all edits that you've asked for to this line, moves the updated line to stdout (if appropriate), and then brings the next line into the pattern space. To really understand this, a demonstration is in order. The following text is in a file called 'short_story.txt':

    Bill and Michael went to the store. Bill needed to buy some butter, eggs and flour. He and Michael were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Michael realized that they forgot cake icing.

Of course, Michael, being the older brother, feels he should precede Bill in the story. Well, you have sed and that's an easy task, right? Wherever you see 'Michael', change it to 'Bill', and wherever you see 'Bill', change it to 'Michael':

$ sed -e 's/Bill/Michael/g' -e 's/Michael/Bill/g' short_story.txt 

But when you run the command, you get this output:

    Bill and Bill went to the store. Bill needed to buy some butter, eggs and flour. He and Bill were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Bill realized that they forgot cake icing.

What happened? Let's trace:

    1. sed brought the first line into pattern space.

    2. sed found the pattern 'Bill' and substituted 'Michael', making the line:

    "Michael and Michael went to the store. Michael needed to buy"

    3. sed then applied the second edit we asked for, making 'Michael' into 'Bill', resulting in the output we just saw.

Be aware of that interaction. Later on (read: next month's column), I'll show how to deal with that. Using the simple substitution above is great static files, like a checklist:

Hello -=firstname=-!  Welcome to Acme and Associates.  There are some things 
   you'll need to know to get started with our network.  Please keep a record of these items:
Name: -=username=-
Phone: -=phonenum=-
IP address: -=ipaddr=-
Thanks!

You could write a sed script that alters this file appropriately before mailing it out. Of course, substitution gets much more interesting when combined with regular expressions. But first, we need to look at some of the other sed commands.

Rice Krispies

As mentioned, sed is an editor. What good would an editor be if it couldn't add and delete lines? The simpler of the two is delete. Delete erases the entire pattern space, and then continues on to the next line - there's nothing left to match after that.

sed -e '1d' short_story.txt

This prints out our short story, minus the first line - that gets deleted. Of course, delete is even more powerful when combined with regular expressions. How about one that gets rid of bash comments:

sed -e '/^#/d' bashscript.sh

To create more complex sed interactions, we can put all of our commands in a file to make a sed script. By placing your edits into a sed script, you gain the advantage of making your routine reusable, and being able to build it up slowly - these scripts can get complex very quickly, and it's not often that you get it 100% right on the first shot. If you were to save the following into a file named "editscript.sed":

1d
s/Bill/William/g
s/cake/pie/g

...you'd be able to run it against our short story, using the "-f" flag, and see each edit:

$ sed -f editscript.sed short_story.txt 
some butter, eggs and flour.  He and Michael were in a hurry
to bake a pie for their parent's Anniversary.  Once they got
home, William and Michael realized that they forgot pie icing.

First line deleted, "Bill" becomes "William" and "cake" becomes "pie". Using a sed script also brings some other advantages in terms of opening up other commands for our use. First and foremost, as opposed to delete, we can insert. Insert is one of the odd sed commands in that it expects its input to be broken up over more than one line. If we wanted to insert a title at the top of our story, we could use this:

1i\
The Wedding Cake

With insert, you must include a backslash after the command with the text to be inserted on the next line.

Also, within a sed script, we can use functions. Functions in sed? Well, perhaps they're not like traditional functions, but we can opt for a series of edits to take place on a sub-set of the document we're editing:

/^    Bill/ {
        s/Bill/William/
        s/    / /
}

(note here that the second edit is "s slash space space space space slash tab slash". Thank you for your patience.)

This tells sed, "only on lines that start with four spaces immediately followed by 'Bill' will you make the following edits: change Bill to William and then change four spaces to a tab character." Think of all the conditional ways you can program edits with this feature! (OK, did I get too excited over that?)

Again, the power of regular expressions increases sed's value exponentially. This is especially effective when dealing with any kind of mark-up. And to make this even more powerful, we need to introduce addressing. So far, our commands have not had an address, which makes sed apply edits to each line. Another possibility is to supply one address. This can come in the form of a number, or a pattern. If we only wanted to make substitutions on lines that end with a period, we could use something like this:

sed -e '/\.$/s/\./!/g' short_story.txt

(The address to affect is in bold)

This tells sed, "only on lines that end with a period ("\." - we have to escape the period, otherwise it'll match any single character), substitute an exclamation point for any period on the line." OK, it's a completely contrived example, but it should convey the power this brings. I used a not so contrived example the day I finished this article. I was working on a mailing for a client (not a spammer), that needed to retrieve their client e-mail addresses from a Crystal Reports CSV file. CR writes crummy CSV, only enclosing fields in quotes if they have an embedded comma. While that's valid, the software this client was using for mailing didn't like that format too much. sed to the rescue! Here's what I did:

$ sed '/^\"/s/,/ /1' clients.csv > clients2.csv

This told sed, "only on lines that begin with a quote (gotta escape the quote there), substitute a space for the first comma that you find." This output was then redirected to another file, which we had the mailing software (happily) read in.

Don't feel confuSED

Just learning the sed presented in this month's column will bring you incredible power on the command line and when editing files in batch or under the command of a shell script. Believe it or not, there's more! If this was your first exposure to sed, practice, practice, practice! Get a test file (or two or three) and see what sed does when you issue various commands. Next month, I'll cover a little more on the quirky interactions, other commands and more advanced usage.

While I normally say, 'see you next month,' and I mean 'in print,' I really do hope to see everyone next month in person: at MacWorld! I'll be there all week, and presenting the, "From the Chime to the Desktop" session on Wednesday in the IT track conference. Please say hello if you'll be in San Francisco! Of course, I'll still see everyone in print. Until then, let sed rip through some files for you!


Ed Marczak owns and operates Radiotope, a technology consulting company. Ed will be speaking and hanging out at MacWorld SF 2006 all week - hope to see you there! He also deposits tech thoughts on-line at http://www.radiotope.com

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Lyn 1.8.9 - Lightweight image browser an...
Lyn is a fast, lightweight image browser and viewer designed for photographers, graphic artists, and Web designers. Featuring an extremely versatile and aesthetically pleasing interface, it delivers... Read more
Tweetbot 2.5 - Popular Twitter client.
Tweetbot is a full-featured OS X Twitter client with a lot of personality. Whether it's the meticulously-crafted interface, sounds and animation, or features like multiple timelines and column views... Read more
Monolingual 1.7.8 - Remove unwanted OS X...
Monolingual is a program for removing unnecesary language resources from OS X, in order to reclaim several hundred megabytes of disk space. If you use your computer in only one (human) language, you... Read more
Dash 4.0.3 - Instant search and offline...
Dash is an API documentation browser and code snippet manager. Dash helps you store snippets of code, as well as instantly search and browse documentation for almost any API you might use (for a full... Read more
Posterino 3.3.6 - Create posters, collag...
Posterino offers enhanced customization and flexibility including a variety of new, stylish templates featuring grids of identical or odd-sized image boxes. You can customize the size and shape of... Read more
Apple Numbers 4.1.1 - Apple's sprea...
With Apple Numbers, sophisticated spreadsheets are just the start. The whole sheet is your canvas. Just add dramatic interactive charts, tables, and images that paint a revealing picture of your data... Read more
Apple Pages 6.1.1 - Apple's word pr...
Apple Pages is a powerful word processor that gives you everything you need to create documents that look beautiful. And read beautifully. It lets you work seamlessly between Mac and iOS devices, and... Read more
iClock Pro 3.4.9 - Customize your menuba...
iClock Pro is a menu bar replacement clock for Apple's default clock. iClock Pro is an update, total rewrite and improvement to the popular iClock. Have the day, date and time in different fonts and... Read more
Typinator 7.2 - Speedy and reliable text...
Typinator turbo-charges your typing productivity. Type a little. Typinator does the rest. We've all faced projects that require repetitive typing tasks. With Typinator, you can store commonly used... Read more
ExpanDrive 5.4.6 - $49.99
ExpanDrive builds cloud storage in every application, acts just like a USB drive plugged into your Mac. With ExpanDrive, you can securely access any remote file server directly from the Finder or... Read more

Latest Forum Discussions

See All

Minimalist puzzle game, Cuts, free on iO...
If you're looking for a gorgeous puzzle experience on iOS devices, developer Gamebra.in's aesthetically interesting puzzler, Cuts, is discounted to free on the iOS App Store right now. [Read more] | Read more »
Anime tactical RPG, War of Crown, comes...
If you're looking for another tactical RPG fix to go alongside your Fire Emblem Heroes campaigns check out Gamevil's newest, anime-inspired tactics RPG, War of Crown, which comes out tomorrow. [Read more] | Read more »
Fantasy MMORPG MU Origin adds new modes,...
MU Origin, Webzen’s highly popular fantasy MMORPG is getting ready to shake things up for the second time this year, as a new update makes its way to the Google Play and App Store from today. Introducing new systems, modes, and events, the land of... | Read more »
Blizzard is looking to hire a mobile dev...
A new thread on the popular video game rumor forum, NeoGAF, uncovered an interesting job listing over at Blizzard Entertainment. It appears the studio behindStarCraft, World of WarCraft, Hearthstone,andOverwatch is looking to bring on a new hire... | Read more »
Legend of Zelda meets Cooking Mama in ne...
Dungeon Chef is what happens when you mix the RPG elements (and style) of a Legend of Zelda game, with cooking elements. Although, now that The Legend of Zelda: Breath of the Wild also has cookingelements, so maybe the gameplay is not so novel.... | Read more »
ChordFlow (Music)
ChordFlow 1.0.0 Device: iOS Universal Category: Music Price: $6.99, Version: 1.0.0 (iTunes) Description: ChordFlow is a chord sequencer with a unique 4-track polyphonic arpeggiator, extensive chord library, MIDI out and Ableton Link... | Read more »
The Walking Dead: A New Frontier is out...
The newest season of Telltale Games'The Walking Dead is well underway. After the release of the third episode, "Above the Law" about a month ago, episode four, "Thicker Than Water" is hot and ready for more zombies and gut-wrenching emotional... | Read more »
Best games we played this week
Another week, another new wave of mobile games do dive into. We've dug through the list of apps that came out this week to tell you which apps are worth your sweet time. And while there weren't too many games this week, there were some big ones.... | Read more »
Vignettes (Games)
Vignettes 1.0.1 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0.1 (iTunes) Description: Vignettes is a casual but unique exploration game without text or characters, where objects shapeshift as you spin them around... | Read more »
Get Me Outta Here is an 80s retro shoote...
Are you ready to fight some aliens? Because Crescent Moon Games has released the retro shooter Get Me Outta Here on iOS devices today. [Read more] | Read more »

Price Scanner via MacPrices.net

HyperX Ships Pulsefire FPS Gaming Mouse, Winn...
Your reporter is a longtime fan of gaming mice for general purpose coomnputing use, finding them typically superior in comfort and performance. HyperX, a division of Kingston Technology Company, Inc... Read more
Penske Truck Leasing Unveils “Penske Fleet” M...
Penske Truck Leasing has introduced a free mobile app called “Penske Fleet” to benefit its full-service truck leasing and contract maintenance customers. The mobile app enables Penske’s customers to... Read more
DEVONthink To Go 2.1.3 For iOS Adds New Declu...
DEVONtechnologies has updated DEVONthink To Go, its iOS companion to DEVONthink for Mac. Enhancements and bug fixes in Version 2.1.3 include DEVONtechnologies’ new decluttering service for clipping... Read more
Use Apple’s Education discount to save up to...
Purchase a new Mac or iPad using Apple’s Education Store and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free: -... Read more
Apple refurbished Apple Watches available sta...
Apple is now offering Certified Refurbished Series 1 and Series 2 Apple Watches for 14-16% off MSRP, starting at $229. An Apple one-year warranty is included with each watch. Shipping is free: Series... Read more
15-inch Touch Bar MacBook Pros, Apple refurbi...
Apple is offering Certified Refurbished 2016 15″ Touch Bar MacBook Pros for $360 to $420 off original MSRP. An Apple one-year warranty is included with each model, and shipping is free: - 15″ 2.6GHz... Read more
13-inch MacBook Airs on sale for up to $150 o...
Overstock.com has 13″ MacBook Airs on sale for up to $150 off MSRP including free shipping: - 13″ 1.6GHz/128GB MacBook Air (sku MMGF2LL/A): $869.99 $130 off MSRP - 13″ 1.6GHz/256GB MacBook Air (sku... Read more
15-inch Touch Bar MacBook Pros on sale for $1...
B&H Photo has the new 2016 15″ Apple Touch Bar MacBook Pros in stock today and on sale for up to $200 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.7GHz Touch Bar... Read more
15-inch 2.7GHz Touch Bar MacBook Pros on sale...
Amazon has 2016 15″ 2.7GHz Apple Touch Bar MacBook Pros in stock today and on sale for $150-$200 off MSRP. Shipping is free: - 15″ 2.7GHz Touch Bar MacBook Pro Space Gray (sku MLH42LL/A): $2599 $200... Read more
Apple now offering Certified Refurbished 13-i...
Apple is now offering Certified Refurbished 2016 13″ Touch Bar MacBook Pros for $270-$300 off original MSRP. An Apple one-year warranty is included with each model, and shipping is free: - 13″ 2.9GHz... Read more

Jobs Board

*Apple* Engineering Specialist - CSRA (Unite...
Apple Engineering Specialist All times are in Eastern Daylight Time Requisition ID Job Locations US DC Washington DC Posted Date Category Engineering Sciences Read more
*Apple* Mac Computer Technician - GeekHampto...
…complex computer issues over the phone and in person? GeekHampton, Long Island's Apple Premium Service Provider, is looking for you! Come work with our crew Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Solutions Consultant - Apple (United...
# Apple Solutions Consultant Job Number: 56881986 Middletown, NY, New York, United States Posted: Apr. 17, 2017 Weekly Hours: 40.00 **Job Summary** As an Apple Read more
*Apple* Mobile Master - Best Buy (United Sta...
**489879BR** **Job Title:** Apple Mobile Master **Location Number:** 000402-Utica-Store **Job Description:** **What does a Best Buy Apple Mobile Master do?** At Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.