TweetFollow Us on Twitter

What's Rattling Around in my sed

Volume Number: 21 (2005)
Issue Number: 12
Column Tag: Programming

Mac In The Shell

What's Rattling Around in my sed

by Edward Marczak

Automating edits.

OS X is Unix" - a statement that has been under a bit of debate since the OS shipped. It does look like a duck and does smell like a duck, but it doesn't really quack like a duck. While many, many projects will, unmodified, compile and run under OS X, there are enough that do not thanks to the acrobatics that OS X engages in underneath it all. That's why we all have to delight in any project that comes pre-installed for us - no compile necessary, and we know it'll be on all OS X boxes that we touch - and runs identically to the Solaris/IRIX/Linux/HP versions. There are two indispensable utilities that fall into this category: sed and awk. These will consume us the next few columns. With sed ("the stream editor") being the more simple of the two, that's where we'll start.

Cuckoo for Cocoa Puffs!

OK - it's not that kind of cereal! sed is a "serial editor" as in it will repeatedly make changes to an input stream in series. I was, not all that long ago, amongst a group of like-minded Mac techs and one person asked how he could take a single file and alter it many times over for specific destinations. I immediately offered sed as a solution. "What?" "sed," I repeated. Blank stares. sed and awk are too useful to not use!

sed takes input, via file name or standard input, processes the file and pumps the results back via standard out. However, instead of making edits interactively, like you might with vi or a word processor, sed allows you to script your edits, almost as if you're creating a macro. This script can then be applied to many (or, any) files. Just like your editor has commands that will insert a line, delete a line and so on, so does sed. Even more impressive, is sed's ability to alter a file, or stream through substitution and regular expressions. And there's the rub. We can 't talk about sed without talking about regular expressions (RE).

Lucky Charms

Regular expressions are fundamental to the Unix landscape. Understanding them makes your computing life better and easier on many levels. Even many GUI editors have support for RE built in. Notable examples are BBEdit and Dreamweaver. There are patterns that you can describe with RE that just can't be done cleanly otherwise. A full discourse on regular expressions is its own article or even book. Fortunately, there's plenty of material out there. The February 2005 issue of MacTech ran, "Matchmaking With Regular Expressions" by Paul Ammann, so if you've been a subscriber, or have that issue, you should start there. O'Reilly has an entire book on RE alone called "Mastering Regular Expressions." If you've already mastered RE, please skip ahead to the next section. Otherwise, I'm ready to present the world's shortest intro and tutorial on regular expressions.

Regular expressions are one of those things that can be a little daunting at first, because the syntax is a little out of the ordinary. However once you dive in, you'll start seeing patterns everywhere that are a great fit. I will admit that I'm going to gloss over some of the finer details regarding RE.

A regular expression is just that: an expression. It's not to be taken literally as it is by itself. This is because most patterns involve metacharacters. We all intrinsically now understand the wildcard metacharacter, "*". If we were to type into our shell, "ls -l *", we'd expect a listing of all files, as we're not literally looking for a file named "*". Similarly, sed is heavily dependant on metacharacters in RE. Also, as an expression, it is something that the interpreter evaluates, much like a mathematical expression. Since understanding how the interpreter evaluates each metacharacter is fundamental to your understanding of the concepts covered in this piece, that's where we'll start.

Non-metacharacters are matched verbatim. If you ask sed (or grep) to match "Cat", it will match a capital "C" immediately followed by a lower case "a", adjacent to a "t". No other permutation will match. Each character is its own RE that matches that single character. To match any single character, use the "." (dot) meta. So, a regexp of "ca." would match "cat" or "cab", but not "rat". To match a wider swath of "any" characters, use the wildcard "*" (asterisk). The asterisk behaves a little differently than you may think. It modifies the previous character, and matches zero or more characters. Again: it modifies the previous character. If you are looking for "car", a RE of "c*r" will not do what you expect. Yes, it will match "car", but will also match any word with "r" in it. Huh? "c*r" says, "match zero or more of the letter c, and then an r." That re-enforces the second point: it will match zero or more characters. To find words that begin with "c" and end with "r", you can use "c.*r", which will match "cr" (if that was a word), "car" and "cur", but also "choir", "czar", "carpetmonger", "calcaneoplantar"...and much more, as we'll see. This says, "match the letter 'c', zero or more of any character, and then an 'r'".

Well, if you actually ran the above on a dictionary file, you'd notice that you'd get a lot more that expected. How does "c.*r" match "buckaroo"? We need to learn two more metacharacters: beginning-of-line "^" (circumflex) and end-of-line "$" (dollar sign). So, to truly get words that begin with "c" and end with "r", you'd use a RE of "^c.*r$". That says, "at the beginning of the line, look for a 'c', then match zero or more characters, and finally match an 'r' at the end of the line."

Cheerios

Let's jump right in with an example, shall we? Probably the sed command (or, mnemonic) used most often is substitute - "s":

sed -e 's/Mike/Michael/' letter.txt

This tells sed to run through the file letter.txt and replace 'Mike' with 'Michael'. Note the use of the forward slash as a delimiter. However, if you read the preceding section, you may have come to expect that it's never quite that simple. The substitute command will only make the substitution for the first occurrence on a line, unless you add the global flag - 'g'. Simply, the command should be:

sed -e 's/Mike/Michael/g' letter.txt

That'll get all of them. You can even make multiple changes in a file:

sed -e 's/Dot/Dorothy/g' -e 's/Chris/Christopher/g' personnel.txt

This gives us an opportunity to talk about how sed applies edits, which is critical to understanding what's going on.

sed works with a line that it brings into its pattern space. It then applies all edits that you've asked for to this line, moves the updated line to stdout (if appropriate), and then brings the next line into the pattern space. To really understand this, a demonstration is in order. The following text is in a file called 'short_story.txt':

    Bill and Michael went to the store. Bill needed to buy some butter, eggs and flour. He and Michael were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Michael realized that they forgot cake icing.

Of course, Michael, being the older brother, feels he should precede Bill in the story. Well, you have sed and that's an easy task, right? Wherever you see 'Michael', change it to 'Bill', and wherever you see 'Bill', change it to 'Michael':

$ sed -e 's/Bill/Michael/g' -e 's/Michael/Bill/g' short_story.txt 

But when you run the command, you get this output:

    Bill and Bill went to the store. Bill needed to buy some butter, eggs and flour. He and Bill were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Bill realized that they forgot cake icing.

What happened? Let's trace:

    1. sed brought the first line into pattern space.

    2. sed found the pattern 'Bill' and substituted 'Michael', making the line:

    "Michael and Michael went to the store. Michael needed to buy"

    3. sed then applied the second edit we asked for, making 'Michael' into 'Bill', resulting in the output we just saw.

Be aware of that interaction. Later on (read: next month's column), I'll show how to deal with that. Using the simple substitution above is great static files, like a checklist:

Hello -=firstname=-!  Welcome to Acme and Associates.  There are some things 
   you'll need to know to get started with our network.  Please keep a record of these items:
Name: -=username=-
Phone: -=phonenum=-
IP address: -=ipaddr=-
Thanks!

You could write a sed script that alters this file appropriately before mailing it out. Of course, substitution gets much more interesting when combined with regular expressions. But first, we need to look at some of the other sed commands.

Rice Krispies

As mentioned, sed is an editor. What good would an editor be if it couldn't add and delete lines? The simpler of the two is delete. Delete erases the entire pattern space, and then continues on to the next line - there's nothing left to match after that.

sed -e '1d' short_story.txt

This prints out our short story, minus the first line - that gets deleted. Of course, delete is even more powerful when combined with regular expressions. How about one that gets rid of bash comments:

sed -e '/^#/d' bashscript.sh

To create more complex sed interactions, we can put all of our commands in a file to make a sed script. By placing your edits into a sed script, you gain the advantage of making your routine reusable, and being able to build it up slowly - these scripts can get complex very quickly, and it's not often that you get it 100% right on the first shot. If you were to save the following into a file named "editscript.sed":

1d
s/Bill/William/g
s/cake/pie/g

...you'd be able to run it against our short story, using the "-f" flag, and see each edit:

$ sed -f editscript.sed short_story.txt 
some butter, eggs and flour.  He and Michael were in a hurry
to bake a pie for their parent's Anniversary.  Once they got
home, William and Michael realized that they forgot pie icing.

First line deleted, "Bill" becomes "William" and "cake" becomes "pie". Using a sed script also brings some other advantages in terms of opening up other commands for our use. First and foremost, as opposed to delete, we can insert. Insert is one of the odd sed commands in that it expects its input to be broken up over more than one line. If we wanted to insert a title at the top of our story, we could use this:

1i\
The Wedding Cake

With insert, you must include a backslash after the command with the text to be inserted on the next line.

Also, within a sed script, we can use functions. Functions in sed? Well, perhaps they're not like traditional functions, but we can opt for a series of edits to take place on a sub-set of the document we're editing:

/^    Bill/ {
        s/Bill/William/
        s/    / /
}

(note here that the second edit is "s slash space space space space slash tab slash". Thank you for your patience.)

This tells sed, "only on lines that start with four spaces immediately followed by 'Bill' will you make the following edits: change Bill to William and then change four spaces to a tab character." Think of all the conditional ways you can program edits with this feature! (OK, did I get too excited over that?)

Again, the power of regular expressions increases sed's value exponentially. This is especially effective when dealing with any kind of mark-up. And to make this even more powerful, we need to introduce addressing. So far, our commands have not had an address, which makes sed apply edits to each line. Another possibility is to supply one address. This can come in the form of a number, or a pattern. If we only wanted to make substitutions on lines that end with a period, we could use something like this:

sed -e '/\.$/s/\./!/g' short_story.txt

(The address to affect is in bold)

This tells sed, "only on lines that end with a period ("\." - we have to escape the period, otherwise it'll match any single character), substitute an exclamation point for any period on the line." OK, it's a completely contrived example, but it should convey the power this brings. I used a not so contrived example the day I finished this article. I was working on a mailing for a client (not a spammer), that needed to retrieve their client e-mail addresses from a Crystal Reports CSV file. CR writes crummy CSV, only enclosing fields in quotes if they have an embedded comma. While that's valid, the software this client was using for mailing didn't like that format too much. sed to the rescue! Here's what I did:

$ sed '/^\"/s/,/ /1' clients.csv > clients2.csv

This told sed, "only on lines that begin with a quote (gotta escape the quote there), substitute a space for the first comma that you find." This output was then redirected to another file, which we had the mailing software (happily) read in.

Don't feel confuSED

Just learning the sed presented in this month's column will bring you incredible power on the command line and when editing files in batch or under the command of a shell script. Believe it or not, there's more! If this was your first exposure to sed, practice, practice, practice! Get a test file (or two or three) and see what sed does when you issue various commands. Next month, I'll cover a little more on the quirky interactions, other commands and more advanced usage.

While I normally say, 'see you next month,' and I mean 'in print,' I really do hope to see everyone next month in person: at MacWorld! I'll be there all week, and presenting the, "From the Chime to the Desktop" session on Wednesday in the IT track conference. Please say hello if you'll be in San Francisco! Of course, I'll still see everyone in print. Until then, let sed rip through some files for you!


Ed Marczak owns and operates Radiotope, a technology consulting company. Ed will be speaking and hanging out at MacWorld SF 2006 all week - hope to see you there! He also deposits tech thoughts on-line at http://www.radiotope.com

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Google Earth 7.1.8.3036 - View and contr...
Google Earth gives you a wealth of imagery and geographic information. Explore destinations like Maui and Paris, or browse content from Wikipedia, National Geographic, and more. Google Earth combines... Read more
QuickBooks 16.1.11.1556 R12 - Financial...
QuickBooks helps you manage your business easily and efficiently. Organize your finances all in one place, track money going in and out of your business, and spot areas where you can save. Built for... Read more
FileZilla 3.24.0 - Fast and reliable FTP...
FileZilla (ported from Windows) is a fast and reliable FTP client and server with lots of useful features and an intuitive interface. Version 3.24.0: New The context menu for remote file search... Read more
Bookends 12.7.8 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more
Duplicate Annihilator 5.8.3 - Find and d...
Duplicate Annihilator takes on the time-consuming task of comparing the images in your iPhoto library using effective algorithms to make sure that no duplicate escapes. Duplicate Annihilator detects... Read more
BusyContacts 1.1.6 - Fast, efficient con...
BusyContacts is a contact manager for OS X that makes creating, finding, and managing contacts faster and more efficient. It brings to contact management the same power, flexibility, and sharing... Read more
MarsEdit 3.7.10 - Quick and convenient b...
MarsEdit is a blog editor for OS X that makes editing your blog like writing email, with spell-checking, drafts, multiple windows, and even AppleScript support. It works with with most blog services... Read more
BusyCal 3.1.4 - Powerful calendar app wi...
BusyCal is an award-winning desktop calendar that combines personal productivity features for individuals with powerful calendar sharing capabilities for families and workgroups. Its unique features... Read more
VirtualBox 5.1.14 - x86 virtualization s...
VirtualBox is a family of powerful x86 virtualization products for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers... Read more
Bookends 12.7.8 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more

Super Mario Run dashes onto Android in M...
Super Mario Run was one of the biggest mobile launches in 2016 before it was met with a lukewarm response by many. While the game itself plays a treat, it's pretty hard to swallow the steep price for the full game. With that said, Android users... | Read more »
WarFriends Beginner's Guide: How to...
Chillingo's new game, WarFriends, is finally available world wide, and so far it's a refreshing change from common mobile game trends. The game's a mix of tower defense, third person shooter, and collectible card game. There's a lot to unpack here... | Read more »
Super Gridland (Entertainment)
Super Gridland 1.0 Device: iOS Universal Category: Entertainment Price: $1.99, Version: 1.0 (iTunes) Description: Match. Build. Survive. "exquisitely tuned" - Rock Paper Shotgun No in-app purches, and no ads! | Read more »
Red's Kingdom (Games)
Red's Kingdom 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: Mad King Mac has kidnapped your father and stolen your golden nut! Solve puzzles and battle goons as you explore and battle your... | Read more »
Turbo League Guide: How to tame the cont...
| Read more »
Fire Emblem: Heroes coming to Google Pla...
Nintendo gave us our first look at Fire Emblem: Heroes, the upcoming mobile Fire Emblem game the company hinted at last year. Revealed at the Fire Emblem Direct event held today, the game will condense the series' tactical RPG combat into bite-... | Read more »
ReSlice (Music)
ReSlice 1.0 Device: iOS Universal Category: Music Price: $9.99, Version: 1.0 (iTunes) Description: Audio Slice Machine Slice your audio samples with ReSlice and create flexible musical atoms which can be triggered by MIDI notes or... | Read more »
Stickman Surfer rides in with the tide t...
Stickson is back and this time he's taken up yet another extreme sport - surfing. Stickman Surfer is out this Thursday on both iOS and Android, so if you've been following the other Stickman adventures, you might be interested in picking this one... | Read more »
Z-Exemplar (Games)
Z-Exemplar 1.4 Device: iOS Universal Category: Games Price: $3.99, Version: 1.4 (iTunes) Description: | Read more »
5 dastardly difficult roguelikes like th...
Edmund McMillen's popular roguelike creation The Binding of Isaac: Rebirth has finally crawled onto mobile devices. It's a grotesque dual-stick shooter that tosses you into an endless, procedurally generated basement as you, the pitiable Isaac,... | Read more »

Price Scanner via MacPrices.net

Twelve South Releases RelaxedLeather Cases fo...
Inspired by the laid-back luxury of burnished leather boots and crafted in rich tones of taupe, herb and marsala, RelaxedLeather cases deliver smart, easy protection for the iPhone 7. Each genuine... Read more
Week’s Best Deal: New 2016 13-inch 2.0GHz Mac...
Amazon has the new 2016 13″ 2.0GHz non-Touch Bar MacBook Pros on sale for a limited time for $225 off MSRP including free shipping: - 13″ 2.0GHz MacBook Pro, Space Gray (MLL42LL/A): $1274.99 $225 off... Read more
Back in stock: Apple refurbished Mac minis fr...
Apple has Certified Refurbished Mac minis available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: - 1.4GHz Mac mini: $419 $80 off MSRP - 2.6GHz Mac... Read more
Apple Ranked ‘Most Intimate Brand’
The top ranked ‘”intimate” brands continued to outperform the S&P and Fortune 500 indices in revenue and profit over the past 10 years, according to MBLM’s Brand Intimacy 2017 Report, the largest... Read more
B-Eng introduces SSD Health Check for Mac OS
Fehraltorf, Switzerland based independant Swiss company- B-Eng has announced the release and immediate availability of SSD Health Check 1.0, the company’s new hard drive utility for Mac OS X. As the... Read more
Apple’s Education discount saves up to $300 o...
Purchase a new Mac or iPad using Apple’s Education Store and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free: -... Read more
4-core 3.7GHz Mac Pro on sale for $2290, save...
Guitar Center has the 3.7GHz 4-core Mac Pro (MD253LL/A) on sale for $2289.97 including free shipping or free local store pickup (if available). Their price is a $710 savings over standard MSRP for... Read more
128GB Apple iPad Air 2, refurbished, availabl...
Apple has Certified Refurbished 128GB iPad Air 2s WiFis available for $419 including free shipping. That’s an $80 savings over standard MSRP for this model. A standard Apple one-year warranty is... Read more
13-inch 2.7GHz Retina MacBook Pro on sale for...
B&H Photo has the 2015 13″ 2.7GHz/128GB Retina Apple MacBook Pro on sale for $100 off MSRP. Shipping is free, and B&H charges NY tax only: - 13″ 2.7GHz/128GB Retina MacBook Pro (MF839LL/A): $... Read more
Laptop Market – Flight To Quality? – The ‘Boo...
Preliminary quarterly PC shipments data released by Gartner Inc. last week reveal an interesting disparity between sales performance of major name PC vendors as opposed to that of less well-known... Read more

Jobs Board

*Apple* Retail - Multiple Positions- Deer Pa...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Technician - nfrastructure (United S...
Let’s Work Together Apple Technician This position is based in Portland, ME Life at nfrastructure At nfrastructure, we understand that our success results from our Read more
*Apple* Mobile Master - Best Buy (United Sta...
**467692BR** **Job Title:** Apple Mobile Master **Location Number:** 000602-Columbia MO-Store **Job Description:** **What does a Best Buy Apple Mobile Master Read more
*Apple* MAC Infrastructure Engineer - InnoCo...
Summary: Responsible for all aspects of Apple Desktop hardware. This includes research, design, test, and deploy technologies being researched by the desktop Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.