TweetFollow Us on Twitter

Not CurSED, BlesSED!

Volume Number: 22 (2006)
Issue Number: 2
Column Tag: Programming

Mac In The Shell

Not CurSED, BlesSED!

by Edward Marczak

But sometimes, the keys to sed can be disguised.

This month, we're going to follow up on sed, the powerhouse non-interactive editor, introduced in part 1 in December's issue. Hopefully the holidays of that month didn't take you away from really digging in and putting the lessons into practice. If you did take it all in, you've probably found several uses for it already. This month, we'll visit more sed mnemonics and add a few new tricks.

Bob's Yer Uncle

A few reminders about sed regarding redirection and piping. In part 1, we worked on a single file, viewing the results on screen. There are some things I need to clear up before continuing.

Naturally, you can redirect the output from sed to a file. What I didn't mention last month is a tendency that many first-time sed users need to stay away from. Namely, redirecting to the same file that you're working on. Sure, you've tested your script and you're 100% confident that it's going to Do The Right Thing. So, you do this:

sed -f myscript.sed long_text_file.txt > long_text_file.txt

Seems perfectly natural, right? We'll, sorry, you can't do that. You can't redirect to the same file you're altering. This is not an issue with sed, but just the way the shell works. The shell will truncate the destination file before it executes your command. So, while it seems like it might work, it doesn't. You need to write to a temporary file first, and then overwrite the original if you desire to do so. The desired effect is achieved like this:

$ sed -f myscript.sed long_text_file.txt > tmp_file.txt
$ mv -f tmp_file.txt long_text_file.txt

Also, it may not have been clear from the introductory article that sed is happy to work 'on-the-fly' by accepting and pumping out data via a pipe. To get a count of directories in a certain folder, for example, you could do this:

ls -lRF /Users/marczak/Documents | grep "^/" | sed s/:/'\/'/ | sed s/\ /'\\ '/g | wc -l

Pipe-to-pipe-to-pipe-to-pipe! However, a sed master will never use two sed statements where one would do. This can also be written as:

ls -lRF /Users/marczak/Documents | grep "^/" | sed -e s/:/'\/'/ -e s/\ /'\\ '/g | wc -l

In short, pipe away, redirect carefully, and Bob's yer uncle!

Do it Better

While the information in part 1 of this article is more than enough for very powerful manipulations, you still may run into some limitations. That's why part 1 was the introduction: there's still more!

First, I mentioned that I would give the solution the 'swapping Bill and Michael' problem. The answer, of course, is, "it depends." You really have to be familiar with your source material. For now, I'll show the easy, yet most brute-force way of handling our scenario. Remember, the source text is this:

    Bill and Michael went to the store. Bill needed to buy some butter, eggs and flour. He and Michael were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Michael realized that they forgot cake icing.

Just like writing any code, you can swap using a temporary variable (sorry - sed doesn't quite have anything like a bitwise swap!). So, here you go:

sed -e 's/Bill/David/g' -e 's/Michael/Bill/g' -e 's/David/Michael/g' short_story.txt

In our case, we probably really only want the fully qualified "Bill and Michael", so we can actually do just that:

sed 's/Bill and Michael/Michael and Bill/g' short_story.txt

However, you may realize, that this does not take into account line endings. What if our phrase crosses a newline boundary? Ah! That's where multi-line commands come in.

Multi-line commands give sed the ability to look at more than one line in the pattern space. This gives the sed script-crafter the ability to inject a little logic into the flow. Since our story does not split "Bill and Michael" anywhere, let's look at something that does: "buy some...". If we wanted to change all occurrences of "buy some" to "purchase some", even if it spans lines, we need to coax sed into doing so. Again, the brute-force way is simply this:

/buy/ {
N
s/buy *\n*some/purchase some/
}

Here, we look for the address "buy" and when found, run a multi-line next ("N") command. This command reads the next line of input and appends it to the current pattern space - still 'separated' by a new line. Like I said, brute-force, and doesn't really scale well. Also, this has the potential of outputting some really long lines. Of course, elegance is just around the corner. A script that gives us use in more general cases could look like this:

/buy/ {
N
s/ *\n/ /
s/buy some */purchase milk,\
/
}

First, we look for the address "buy", and if we find it, we pull the next line into the pattern space with "N". Then, we can ditch the new line character and replace it with a space. From there we can try to match our patterns. However, even this is a little problematic - just a drop. The example just given works just fine, but let's alter our story and script. Adding a line to the story to make it this:

    Bill and Michael went to the store. Bill needed to buy some butter, eggs and flour. He and Michael were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Michael realized that they forgot cake icing. It is important that they had not forgotten anything for the special day.

And changing the script to this:

/got/ {
N
s/ *\n/ /
s/got home */returned to their abode\
/
}

...will lead to problems. The goal of this script is to change all occurrences of "got home" into "returned to their abode". Run it and see what happens. See the problem? "got" matches "forgotten" on the last line, but makes no substitutions, so the script quits without outputting that line. It's just MIA! What to do? Exempt the last line of the script. Change the "N" to "$!N" - sed recognizes "$" as the last line (not EOL like regexp).

The reality is that you'll find many, many, many examples like this. Depending on your source(s), you may only be able to make a script just so general. This goes back to the rule: test, test, test! You can't test your script enough.

Loose Fit

In addition to the main pattern space that sed matches and manipulates, there is also a hold buffer. The commands are pretty self explanatory: 'x' will eXchange the pattern space with the hold buffer. 'h' will copy the current pattern space into the hold buffer, overwriting what was being held previously. 'H' will do the same, but append to the current hold buffer. 'g' gets the contents of the hold buffer and replaces the pattern space. 'G' gets the contents of the hold buffer and appends its contents to the current pattern space.

Before I get into these commands, please remember that sed certainly is a descendent of the phrase TIMTOWTDI - There is more than one way to do it. Many times, there will be multiple solutions to the particular problem you're trying to overcome. Build up one piece at a time, test, and for goodness sake, document your solution! (Did I mention that sed scripts recognize "#" as a comment?)

So, let's say we want to print the line before and the line after our match, so we can see it in context - like grep's 'A', 'B', and 'C' flags. Here's one way to approach this:

sed -n '
'/Anniversary/' !{
# lines that do not match what we're looking for - save
x
# clear the current pattern buffer with delete
d
}
'/Anniversary/' {
# lines that match
# get the previous line from the hold buffer
x
# print it with p
p
# get the current line back from the hold buffer
x
# print that
p
# get the next line
n
# print it
p
# finally, drop this line into the hold buffer
x
}' short_story.txt

Going back to our short story, this will look for a line containing 'Anniversary', print the line before it, the line itself, and the line following. Note the use of the "-n" switch passed into sed. This switch tells sed to not print all output by default. Otherwise, you'll still see all of your input as it filters through sed. Of course, to make all of this more useful, you could drop this right into a shell script, and use $1 for the pattern - this would give you a generic script that will always perform the equivalent of "grep -C 1 pattern file.txt". Just remember: the way I broke this up over several lines is very bash specific. csh users must use the backslash to tell the interpreter that the line continues on.

Step On

Earlier, I alluded to sed as a programming language by mentioning the classic temp-variable-swap. Well, sed tends to be more full featured than most people realize. You can even implement flow-control! sed features two commands that let you control the logic of your script. 'b' branches to a label. (Reminds me of my favorite Motorola Assembly mnemonic - BRA - BRanch Always). One example, and you'll get it. This script is an alternate to the script just presented that emulates 'grep -C 1':

# find our pattern? jump!
/Anniversary/ b printit
# else hold it
h
# jump to end of script
b
:printit
# get previous line from hold buf
x
# print it
p
# get current line back
x
# print it
p
# get next line
n
# print that
p

First to note is the label. A label starts a line with a colon, and should contain seven characters or less. While most modern implementations allow a label to be any length - and is actually the POSIX spec - there are still some versions of sed that restrict a label to 7 characters max. With 'b', we just branch - make the jump. Another flow-control command is 't', or, test. Test allows us to branch conditionally. The jump only happens if the previous substitution was successful. Another example is in order. Imagine a file that lists a userclass by number, and it should be by name. Additionally, you must process the file differently for each userclass. Here's a mock script that could handle this:

/userclass/{
s/2000/executive/
t excpath
s/1500/management/
t manpath
s/1000/staff/
t stpath
# default action
=
b
:excpath
...
b
:manpath
...
b
:stpath
...
# end of script

Those examples should get you going with flow control in sed. Remember, you can certainly jump to a label ahead of your test and even get into (basic) recursion!

Harmony

There are a bunch of miscellaneous things that I'd like to point out before we wrap up our conversation about sed. Some of these fall into the "you have to know the rules before you can break them" category, so I waited to present them.

All of the examples I've given, and most that you'll see, use a forward slash ("/") as the delimiter. Amazingly, sed lets you use whatever you like. The delimiter is great if you have a very simple replacement, such as sed s/one/two/g. However, if you're replacing file paths, that could get messy. So, use an underscore, if you like: sed 's_/usr/sbin_/usr/bin_'. Easier to read, right?

The "=" (equal sign) mnemonic prints out the current line number. So, you can simply line number any text file:

sed '=' textfile.txt

Or, you can use to just find the lines that the pattern you're looking for lie in:

sed -n '/Bill/ =' story.txt

You've probably picked these two up by now, but I should make them clear. Quoting: you really only need to quote if you have metacharacters, but it's a good habit to get into. This: sed s/Bill/Mike/g is the same as this: sed 's/Bill/ Mike/g'. However, try this: sed s/.*address*\ (astring\)\(\1)$1)/ and sed is just going to cough up electrons. The second thing is the use of the "-e" switch. If you only have one command, you can forgo the "-e". If you have multiple commands, you need to add each of them to the list of editing commands with the "-e" switch.

Lastly, a note about newlines, or, EOL. Somewhat confusingly, you match a newline with the standard regexp '\n'. However, to output a newline, you use a literal newline:

sed 's/Bill/Bill\
/g' story.txt

This will drop a newline after every occurrence of 'Bill' in our sample text.

Cut 'Em Loose Bruce

...and I thought I was going to get to awk this month! Hopefully, this gives you some ideas about sed and its power. I also hope with practice, that you use this power. You really have to see sed as an editor. It just happens to be one that you don't use interactively like vi, emacs or pico. Go forth and edit non-interactively! sed is an indispensable tool for any system administrator's toolchest, and there are plenty of repetitive tasks waiting for you to automate.


Ed Marczak owns and operates Radiotope, a technology consulting company. Despite being around the technology block once or twice, he's thankful that there's always something new to look forward to. Something new at http://www.radiotope.com

 
AAPL
$529.08
Apple Inc.
-2.62
MSFT
$39.63
Microsoft Corpora
-0.36
GOOG
$531.17
Google Inc.
-3.64

MacTech Search:
Community Search:

Software Updates via MacUpdate

TeamViewer 9.0.28116 - Establish remote...
TeamViewer gives you remote control of any computer or Mac over the Internet within seconds, or can be used for online meetings. Find out why more than 200 million users trust TeamViewer! Free for... Read more
Viber 4.1.0 - Send messages and make cal...
Viber lets you send free messages and make free calls to other Viber users, on any device and network, in any country! Viber syncs your contacts, messages and call history with your mobile device,... Read more
Apple iOS 7.1.1 - The latest version of...
The latest version of iOS can be downloaded through iTunes. Apple iOS 7 brings an all-new design and all-new features. Simplicity Simplicity is often equated with minimalism. Yet true simplicity is... Read more
1Password 4.3 - Powerful password manage...
1Password is a password manager that uniquely brings you both security and convenience. It is the only program that provides anti-phishing protection and goes beyond password management by adding Web... Read more
Lens Blur 1.3.0 - True out-of-focus boke...
Let Lens Blur transform your existing photo into true SLR-quality out-of-focus bokeh effect! Everyone needs a gorgeous personalized background for a social profile, blog, Web/UI design, presentation... Read more
VMware Fusion 6.0.3 - Run Windows apps a...
VMware Fusion allows you to create a Virtual Machine on your Mac and run Windows (including Windows 8.1) and Windows software on your Mac. Run your favorite Windows applications alongside Mac... Read more
Tweetbot 1.5.1 - Popular iOS twitter cli...
Tweetbot is a full-featured OS X Twitter client with a lot of personality. Whether it's the meticulously-crafted interface, sounds and animation, or features like multiple timelines and column views... Read more
Mac DVDRipper Pro 4.1.7 - Copy, backup,...
Mac DVDRipper Pro is the DVD backup solution that lets you protect your DVDs from scratches, save your batteries by reading your movies from your hard disk, manage your collection with just a few... Read more
PDFpenPro 6.2 - Advanced PDF toolkit for...
PDFpenPro allows users to edit PDF's easily. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Create... Read more
PDFpen 6.2 - Edit and annotate PDFs with...
PDFpen allows users to easily edit PDF's. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Features... Read more

Latest Forum Discussions

See All

What’s Up with Readdle? – The Verge Exam...
What’s Up with Readdle? | Read more »
Knight Saves Queen Review
Knight Saves Queen Review By Jennifer Allen on April 23rd, 2014 Our Rating: :: PUZZLING MASH UPUniversal App - Designed for iPhone and iPad Mixing up puzzle elements with Chess-based sensibilities is a fun move for this simplistic... | Read more »
Scrap Squad Review
Scrap Squad Review By Jennifer Allen on April 23rd, 2014 Our Rating: :: SIMPLE TRASH COLLECTIONUniversal App - Designed for iPhone and iPad It turns out that dividing up trash can be quite fun in a chaotic kind of way.   | Read more »
Micronytes Director’s Cut Review
Micronytes Director’s Cut Review By Cata Modorcea on April 23rd, 2014 Our Rating: :: DECENTUniversal App - Designed for iPhone and iPad Micronytes isn’t for everyone, but for people who’ve seen the screenshots and haven’t lost... | Read more »
148Apps is Looking for News and Review W...
Looks like it’s that time again. We’ve finished draining the nutritious goo from our victi-I mean we’re looking for more writers! So if you love you some iPods, iPads, and iPhones, and have access to up-to-date iOS hardware (and have a knack for... | Read more »
Wonder Golf Review
Wonder Golf Review By Lee Hamlet on April 23rd, 2014 Our Rating: :: MAD AS A HATTERUniversal App - Designed for iPhone and iPad Help Alice explore a mini-golf version of Wonderland in the quirky Wonder Golf.   | Read more »
Zeebox is Now Beamly TV – Still Offers t...
Zeebox is Now Beamly TV – Still Offers the Same Social Networking TV Fun, Plus a New Look Posted by Rob Rich on April 22nd, 2014 [ permalink ] | Read more »
Bandai Namco Unveils Upcoming Slate of M...
At their recent Global Gamers’ Day event, Bandai Namco was largely focused on their console and PC offerings for the upcoming year. However mobile still had a small presence, with some upcoming titles revealed by the company – though few were in a... | Read more »
The Story Behind Grammar Girl’s Grammar...
Learning is so much more rewarding when it’s also fun. That’s part of the reason why recent educational app, Grammar Pop HD, garnered such a positive review from us last week. The brainchild of Mignon Fogarty, otherwise known as Grammar Girl, we... | Read more »
Pinnacle Studio for iPhone Review
Pinnacle Studio for iPhone Review By Jennifer Allen on April 22nd, 2014 Our Rating: :: SIMPLY POWERFULiPhone App - Designed for the iPhone, compatible with the iPad Video editing while on the move has never been so convenient.   | Read more »

Price Scanner via MacPrices.net

New Mac Pro on sale, save $100 on the 4-Core...
J&R has the new 4-Core Mac Pro in stock today and on sale for $2899 including free shipping plus NY sales tax only. Their price is $100 off MSRP, and it’s the lowest price available for this... Read more
Apple refurbished iMacs available for up to $...
The Apple Store has Apple Certified Refurbished 2013 iMacs available for up to $300 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free. - 27″ 3.4GHz iMac – $1699... Read more
Updated iPad Price Trackers
We’ve updated our iPad Price Tracker and our iPad mini Price Tracker with the latest information on prices and availability from Apple and other resellers. Using a mobile device? We’ve also updated... Read more
Everything You Wanted To Know And Probably Mo...
Macworld UK’s Lou Hattersley takes a look inside Apple’s A7 System On Chip (SoC) , noting that its processor module is much more powerful than other smartphone chipsets. He notes that the A7 was a... Read more
Mavericks Now Runs Half Of Internet Connected...
Computerworld’s Greg Keizer reports that half of all Macs that went online in March were running OS X Mavericks with Version 10.9 accounting for the largest adoption percentage of any individual OS X... Read more
WinZip Mac 3 Simplifies Cloud Sharing And Pri...
WinZip has announced WinZip Mac 3, a new file sharing app that makes it simple to manage, protect and share files across email and the cloud. With direct support for Dropbox and Google Drive,... Read more
2.5GHz Mac mini, Apple refurbished, available...
The Apple Store has Apple Certified Refurbished 2.5GHz Mac minis available for $509, $90 off MSRP. Apple’s one-year warranty is included, and shipping is free. Read more
13-inch MacBook Pro, Apple refurbished, avail...
The Apple Store continues to offer Apple Certified Refurbished 13″ 2.5GHz MacBook Pros (4GB RAM/500GB HD) for $999 which is $200 off MSRP. Apple’s one-year warranty is included, and shipping is free. Read more
15-inch MacBook Pros (refurbished) available...
The Apple Store has Apple Certified Refurbished October 2013 15″ Retina MacBook Pros available starting at $1699, with models up to $400 off MSRP. Apple’s one-year warranty is standard, and shipping... Read more
$100 off cellular iPad minis plus free $20 gi...
Best Buy is discounting cellular iPad minis with Retina Displays by $100 on their online store for a limited time. They’ll also throw in a free $20 Best Buy gift card. Choose free shipping or free... Read more

Jobs Board

*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
Position Opening at *Apple* - Apple (United...
…customers purchase our products, you're the one who helps them get more out of their new Apple technology. Your day in the Apple Store is filled with a range of Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Inc. Research Data Specialist - Appl...
…of Worldwide Market Research & Intelligence. The team is responsible for conducting Apple branded consumer market research. It is also responsible for analyzing data Read more
*Apple* Automotive Parts Department position...
Apple Automotive is one of the fastest growing dealer…and it shows. Consider making the switch to the Apple Automotive Group today! At Apple Automotive, we Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.