TweetFollow Us on Twitter

December 96 - MPW Tips and Tricks: Automated Editing With StreamEdit

MPW Tips and Tricks:
Automated Editing With StreamEdit

Tim Maroney

In this column in Issue 26 of develop, I showed you a wide range of scriptable editing commands available from the MPW Shell. This time I'll discuss a single tool that provides a powerful self-contained text-editing scripting language, StreamEdit.

Why would you want to use StreamEdit instead of the other text-editing features of the MPW Shell?

  • Performance -- A StreamEdit script is faster than an MPW script containing various Replace and Find commands.

  • Self-containment -- Because StreamEdit is a self-contained tool, you can run it from within ToolServer, unlike the scriptable editing commands discussed in Issue 26, which are available only in the MPW Shell itself. This means you can use StreamEdit to create lightweight drag-and-drop grinder AppleScript scripts that send StreamEdit commands to ToolServer.

  • Consistency -- Keeping all your editing in a single scripting language confers the elusive mystical boon of code consistency, making your system easier to maintain and modify in the future.


GETTING TO KNOW YOU

StreamEdit is based very closely on the hoary UNIX tool named sed. If you already know sed, much of this will be familiar, but StreamEdit isn't directly compatible with sed scripts. StreamEdit implements a pattern-matching language. Every time a particular pattern is matched, a sequence of commands will be executed. As in most pattern-matching languages, StreamEdit's scripts are lists of pattern/command pairs, with the pattern coming before the command. The input file or files are read through the script interpreter, which searches for instances of the patterns and executes the corresponding commands. Anything that doesn't match a pattern is passed through unchanged.

StreamEdit scans one line at a time through the input, matching its current line to every pattern in its script. After processing each line, it writes out the modified line. The result is a concatenation of three internal buffers: the insert buffer, then the edit buffer, and finally the append buffer. The edit buffer gets filled with the current line, while the other buffers are empty at the start. The Insert and Append commands place text in the insert and append buffers, allowing you to add text to the beginning and end of the output line. The Change, Delete, and Replace commands modify the contents of the edit buffer.


SHARING ADDRESSES

As usual, MPW uses words in ways previously unknown in human speech. In StreamEdit, patterns are referred to as "addresses." There are two kinds of addresses: line numbers and regular expressions. Line numbers ought to be self-explanatory, but it may help to note that the numbers must be Arabic numerals rather than Roman, and must be in base 10 rather than the hexadecimal or sexagesimal number systems. There are three special line numbers:
  • the bullet symbol (·, Option-8), meaning the point before the first line (enabling you to add a line before the first line, for example)

  • the infinity symbol (Option-5), meaning the point after the last line

  • dollar sign ($), meaning the last line

    The keyboard shortcuts, as always in this column, are for American QWERTY keyboards; if you've got some other type of keyboard, you're on your own.*
Regular expressions are expressions that manage their diets sensibly. They can be used for searching, and were explained in detail in Issue 26. In StreamEdit addresses, though, regular expressions find the entire line containing the pattern, rather than just the pattern. Regular expressions are denoted by slashes. Only forward slashes are used (StreamEdit doesn't have a backward search mode, having been frightened at an early age by the legends of Eurydice and Lot's wife). Three new constructs have been added to regular expressions in StreamEdit:
  • ç (Option-C), which indicates a case-sensitive search

  • // (two slashes), which means the last regular expression that was matched

  • <=variable>= (a variable name embedded in inequality operators, here overloaded as a special kind of angle brackets, and typed as Option-comma and Option-period), which means the text of an expanded StreamEdit variable, treated as literal text to be matched rather than as a regular expression
StreamEdit has variables that can be set with the Set command (more on this later) or from the command line using the -set variable [=value] option.

You can form more complex addresses using a few operators. The Boolean and, or, and not operators are the same as in C (&&, | |, and !, respectively). Parentheses can be used for grouping within addresses. The comma operator matches the range of lines specified; for example, 3,5 matches lines 3 through 5. A range address matches each of the lines in the range, if any. It can be thought of as matching more than once: it fires off the accompanying command on the first line matched, the last one matched, and all lines in between. If the termination condition is never met, the address continues to match until the end of input. This could happen if you specify a range of lines ending at line 15, for instance, and there are only ten lines in the file, or if your range termination condition is a regular expression that doesn't appear anywhere in the input.


TAKING ACTION

Matching patterns is very nice, but what do you do once you match them? Statements in StreamEdit attach actions to patterns. An action consists of one or more commands, separated by semicolons or by the end of a line. There's no begin or end bracketing as in Pascal or C. Addresses and commands are syntactically distinct, so the script interpreter can figure out where the list of commands for a pattern ends and the next pattern begins.

Editing commands

  • Insert text [-n] -- Adds the specified text to the start of the line by putting it in the insert buffer. The -n option (in this command and in Append and Change) prevents adding a newline character when the line is written out.

  • Append text [-n] -- Adds the specified text to the end of the line by putting it in the append buffer.

  • Change text [-n] -- Changes the line to the specified text by replacing the contents of the edit buffer.

  • Delete -- Clears the edit buffer.

  • Replace [-c count] /pattern/ text -- Replaces the pattern with the specified text. This is the second part of a two-step matching process: first the address matches a line, then Replace searches in the edit buffer and replaces the pattern. The count argument indicates the maximum number of times to perform the replacement in the line. It can be a positive integer or infinity. The default count is 1.

Control commands

  • Exit [status] -- Stops StreamEdit with the given error status. The default is 0, which means execution completed with no errors. Any nonzero error status indicates a problem, and unless the built-in MPW variable Exit is set to something other than 0, this will stop execution of the script (if any) from which the StreamEdit command was executed.

  • Next -- Somewhat like the C keyword continue. When a Next command is executed, all pending changes are written out and no more addresses are matched against the current line; that is, StreamEdit immediately goes on to the next line without matching the rest of the rules against the current edit buffer.

  • Set variable text [-i | -a] -- Much like the MPW Shell Set command. The variable is set to the specified text. The -i and -a options allow text to be added to any existing setting of the variable at the start or the end, respectively.

Output commands

  • Print [text] [-appendto | -to file] -- Writes output to a specified file. If text is empty, the current line is printed without modification. The -appendto and -to options write at the end of the file or overwrite the file, respectively. If no file is specified, standard output is used. If the filename is empty, nothing gets printed.

  • Option AutoDelete -- Deletes all input lines, leaving only output from Next and Print commands. You can get the same effect by specifying the -d option on the StreamEdit command line or by including this in the script:
    /~/    Delete
    
The text arguments to these commands are usually literal text, denoted by single or double quotes. There are a few other forms as well:
  • An unquoted variable name can be used, in which case the variable is expanded; no brackets need be (or even may be) supplied.

  • A period means the current input line up to but not including the newline character at the end.

  • As discussed in Issue 26, you can use ® (Option-R) followed by a digit to mean the expression with that number matched in the pattern.

  • You can read text from a file with -from filename, which reads the next line of text from the specified file. The filename is usually literal text, but it could also be a variable, the current input line (denoted by a period), or a ® expression.


A HYPOTHETICAL EXAMPLE

Let's say you're the director of corporate communications at a major computer maker and, without any warning except for inventory backlogs larger than the gross national products of many developing countries, you experience a sudden transition in chief executive officers, corporate policy, and product line. Your quarterly report (10-Q) is due in the SEC's EDGAR database tomorrow. Fortunately the SEC requires the cutting-edge ASCII format for its filings, and you realize that you can automate 90% of the tedious changes with a single StreamEdit script.
# Change nickname of CEO
/Diesel/
Replace // 'Flyboy'

# Change corporate policy
/1,$/
Replace /capture market share/ 'survive'

# Remove lines referring to obsolete products
/PowerTalk/ || /eWorld/
Delete

# Change developer relations strategy
/third-party developers/
Replace /evangelize/ 'listen to'

# Mark lines referring to old schedules with a
# distinctive string at the start of the line
# for manual editing later
/1996/
Insert 'WHOOPS: '

# Add new final line of report
[[infinity]]
Append 'May God have mercy on our souls.'


CONTROL YOURSELF

StreamEdit is almost too powerful. People have used it for everything, including pretty-printing source code, converting files to HTML, and postprocessing object files for dynamic linking tools. If you use it for finding incriminating passages in coworkers' e-mail, karma may get you, but the limitations of the tool won't. Use your powers for good rather than evil, and a grateful world will thank you.

TIM MARONEY has appeared professionally in newspapers, magazines, compact discs, videotape, and of course, computer software. Tim is a technical lead in human interface software at Apple and is editing a series of books for a horror publisher. His skin burns easily in the sun and tans in the moon. He uses white T-shirts only for house painting and car repair.*

Thanks to Arno Gourdol, Alex McKale, and Robert Ulrich for reviewing this column.*

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Duplicate Annihilator 5.7.5 - Find and d...
Duplicate Annihilator takes on the time-consuming task of comparing the images in your iPhoto library using effective algorithms to make sure that no duplicate escapes. Duplicate Annihilator... Read more
BusyContacts 1.0.2 - Fast, efficient con...
BusyContacts is a contact manager for OS X that makes creating, finding, and managing contacts faster and more efficient. It brings to contact management the same power, flexibility, and sharing... Read more
Capture One Pro 8.2.0.82 - RAW workflow...
Capture One Pro 8 is a professional RAW converter offering you ultimate image quality with accurate colors and incredible detail from more than 300 high-end cameras -- straight out of the box. It... Read more
Backblaze 4.0.0.872 - Online backup serv...
Backblaze is an online backup service designed from the ground-up for the Mac.With unlimited storage available for $5 per month, as well as a free 15-day trial, peace of mind is within reach with... Read more
Little Snitch 3.5.2 - Alerts you about o...
Little Snitch gives you control over your private outgoing data. Track background activity As soon as your computer connects to the Internet, applications often have permission to send any... Read more
Monolingual 1.6.4 - Remove unwanted OS X...
Monolingual is a program for removing unnecesary language resources from OS X, in order to reclaim several hundred megabytes of disk space. If you use your computer in only one (human) language, you... Read more
CleanApp 5.0 - Application deinstaller a...
CleanApp is an application deinstaller and archiver.... Your hard drive gets fuller day by day, but do you know why? CleanApp 5 provides you with insights how to reclaim disk space. There are... Read more
Fantastical 2.0 - Create calendar events...
Fantastical is the Mac calendar you'll actually enjoy using. Creating an event with Fantastical is quick, easy, and fun: Open Fantastical with a single click or keystroke Type in your event details... Read more
Cocktail 8.2 - General maintenance and o...
Cocktail is a general purpose utility for OS X that lets you clean, repair and optimize your Mac. It is a powerful digital toolset that helps hundreds of thousands of Mac users around the world get... Read more
Direct Mail 4.0.4 - Create and send grea...
Direct Mail is an easy-to-use, fully-featured email marketing app purpose-built for OS X. It lets you create and send great looking email campaigns. Start your newsletter by selecting from a gallery... Read more

These are All the Apple Watch Apps and G...
The Apple Watch is less than a month from hitting store shelves, and once you get your hands on it you're probably going to want some apps and games to install. Fear not! We've compiled a list of all the Apple Watch apps and games we've been able to... | Read more »
Appy to Have Known You - Lee Hamlet Look...
Being at 148Apps these past 2 years has been an awesome experience that has taught me a great deal, and working with such a great team has been a privilege. Thank you to Rob Rich, and to both Rob LeFebvre and Jeff Scott before him, for helping me... | Read more »
Hands-On With Allstar Heroes - A Promisi...
Let’s get this out of the way quickly. Allstar Heroes looks a lot like a certain other recent action RPG release, but it turns out that while it’s not yet available here, Allstar Heroes has been around for much longer than that other title. Now that... | Read more »
Macho Man and Steve Austin Join the Rank...
WWE Immortals, by Warner Bros. Interactive Entertainment and WWE, has gotten a superstar update. You'll now have access to Macho Man Randy Savage and Steve Austin. Both characters have two different versions: Macho Man Randy Savage Renegade or Macho... | Read more »
Fearless Fantasy is Fantastic for the iF...
I actually had my first look at Fearless Fantasy last year at E3, but it was on a PC so there wasn't much for me to talk about. But now that I've been able to play with a pre-release version of the iOS build, there's quite a bit for me to talk... | Read more »
MLB Manager 2015 (Games)
MLB Manager 2015 5.0.14 Device: iOS Universal Category: Games Price: $4.99, Version: 5.0.14 (iTunes) Description: Guide your favorite MLB franchise to glory! MLB Manager 2015, officially licensed by MLB.com and based on the award-... | Read more »
Breath of Light (Games)
Breath of Light 1.0.1421 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0.1421 (iTunes) Description: Hold a quiet moment. Breath of Light is a meditative and beautiful puzzle game with a hypnotic soundtrack by... | Read more »
WWE WrestleMania Tags into the App Store
Are You ready to rumble? The official WWE WrestleMania app, by World Wrestling Entertainment, is now available. Now you can get all your WrestleMania info in one place before anyone else. The app offers details on superstar signings, interactive... | Read more »
Bio Inc's New Expansion is Infectin...
Bio Inc., by DryGin Studios, is the real time strategy game where you infect a human body with the worst virus your evil brain can design. Recently, the game was updated to add a whole lot of new features. Now you can play the new “Lethal”... | Read more »
The Monocular Minion is Here! Despicable...
Despicable Me: Minion Rush, by Gameloft, is introducing a new runner to the mix in their latest update. Now you can play as Carl, the prankster minion. Carl has a few new abilities to play with, including running at a higher speed from the start.... | Read more »

Price Scanner via MacPrices.net

13-inch 2.5GHz MacBook Pro (refurbished) avai...
The Apple Store has Apple Certified Refurbished 13″ 2.5GHz MacBook Pros available for $829, or $270 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 13″ 2.... Read more
Save up to $80 on iPad Air 2s, NY tax only, f...
 B&H Photo has iPad Air 2s on sale for $80 off MSRP including free shipping plus NY sales tax only: - 16GB iPad Air 2 WiFi: $469.99 $30 off - 64GB iPad Air 2 WiFi: $549.99 $50 off - 128GB iPad... Read more
iMacs on sale for up to $205 off MSRP
B&H Photo has 21″ and 27″ iMacs on sale for up to $205 off MSRP including free shipping plus NY sales tax only: - 21″ 1.4GHz iMac: $1019 $80 off - 21″ 2.7GHz iMac: $1189 $110 off - 21″ 2.9GHz... Read more
Färbe Technik Offers iPhone Battery Charge LI...
Färbe Technik, which manufactures and markets of mobile accessories for Apple, Blackberry and Samsung mobile devices, is offering tips on how to keep your iPhone charged while in the field: •... Read more
Electronic Recyclers International CEO Urges...
Citing a recent story on CNBC about concerns some security professionals have about the forthcoming Apple Watch, John Shegerian, Chairman and CEO of Electronic Recyclers International (ERI), the... Read more
Save up to $380 with Apple refurbished iMacs
The Apple Store has Apple Certified Refurbished iMacs available for up to $380 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 27″ 3.5GHz 5K iMac – $2119 $... Read more
Mac minis on sale for up to $75 off, starting...
MacMall has Mac minis on sale for up to $75 off MSRP including free shipping. Their prices are the lowest available for these models from any reseller: - 1.4GHz Mac mini: $459.99 $40 off - 2.6GHz Mac... Read more
College Student Deals: Additional $50 off Mac...
Take an additional $50 off all MacBooks and iMacs at Best Buy Online with their College Students Deals Savings, valid through April 11, 2015. Anyone with a valid .EDU email address can take advantage... Read more
Mac Pros on sale for up to $260 off MSRP
B&H Photo has Mac Pros on sale for up to $260 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3719.99... Read more
13-inch 2.5GHz MacBook Pro on sale for $100 o...
B&H Photo has the 13″ 2.5GHz MacBook Pro on sale for $999 including free shipping plus NY sales tax only. Their price is $100 off MSRP. Read more

Jobs Board

DevOps Software Engineer - *Apple* Pay, iOS...
**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Sr. Technical Services Consultant, *Apple*...
**Job Summary** Apple Professional Services (APS) has an opening for a senior technical position that contributes to Apple 's efforts for strategic and transactional Read more
Lead *Apple* Solutions Consultant - Retail...
**Job Summary** Job Summary The Lead ASC is an Apple employee who serves as the Apple business manager and influencer in a hyper-business critical Reseller's store Read more
*Apple* Pay - Site Reliability Engineer - Ap...
**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.