TweetFollow Us on Twitter

December 96 - MPW Tips and Tricks: Automated Editing With StreamEdit

MPW Tips and Tricks:
Automated Editing With StreamEdit

Tim Maroney

In this column in Issue 26 of develop, I showed you a wide range of scriptable editing commands available from the MPW Shell. This time I'll discuss a single tool that provides a powerful self-contained text-editing scripting language, StreamEdit.

Why would you want to use StreamEdit instead of the other text-editing features of the MPW Shell?

  • Performance -- A StreamEdit script is faster than an MPW script containing various Replace and Find commands.

  • Self-containment -- Because StreamEdit is a self-contained tool, you can run it from within ToolServer, unlike the scriptable editing commands discussed in Issue 26, which are available only in the MPW Shell itself. This means you can use StreamEdit to create lightweight drag-and-drop grinder AppleScript scripts that send StreamEdit commands to ToolServer.

  • Consistency -- Keeping all your editing in a single scripting language confers the elusive mystical boon of code consistency, making your system easier to maintain and modify in the future.


GETTING TO KNOW YOU

StreamEdit is based very closely on the hoary UNIX tool named sed. If you already know sed, much of this will be familiar, but StreamEdit isn't directly compatible with sed scripts. StreamEdit implements a pattern-matching language. Every time a particular pattern is matched, a sequence of commands will be executed. As in most pattern-matching languages, StreamEdit's scripts are lists of pattern/command pairs, with the pattern coming before the command. The input file or files are read through the script interpreter, which searches for instances of the patterns and executes the corresponding commands. Anything that doesn't match a pattern is passed through unchanged.

StreamEdit scans one line at a time through the input, matching its current line to every pattern in its script. After processing each line, it writes out the modified line. The result is a concatenation of three internal buffers: the insert buffer, then the edit buffer, and finally the append buffer. The edit buffer gets filled with the current line, while the other buffers are empty at the start. The Insert and Append commands place text in the insert and append buffers, allowing you to add text to the beginning and end of the output line. The Change, Delete, and Replace commands modify the contents of the edit buffer.


SHARING ADDRESSES

As usual, MPW uses words in ways previously unknown in human speech. In StreamEdit, patterns are referred to as "addresses." There are two kinds of addresses: line numbers and regular expressions. Line numbers ought to be self-explanatory, but it may help to note that the numbers must be Arabic numerals rather than Roman, and must be in base 10 rather than the hexadecimal or sexagesimal number systems. There are three special line numbers:
  • the bullet symbol (·, Option-8), meaning the point before the first line (enabling you to add a line before the first line, for example)

  • the infinity symbol (Option-5), meaning the point after the last line

  • dollar sign ($), meaning the last line

    The keyboard shortcuts, as always in this column, are for American QWERTY keyboards; if you've got some other type of keyboard, you're on your own.*
Regular expressions are expressions that manage their diets sensibly. They can be used for searching, and were explained in detail in Issue 26. In StreamEdit addresses, though, regular expressions find the entire line containing the pattern, rather than just the pattern. Regular expressions are denoted by slashes. Only forward slashes are used (StreamEdit doesn't have a backward search mode, having been frightened at an early age by the legends of Eurydice and Lot's wife). Three new constructs have been added to regular expressions in StreamEdit:
  • ç (Option-C), which indicates a case-sensitive search

  • // (two slashes), which means the last regular expression that was matched

  • <=variable>= (a variable name embedded in inequality operators, here overloaded as a special kind of angle brackets, and typed as Option-comma and Option-period), which means the text of an expanded StreamEdit variable, treated as literal text to be matched rather than as a regular expression
StreamEdit has variables that can be set with the Set command (more on this later) or from the command line using the -set variable [=value] option.

You can form more complex addresses using a few operators. The Boolean and, or, and not operators are the same as in C (&&, | |, and !, respectively). Parentheses can be used for grouping within addresses. The comma operator matches the range of lines specified; for example, 3,5 matches lines 3 through 5. A range address matches each of the lines in the range, if any. It can be thought of as matching more than once: it fires off the accompanying command on the first line matched, the last one matched, and all lines in between. If the termination condition is never met, the address continues to match until the end of input. This could happen if you specify a range of lines ending at line 15, for instance, and there are only ten lines in the file, or if your range termination condition is a regular expression that doesn't appear anywhere in the input.


TAKING ACTION

Matching patterns is very nice, but what do you do once you match them? Statements in StreamEdit attach actions to patterns. An action consists of one or more commands, separated by semicolons or by the end of a line. There's no begin or end bracketing as in Pascal or C. Addresses and commands are syntactically distinct, so the script interpreter can figure out where the list of commands for a pattern ends and the next pattern begins.

Editing commands

  • Insert text [-n] -- Adds the specified text to the start of the line by putting it in the insert buffer. The -n option (in this command and in Append and Change) prevents adding a newline character when the line is written out.

  • Append text [-n] -- Adds the specified text to the end of the line by putting it in the append buffer.

  • Change text [-n] -- Changes the line to the specified text by replacing the contents of the edit buffer.

  • Delete -- Clears the edit buffer.

  • Replace [-c count] /pattern/ text -- Replaces the pattern with the specified text. This is the second part of a two-step matching process: first the address matches a line, then Replace searches in the edit buffer and replaces the pattern. The count argument indicates the maximum number of times to perform the replacement in the line. It can be a positive integer or infinity. The default count is 1.

Control commands

  • Exit [status] -- Stops StreamEdit with the given error status. The default is 0, which means execution completed with no errors. Any nonzero error status indicates a problem, and unless the built-in MPW variable Exit is set to something other than 0, this will stop execution of the script (if any) from which the StreamEdit command was executed.

  • Next -- Somewhat like the C keyword continue. When a Next command is executed, all pending changes are written out and no more addresses are matched against the current line; that is, StreamEdit immediately goes on to the next line without matching the rest of the rules against the current edit buffer.

  • Set variable text [-i | -a] -- Much like the MPW Shell Set command. The variable is set to the specified text. The -i and -a options allow text to be added to any existing setting of the variable at the start or the end, respectively.

Output commands

  • Print [text] [-appendto | -to file] -- Writes output to a specified file. If text is empty, the current line is printed without modification. The -appendto and -to options write at the end of the file or overwrite the file, respectively. If no file is specified, standard output is used. If the filename is empty, nothing gets printed.

  • Option AutoDelete -- Deletes all input lines, leaving only output from Next and Print commands. You can get the same effect by specifying the -d option on the StreamEdit command line or by including this in the script:
    /~/    Delete
    
The text arguments to these commands are usually literal text, denoted by single or double quotes. There are a few other forms as well:
  • An unquoted variable name can be used, in which case the variable is expanded; no brackets need be (or even may be) supplied.

  • A period means the current input line up to but not including the newline character at the end.

  • As discussed in Issue 26, you can use ® (Option-R) followed by a digit to mean the expression with that number matched in the pattern.

  • You can read text from a file with -from filename, which reads the next line of text from the specified file. The filename is usually literal text, but it could also be a variable, the current input line (denoted by a period), or a ® expression.


A HYPOTHETICAL EXAMPLE

Let's say you're the director of corporate communications at a major computer maker and, without any warning except for inventory backlogs larger than the gross national products of many developing countries, you experience a sudden transition in chief executive officers, corporate policy, and product line. Your quarterly report (10-Q) is due in the SEC's EDGAR database tomorrow. Fortunately the SEC requires the cutting-edge ASCII format for its filings, and you realize that you can automate 90% of the tedious changes with a single StreamEdit script.
# Change nickname of CEO
/Diesel/
Replace // 'Flyboy'

# Change corporate policy
/1,$/
Replace /capture market share/ 'survive'

# Remove lines referring to obsolete products
/PowerTalk/ || /eWorld/
Delete

# Change developer relations strategy
/third-party developers/
Replace /evangelize/ 'listen to'

# Mark lines referring to old schedules with a
# distinctive string at the start of the line
# for manual editing later
/1996/
Insert 'WHOOPS: '

# Add new final line of report
[[infinity]]
Append 'May God have mercy on our souls.'


CONTROL YOURSELF

StreamEdit is almost too powerful. People have used it for everything, including pretty-printing source code, converting files to HTML, and postprocessing object files for dynamic linking tools. If you use it for finding incriminating passages in coworkers' e-mail, karma may get you, but the limitations of the tool won't. Use your powers for good rather than evil, and a grateful world will thank you.

TIM MARONEY has appeared professionally in newspapers, magazines, compact discs, videotape, and of course, computer software. Tim is a technical lead in human interface software at Apple and is editing a series of books for a horror publisher. His skin burns easily in the sun and tans in the moon. He uses white T-shirts only for house painting and car repair.*

Thanks to Arno Gourdol, Alex McKale, and Robert Ulrich for reviewing this column.*

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Geekbench 4.1.1 - Measure processor and...
Geekbench provides a comprehensive set of benchmarks engineered to quickly and accurately measure processor and memory performance. Designed to make benchmarks easy to run and easy to understand,... Read more
iMazing 2.3.3 - Complete iOS device mana...
iMazing (was DiskAid) is the ultimate iOS device manager with capabilities far beyond what iTunes offers. With iMazing and your iOS device (iPhone, iPad, or iPod), you can: Copy music to and from... Read more
TeamViewer 12.0.81279 - Establish remote...
TeamViewer gives you remote control of any computer or Mac over the Internet within seconds, or can be used for online meetings. Find out why more than 200 million users trust TeamViewer! Free for... Read more
Safari Technology Preview 11.0 - The new...
Safari Technology Preview contains the most recent additions and improvements to WebKit and the latest advances in Safari web technologies. And once installed, you will receive notifications of... Read more
PDFpen 9.1 - $74.95
PDFpen allows users to easily edit PDF's. Add text, images and signatures. Fill out PDF forms. Merge or split PDF documents. Reorder and delete pages. Even correct text and edit graphics! Features... Read more
GraphicConverter 10.4.3 - $39.95
GraphicConverter is an all-purpose image-editing program that can import 200 different graphic-based formats, edit the image, and export it to any of 80 available file formats. The high-end editing... Read more
Fission 2.3.2 - Streamlined audio editor...
Fission can crop and trim audio, paste in or join files, or just rapidly split one long file into many. It's streamlined for fast editing. Plus, it works without the quality loss caused by other... Read more
EarthDesk 7.2 - Striking real-time anima...
EarthDesk replaces your static desktop picture with a rendered image of Earth showing correct sun, moon, and city illumination. With an Internet connection, EarthDesk displays near-real-time global... Read more
Things 3.1.1 - Elegant personal task man...
Things is a task management solution that helps to organize your tasks in an elegant and intuitive way. Things combines powerful features with simplicity through the use of tags and its intelligent... Read more
iDefrag 5.2.0 - Disk defragmentation and...
iDefrag helps defragment and optimize your disk for improved performance. iDefrag Features Supports HFS and HFS+ (Mac OS Extended). Supports case sensitive and journaled filesystems. Supports... Read more

Latest Forum Discussions

See All

Aero Effect (Games)
Aero Effect 1.0.1 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0.1 (iTunes) Description: Race an aerodynamic thingy through a maze of animated, geometric peril and plunge into a mysterious grid of pixels, all while... | Read more »
Linelight (Games)
Linelight 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Linelight is an elegant, minimalist puzzle game set in a universe of lines. Its puzzles will awake your mind as the music flows... | Read more »
Fighting Fantasy Legends (Games)
Fighting Fantasy Legends 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: Create your own adventures in a dangerous land of monsters, treasures and traps. From renowned authors Steve Jackson... | Read more »
Knight Fever is a new take on the classi...
Knight Fever lands on the Google Play, and you won’t want to miss it if you’re a devoted RPG fan. Developed by indie creatives Buff Studios, the game looks like a retro RPG with some exciting new twists. | Read more »
Steam Panic (Games)
Steam Panic 1.0 Device: iOS Universal Category: Games Price: $3.99, Version: 1.0 (iTunes) Description: Steampunk puzzle in which you spin the playfield to put yourself back together. Dear Reader,Enclosed is a copy of my bizarre 1935... | Read more »
Time Crash (Games)
Time Crash 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: Time is about to Crash! It’s up to you to save the city! Time Crash is a 3D first person runner which lets you play as a powerful... | Read more »
Galaxy of Pen & Paper (Games)
Galaxy of Pen & Paper 1.0.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.0 (iTunes) Description: *** NO IN-APP PURCHASES! ****** PREMIUM FOREVER *** THE ULTIMATE ROLE-PLAYING SIMULATION GOES TO SPACE! | Read more »
Niantic reveals more Pokémon GO legendar...
Oof. Pokémon GO Fest Chicago was pretty much a fiasco, huh? Niantic is feeling the heat from fans for their first big event that ultimately ended in failure. So much so that they've released a much longer apology that gives a full run down of... | Read more »
The 5 best life-saving apps for dog owne...
While it's true that dogs are man's best friend, they're also a pretty big responsibility. We want to give our dogs the best lives, but with busy schedules that's not always easy. Luckily, though, there are a bunch of quality apps out there that... | Read more »
Mix and match magical brews in Miracle M...
Miracle Merchant, the charming fantasy card game by Tiny Touch Tales, is arriving next week. The development team, which also brought you Card Crawl and Card Thief, announced the game's launch with a pleasant little trailer that showcases the game'... | Read more »

Price Scanner via MacPrices.net

27-inch 3.5GHz iMac on sale for $100 off MSRP...
Adorama has the new 27″ 3.5GHz iMac (MNEA2LL/A) on sale for $1899 including free shipping. Their price is $100 off MSRP. Adorama charges sales tax for purchases in NY & NJ only. Read more
Seven Cities, One Phone: OtterBox Sends iPhon...
Plenty of people have trekked around the world, but what about a globe-trotting iPhone? OtterBox is sending an iPhone around the world to capture the adventures of a diverse set of global ambassadors... Read more
L-Card Pro App May Spell End For Paper Busine...
OrangeTreeApps, LLC has announced the release of L-Card Pro 1.1, an update to their business app for iOS and Android devices that introduces eco-friendly, affordable, electronic business cards on the... Read more
Clearance previous generation iMacs, Apple re...
Apple has previous-generation Certified Refurbished 2015 21″ & 27″ iMacs available starting at $849. Apple’s one-year warranty is standard, and shipping is free. The following models are... Read more
27-inch 3.4GHz iMac on sale for $1699, save $...
MacMall has the new 2017 27″ 3.4GHz iMac (MNE92LL/A) in stock and on sale for $1699 including free shipping. Their price is $100 off MSRP. Read more
Photographer Explains Choosing Dell Laptop Ov...
Last week photographer and video blogger Manny Ortiz posted a video explaining the five most important reasons he settled on a Dell XPS 15 laptop instead of a MacBook Pro for his latest portable... Read more
Sale! 10-inch iPad Pros for $50 off MSRP, no...
B&H Photo has 64GB and 256GB 10.5″ iPad Pros in stock today and on sale for $50 off MSRP. Each iPad includes free shipping, and B&H charges sales tax in NY & NJ only: – 10.5″ 64GB iPad... Read more
WaterField Designs Upgrades TSA-friendly Zip...
San Francisco based designer and manufacturer Waterfield Designs has unveiled an upgraded and refined Zip Brief. Ideal for the minimalist professional, the ultra-slim Zip laptop bag actually holds a... Read more
USB 3.0 Promoter Group Announces USB 3.2 Upda...
The USB 3.0 Promoter Group has announced the pending release of the USB 3.2 specification, an incremental update that defines multi-lane operation for new USB 3.2 hosts and devices. USB Developer... Read more
Save on MacBook Pros with Apple Refurbished 2...
Apple recently dropped prices on Certified Refurbished 2016 15″ and 13″ MacBook Pros with models now as much as $590 off original MSRP. An Apple one-year warranty is included with each model, and... Read more

Jobs Board

*Apple* Solutions Consultant - Apple Inc. (U...
Job Summary As an Apple Solutions Consultant, you'll be the link between our future customers and our products. You'll showcase your entrepreneurial spirit as you Read more
Senior Software Engineer, *Apple* Online St...
Job Summary The Apple Online Store is looking for an experienced, self-driven, detail-oriented software engineer who can join our team to help build highly scalable Read more
Frameworks Engineering Manager, *Apple* Wat...
Frameworks Engineering Manager, Apple Watch Job Number: 41632321 Santa Clara Valley, California, United States Posted: Jun. 15, 2017 Weekly Hours: 40.00 Job Summary Read more
Senior Software Engineer, *Apple* Online St...
Changing the world is all in a day's work at Apple . If you love innovation, here's your chance to make a career of it. You'll work hard. But the job comes with more Read more
Frameworks Engineering Manager, *Apple* Wat...
Frameworks Engineering Manager, Apple Watch Job Number: 41632321 Santa Clara Valley, California, United States Posted: Jun. 15, 2017 Weekly Hours: 40.00 Job Summary Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.