December 96 - MPW Tips and Tricks: Automated Editing With StreamEdit
MPW Tips and Tricks:
Automated Editing With StreamEdit
Tim Maroney
In this column in Issue 26 of develop, I showed you a wide range of
scriptable editing commands available from the MPW Shell. This time I'll
discuss a single tool that provides a powerful self-contained text-editing
scripting language, StreamEdit.
Why would you want to use StreamEdit instead of the other text-editing features
of the MPW Shell?
- Performance -- A StreamEdit script is faster than an MPW script containing
various Replace and Find commands.
- Self-containment -- Because StreamEdit is a self-contained tool, you
can run it from within ToolServer, unlike the scriptable editing commands
discussed in Issue 26, which are available only in the MPW Shell itself. This
means you can use StreamEdit to create lightweight drag-and-drop grinder
AppleScript scripts that send StreamEdit commands to ToolServer.
- Consistency -- Keeping all your editing in a single scripting language
confers the elusive mystical boon of code consistency, making your system
easier to maintain and modify in the future.
StreamEdit is based very closely on the hoary UNIX tool named
sed. If you
already know
sed, much of this will be familiar, but StreamEdit isn't directly
compatible with
sed scripts.
StreamEdit implements a pattern-matching language. Every time a particular
pattern is matched, a sequence of commands will be executed. As in most
pattern-matching languages, StreamEdit's scripts are lists of pattern/command
pairs, with the pattern coming before the command. The input file or files are
read through the script interpreter, which searches for instances of the
patterns and executes the corresponding commands. Anything that doesn't match a
pattern is passed through unchanged.
StreamEdit scans one line at a time through the input, matching its current
line to every pattern in its script. After processing each line, it writes out
the modified line. The result is a concatenation of three internal buffers: the
insert buffer, then the edit buffer, and finally the append buffer. The edit
buffer gets filled with the current line, while the other buffers are empty at
the start. The Insert and Append commands place text in the insert and append
buffers, allowing you to add text to the beginning and end of the output line.
The Change, Delete, and Replace commands modify the contents of the edit buffer.
As usual, MPW uses words in ways previously unknown in human speech. In
StreamEdit, patterns are referred to as "addresses." There are two kinds of
addresses: line numbers and regular expressions. Line numbers ought to be
self-explanatory, but it may help to note that the numbers must be Arabic
numerals rather than Roman, and must be in base 10 rather than the hexadecimal
or sexagesimal number systems. There are three special line numbers:
- the bullet symbol (·, Option-8), meaning the point before the first line
(enabling you to add a line before the first line, for example)
- the infinity symbol (Option-5), meaning the point after
the last line
- dollar sign ($), meaning the last line
The keyboard shortcuts, as always in this column, are for American QWERTY
keyboards; if you've got some other type of keyboard, you're on your own.*
Regular expressions are expressions that manage their diets sensibly. They can
be used for searching, and were explained in detail in Issue 26. In StreamEdit
addresses, though, regular expressions find the entire line containing the
pattern, rather than just the pattern. Regular expressions are denoted by
slashes. Only forward slashes are used (StreamEdit doesn't have a backward
search mode, having been frightened at an early age by the legends of Eurydice
and Lot's wife). Three new constructs have been added to regular expressions in
StreamEdit:
- ç (Option-C), which indicates a case-sensitive search
- // (two slashes), which means the last regular expression that was
matched
- <=variable>= (a variable name embedded in inequality operators,
here overloaded as a special kind of angle brackets, and typed as Option-comma
and Option-period), which means the text of an expanded StreamEdit variable,
treated as literal text to be matched rather than as a regular
expression
StreamEdit has variables that can be set with the Set command
(more on this later) or from the command line using the
-set variable [
=value]
option.
You can form more complex addresses using a few operators. The Boolean and, or,
and not operators are the same as in C (&&, | |, and !, respectively).
Parentheses can be used for grouping within addresses. The comma operator
matches the range of lines specified; for example, 3,5 matches lines 3 through
5. A range address matches each of the lines in the range, if any. It can be
thought of as matching more than once: it fires off the accompanying command on
the first line matched, the last one matched, and all lines in between. If the
termination condition is never met, the address continues to match until the
end of input. This could happen if you specify a range of lines ending at line
15, for instance, and there are only ten lines in the file, or if your range
termination condition is a regular expression that doesn't appear anywhere in
the input.
Matching patterns is very nice, but what do you do once you match them?
Statements in StreamEdit attach actions to patterns. An action consists of one
or more commands, separated by semicolons or by the end of a line. There's no
begin or end bracketing as in Pascal or C. Addresses and commands are
syntactically distinct, so the script interpreter can figure out where the list
of commands for a pattern ends and the next pattern begins.
- Insert text [-n] -- Adds the specified text to the start of the line by
putting it in the insert buffer. The -n option (in this command and in Append
and Change) prevents adding a newline character when the line is written out.
- Append text [-n] -- Adds the specified text to the end of the line by
putting it in the append buffer.
- Change text [-n] -- Changes the line to the specified text by replacing
the contents of the edit buffer.
- Delete -- Clears the edit buffer.
- Replace [-c count] /pattern/ text -- Replaces the pattern with the
specified text. This is the second part of a two-step matching process: first
the address matches a line, then Replace searches in the edit buffer and
replaces the pattern. The count argument indicates the maximum number of times
to perform the replacement in the line. It can be a positive integer or
infinity. The default count is 1.
- Exit [status] -- Stops StreamEdit with the given error status. The default
is 0, which means execution completed with no errors. Any nonzero error status
indicates a problem, and unless the built-in MPW variable Exit is set to
something other than 0, this will stop execution of the script (if any) from
which the StreamEdit command was executed.
- Next -- Somewhat like the C keyword continue. When a Next command is
executed, all pending changes are written out and no more addresses are matched
against the current line; that is, StreamEdit immediately goes on to the next
line without matching the rest of the rules against the current edit buffer.
- Set variable text [-i | -a] -- Much like the MPW Shell Set command. The
variable is set to the specified text. The -i and -a options allow text to be
added to any existing setting of the variable at the start or the end,
respectively.
- Print [text] [-appendto | -to file] -- Writes output to a specified file.
If text is empty, the current line is printed without modification. The
-appendto and -to options write at the end of the file or overwrite the file,
respectively. If no file is specified, standard output is used. If the filename
is empty, nothing gets printed.
- Option AutoDelete -- Deletes all input lines, leaving only output from
Next and Print commands. You can get the same effect by specifying the -d
option on the StreamEdit command line or by including this in the script:
/~/ Delete
The text arguments to these commands are usually literal text, denoted by single or
double quotes. There are a few other forms as well:
- An unquoted variable name can be used, in which case the variable is
expanded; no brackets need be (or even may be) supplied.
- A period means the current input line up to but not including the
newline character at the end.
- As discussed in Issue 26, you can use ® (Option-R) followed by a
digit to mean the expression with that number matched in the pattern.
- You can read text from a file with -from filename, which reads the next
line of text from the specified file. The filename is usually literal text, but
it could also be a variable, the current input line (denoted by a period), or a
® expression.
Let's say you're the director of corporate communications at a major computer
maker and, without any warning except for inventory backlogs larger than the
gross national products of many developing countries, you experience a sudden
transition in chief executive officers, corporate policy, and product line.
Your quarterly report (10-Q) is due in the SEC's EDGAR database tomorrow.
Fortunately the SEC requires the cutting-edge ASCII format for its filings, and
you realize that you can automate 90% of the tedious changes with a single
StreamEdit script.
# Change nickname of CEO
/Diesel/
Replace // 'Flyboy'
# Change corporate policy
/1,$/
Replace /capture market share/ 'survive'
# Remove lines referring to obsolete products
/PowerTalk/ || /eWorld/
Delete
# Change developer relations strategy
/third-party developers/
Replace /evangelize/ 'listen to'
# Mark lines referring to old schedules with a
# distinctive string at the start of the line
# for manual editing later
/1996/
Insert 'WHOOPS: '
# Add new final line of report
[[infinity]]
Append 'May God have mercy on our souls.'
StreamEdit is almost too powerful. People have used it for everything,
including pretty-printing source code, converting files to HTML, and
postprocessing object files for dynamic linking tools. If you use it for
finding incriminating passages in coworkers' e-mail, karma may get you, but the
limitations of the tool won't. Use your powers for good rather than evil, and a
grateful world will thank you.
TIM MARONEY has appeared professionally in newspapers, magazines, compact
discs, videotape, and of course, computer software. Tim is a technical lead in
human interface software at Apple and is editing a series of books for a horror
publisher. His skin burns easily in the sun and tans in the moon. He uses white
T-shirts only for house painting and car repair.*
Thanks to Arno Gourdol, Alex McKale, and Robert Ulrich for reviewing this
column.*
