TweetFollow Us on Twitter

Mac In The Shell: More from the Unknown

Volume Number: 23 (2007)
Issue Number: 07
Column Tag: Mac In The Shell

More from the Unknown

More shell tools that typically go unnoticed

by Edward Marczak

Introduction

OS X has an incredible breadth and depth available in the tools accessible from a shell. So much so, that it's hard to know each and every tool available. Some are almost hidden: tucked away in places one does not typically look. Others are hidden in plain sight: exactly where you'd expect them, but buried among the volume. This month, I'm going to dig out and introduce some more useful shell tools that you may not know about.

text manipulation

It's long been the domain of scripting languages like perl, and perhaps now to a lesser extent PHP, to slice, dice and otherwise manipulate text. However, there's a framework built right into OS X, so why not take advantage of it? The Cocoa text system is exposed to the shell via the "textutil" application. "textutil" can convert and manipulate many different formats of text documents. In fact, it works with a surprising number formats:

   txt
   html
   Microsoft Word "doc"
   Microsoft Word XML
   rtf
   webarchive

It's also incredibly easy to get started. Using the "-convert" switch, you can accomplish most conversions that you'd ever want:

textutil -convert html resume.doc

This will convert the Microsoft Word-based "resume.doc" into a separate HTML document named "resume.html". I'll immediately point out that textutil writes much, much better HTML than Word.

Note that the input format is determined 'automagically' by a number of methods. If textutil misidentifies the format of the source file, you tell it with the "-format" switch:

textutil -format html -convert rtf file1

This will tell textutil that "file1" is an HTML file, and we want to convert it into a rich text format file named "file1.rtf".

There's also a very handy "-info" switch that returns info on a given file:

$ textutil -info 2007-01-On\ Logs.doc
File:  2007-01-Marczak On Logs.doc
Type:  Word format
Size:  25600 bytes
Length:  3746 characters
Title:  MacTech Template
Author:  Classics Department
Last Editor:  Edward Marczak
Company:  Radiotope
Subject:  
Keywords:  
Created:  2006-11-02 17:56:00 -0500
Last Modified:  2006-11-06 09:47:00 -0500

As you can see, documents sometimes have incorrect metadata. Well, textutil can help you handle that as well! There are a number of metadata related switches:

strip: Remove metadata in target during conversion.

title: Set the title metadata attribute for output files.

author: Set the author metadata attribute for output files.

subject: Specify the subject metadata attribute for output files.

keywords: a shell-quoted list of 'tags' ("keyword1,keyword2, ...")

comment: Set the comment attribute for the output files.

editor: val Set the editor attribute for output files.

company: val Set the company attribute for output files.

creationtime: Set the creation time attribute for output files (in "yyyy-mm-ddThh:mm:ssZ" format).

modificationtime: Set the modification time attribute for output files (in "yyyy-mm-ddThh:mm:ssZ" format).

So, we could create an HTML document from a plain text file with the title and keywords set by running textutil like this:

textutil -convert html -title "Anti-gravity Thesis" -keywords "(anti,gravity,'atom power',physics)" -author "Bruce Banner" ag-file.txt

You'll find a well-formed HTML file with a little bit of CSS. Of course, there are many times when you need finer control over the tags that will be used in that HTML. Have no fear! The "-excludedelements" switch can help you out. This switch allows you to specify which HTML elements should not be used in the generated HTML. Like the "-keywords" switch, "-excludedelements" expects a single argument, so you can use a shell-quoted list of tags to skip.

Keep it clean

Speaking of well-formed HTML, manipulating tags and their parameters is a special case of text manipulation. Fortunately, tidy, the standard Unix utility just for this purpose, ships with OS X. So, if you've used textutil to generate HTML based output, tidy can take you even further.

The simplest case lets tidy modify your HTML document in place. To do so, use the "-m" switch:

$ tidy -im ag-file.html
Info: Doctype given is "-//W3C//DTD HTML 4.01//EN"
Info: Document content looks like HTML 4.01 Strict
No warnings or errors were found.

Take look at your file before and after running the tidy command. I also like to include the "-i" switch, which ensures tag levels are indented appropriately. Also, by default, tidy will lowercase all tags (which, by me, is "correct"). This can be overridden with the "-u" switch.

One of my absolute favorite uses of tidy is to get rid of "illegal" characters. This is easily accomplished with the "-b" (bare) switch. This strips the document of smart quotes, and dashes, and other characters that cause problems in HTML documents.

There's much, much more that tidy can do. Check the very well written man page to look at more options. Don't forget, though, that you can combine textutil and tidy into one neat shell statement:

textutil -stdout ag-file.txt -convert html | tidy -i -output ag-file.html

This command uses textutil to convert a text file to HTML, and sends the output to stdout. Then, that output is piped to tidy, which then requires us to use the "-output" switch to write the file to disk.

Finally, realize that both textutil and tidy can be used with a wildcard character to process an entire group of files. If all files are in a single directory, simply change to that directory and:

textutil -convert html *.doc

This will convert all Word documents in the current directory into HTML files. To process the current folder and all subfolders, use the find utility:

find . -name \*txt -print0 | xargs -0 textutil -convert html

I opted for xargs over find's exec command to ensure that large directories can be processed. You can scrub Word metadata in-place using this technique:

textutil -strip -convert doc *.doc

Painless, right?

Image Manipulation

While slinging text around may be a common, and very specific operation, doing the same to images requires a different set of tools. OS X's scriptable image processor server or, "sips" is just the ticket.

Like textutil, sips leans on frameworks already built into the core of OS X. This, of course, lets you do all of the great things that you can with a GUI right in a shell! Why is this exciting? Automation, naturally.

In the past, I've put together both nightly reports and 'one-click-builds' of documents using both user submitted and random images. The trick is not so much the content, but the dimensions of the graphic: you need consistent dimensions so you can automate your layout. Resizing graphic files is one of the things that sips does best:

sips -Z 100x100 IMG_1312.JPG --out image1-sized.jpg

The "-Z" switch is really nice: it keeps the proportions of the original image, but makes sure neither the width or height exceed the dimensions specified. The "--out" switch writes the result to a new file. The image you just made doesn't have a nice preview icon in the Finder? Add one:

sips -i image1-sized.jpg

Oh, you didn't want a JPEG format file? Convert it at the same time using the "-s" (setProperty) switch (let's do it all in one shot):

sips -Z 100x100 IMG_1312.JPG -i -s format png --out image-sized.png

This one command resizes the graphic, converts the output to png and creates the Finder preview icon. Nice.

For automation, though, you often need to find out information about the source before you blindly process it. Again, like textutil, sips has functions for this. You can dump all of an image's data with the "-g all" switch and parse the output yourself:

$ sips -g all IMG_1312.JPG 
/Users/erm/Pictures/Parade/IMG_1312.JPG
  pixelWidth: 2592
  pixelHeight: 1944
  typeIdentifier: public.jpeg
  format: jpeg
  formatOptions: default
  dpiWidth: 180.000
  dpiHeight: 180.000
  samplesPerPixel: 3
  bitsPerSample: 8
  hasAlpha: no
  space: RGB
  profile: Camera RGB Profile
  creation: 2007:04:04 17:23:31
  make: Canon
  model: Canon PowerShot S2 IS

...or, you can request one or more of these parameters directly:

$ sips -g pixelWidth -g pixelHeight IMG_1312.JPG
/Users/marczak/Pictures/Parade/IMG_1312.JPG
  pixelWidth: 2592
  pixelHeight: 1944

Like textutil, feel free to throw a wildcard at sips, or, use the find/xargs trick.

Why?

So, why go through this trouble, when you can fire up Photoshop and make your changes (and even script it!), or use a Word processor to mold your words? Again, automation comes to mind. Specifically, automation with low overhead. Using the shell tools, you can run scripts on a server without having a GUI or installing an application like Photoshop at all. Combine this with some of the techniques I presented in April, and you can achieve some incredibly complex workflows.

Even cooler: more and more GUI utilities have some way to run shell jobs. Textmate, for example, can run a selection or entire document through a shell tool and plop the results right back in the live document. Even apps that don't have this functionality built in can usually be faked thanks to AppleScript. Take Mail.app, for instance. It lets you create signatures, but not dynamic signatures that rely on a running process. What if you wanted to include a snapshot from your iSight in your sig? Script a capture from isightcapture (http://www.intergalactic.de/hacks.html), run it through sips and AppleScript the copy and pasting of into Mail.

Or, consider a simpler alternative: you've installed fortune and want a random fortune in your sig. A simple bash script could be as follows:

#!/bin/bash
# Tell Mail.app to select all and copy to clipboard
osascript <<MailCopy
tell application "System Events"
        tell application "Mail" to activate
        keystroke "a" using {command down}
        keystroke "c" using {command down}
end tell
MailCopy
# Write clipboard to file
pbpaste > /tmp/mailpaste.tmp
fortune -s >> /tmp/mailpaste.tmp
textutil -stdout -convert rtf /tmp/mailpaste.tmp | pbcopy
# Tell Mail.app to activate and paste in the contents of the clipboard
osascript <<MailPaste
tell application "System Events"
        tell application "Mail" to activate
        keystroke "a" using {command down}
        keystroke "v" using {command down}
end tell
MailPaste

This is admittedly a quick hack with absolutely no error checking or other niceties. However, if you open up Mail.app, create a new message, put your cursor anywhere in the body and run this, you'll get a fortune tagged onto the end of your message (it helps if you already have a sig created). Pretty it up a bit and you could run it via the GUI via a ".command" file, or, wrap it in AppleScript and run it from your script menulet. Or, if you're a Quicksilver user, don't ignore the qs shell tool.

Of course, the real power lies in integrating other data: from the web, from Excel, from a database....wherever!

Conclusion

When venerable shell tools are integrated with the power of OS X, the workflow possibilities truly are endless. Be creative in your application of scripting tools, and you'll be able to solve any problem for your client/employer/self.

Media of the month: Brendan Benson's new album, "Alternative to Love". It's just good, fun pop music, no strings attached.

Until next month, keep thinking of ways to keep the work flowing.

References:

"Developer Release Note": http://developer.apple.com/releasenotes/Cocoa/AppKit.html


Ed Marczak owns and operates Radiotope, a technology consultancy that just tries to make life easier through the technology that already exists. He's also the Executive Editor of MacTech Magazine, and CTO of WheresSpot. His copious free time is spent with his wife and two daughters, currently enjoying the weather in the North East U.S.

 
AAPL
$98.77
Apple Inc.
-0.26
MSFT
$44.00
Microsoft Corpora
+0.03
GOOG
$589.11
Google Inc.
-1.49

MacTech Search:
Community Search:

Software Updates via MacUpdate

OS X Yosemite Wallpaper 1.0 - Desktop im...
OS X Yosemite Wallpaper is the gorgeous new background image for Apple's upcoming OS X 10.10 Yosemite. This wallpaper is available for all screen resolutions with a source file that measures 5,418... Read more
Acorn 4.4 - Bitmap image editor. (Demo)
Acorn is a new image editor built with one goal in mind - simplicity. Fast, easy, and fluid, Acorn provides the options you'll need without any overhead. Acorn feels right, and won't drain your bank... Read more
Bartender 1.2.20 - Organize your menu ba...
Bartender lets you organize your menu bar apps. Features: Lets you tidy your menu bar apps how you want. See your menu bar apps when you want. Hide the apps you need to run, but do not need to... Read more
TotalFinder 1.6.2 - Adds tabs, hotkeys,...
TotalFinder is a universally acclaimed navigational companion for your Mac. Enhance your Mac's Finder with features so smart and convenient, you won't believe you ever lived without them. Tab-based... Read more
Vienna 3.0.0 RC 2 :be5265e: - RSS and At...
Vienna is a freeware and Open-Source RSS/Atom newsreader with article storage and management via a SQLite database, written in Objective-C and Cocoa, for the OS X operating system. It provides... Read more
VLC Media Player 2.1.5 - Popular multime...
VLC Media Player is a highly portable multimedia player for various audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, MP3, OGG, ...) as well as DVDs, VCDs, and various streaming protocols. It... Read more
Default Folder X 4.6.7 - Enhances Open a...
Default Folder X attaches a toolbar to the right side of the Open and Save dialogs in any OS X-native application. The toolbar gives you fast access to various folders and commands. You just click... Read more
TinkerTool 5.3 - Expanded preference set...
TinkerTool is an application that gives you access to additional preference settings Apple has built into Mac OS X. This allows to activate hidden features in the operating system and in some of the... Read more
Audio Hijack Pro 2.11.0 - Record and enh...
Audio Hijack Pro drastically changes the way you use audio on your computer, giving you the freedom to listen to audio when you want and how you want. Record and enhance any audio with Audio Hijack... Read more
Intermission 1.1.1 - Pause and rewind li...
Intermission allows you to pause and rewind live audio from any application on your Mac. Intermission will buffer up to 3 hours of audio, allowing users to skip through any assortment of audio... Read more

Latest Forum Discussions

See All

To The End Review
To The End Review By Lee Hamlet on July 29th, 2014 Our Rating: :: A VICIOUS CYCLEUniversal App - Designed for iPhone and iPad To The End will test players’ patience, timing, and dedication as they try to navigate all 13 levels in... | Read more »
Traps n’ Gemstones Review
Traps n’ Gemstones Review By Campbell Bird on July 28th, 2014 Our Rating: :: CASTLEVANIA JONESUniversal App - Designed for iPhone and iPad Fight mummies, dig tunnels, and ride a runaway minecart to discover ancient secrets in this... | Read more »
The Phantom PI Mission Apparition Review
The Phantom PI Mission Apparition Review By Jordan Minor on July 28th, 2014 Our Rating: :: GHOSTS BUSTEDUniversal App - Designed for iPhone and iPad The Phantom PI is an exceedingly clever and well-crafted adventure game.   | Read more »
More Stubies Are Coming Your Way in a Ne...
More Stubies Are Coming Your Way in a New Update Posted by Jessica Fisher on July 28th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
The Great Prank War Review
The Great Prank War Review By Nadia Oxford on July 28th, 2014 Our Rating: :: PRANKING IS SERIOUS BUSINESSUniversal App - Designed for iPhone and iPad Though short, The Great Prank War offers an interesting and fun mix of action and... | Read more »
Marvel Contest of Champions Announced at...
Marvel Contest of Champions Announced at Comic-Con Posted by Jennifer Allen on July 28th, 2014 [ permalink ] Announced over the weekend at San Diego Comic-Con was the fairly exciting looking Marvel Contest of Champions. | Read more »
Teenage Mutant Ninja Turtles Review
Teenage Mutant Ninja Turtles Review By Jennifer Allen on July 28th, 2014 Our Rating: :: DULL SWIPINGUniversal App - Designed for iPhone and iPad The pizza power is weak when it comes to this Teenage Mutant Ninja Turtles game.   | Read more »
Exploration Focused Puzzle Game Beatbudd...
Exploration Focused Puzzle Game Beatbuddy Set to Make Transition from PC to iOS this September Posted by Jennifer Allen on July 28th, 2014 [ permalink ] | Read more »
PlanetHD
PlanetHD By Nadia Oxford on July 28th, 2014 Our Rating: :: SPACE MADNESSUniversal App - Designed for iPhone and iPad PlanetHD will keep players busy for a while, though its unpredictable physics are a handful to deal with.   | Read more »
This Week at 148Apps: July 21-25, 2014
Another Week of Expert App Reviews   At 148Apps, we help you sort through the great ocean of apps to find the ones we think you’ll like and the ones you’ll need. Our top picks become Editor’s Choice, our stamp of approval for apps with that little... | Read more »

Price Scanner via MacPrices.net

Apple updates MacBook Pros with slightly fast...
Apple updated 13″ and 15″ Retina MacBook Pros today with slightly faster Haswell processors. 13″ models now ship with 8GB of RAM standard, while 15″ MacBook Pros ship with 16GB across the board. Most... Read more
Apple drops price on 13″ 2.5GHz MacBook Pro b...
The Apple Store has dropped their price for the 13″ 2.5GHz MacBook Pro by $100 to $1099 including free shipping. Read more
Apple drops prices on refurbished 2013 MacBoo...
The Apple Store has dropped prices on Apple Certified Refurbished 13″ and 15″ 2013 MacBook Pros, with model now available starting at $929. Apple’s one-year warranty is standard, and shipping is free... Read more
iOS 8 and OS X 10.10 To Support DuckDuckGo As...
Writing for Quartz, Dan Frommer reports that Apple’s forthcoming iOS 8 and OS X 10.10 operating systems version updates will allow users to select DuckDuckGo as their default search engine. He notes... Read more
U.K. Hospital Using iPods and iPads To Record...
British news journal GazetteLive’s. Ian McNeal notes that the old “an apple a day keeps the doctor away” proverb is being turned on its head at http://southtees.nhs.uk/hospitals/james-cook/ James... Read more
13-inch 2.5GHz MacBook Pro on sale for $1099,...
Best Buy has the 13″ 2.5GHz MacBook Pro available for $1099.99 on their online store. Choose free shipping or free instant local store pickup (if available). Their price is $100 off MSRP. Price is... Read more
Roundup of Apple refurbished MacBook Pros, th...
The Apple Store has Apple Certified Refurbished 13″ and 15″ MacBook Pros available for up to $400 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free. Their prices... Read more
Record Mac Shipments In Q2/14 Confound Analys...
A Seeking Alpha Trefis commentary notes that Apple’s fiscal Q3 2014 results released July 22, beat market predictions on earnings, although revenues were slightly lower than anticipated. Apple’s Mac’... Read more
Intel To Launch Core M Silicon For Use In Not...
Digitimes’ Monica Chen and Joseph Tsai, report that Intel will launch 14nm-based Core M series processors specifically for use in fanless notebook/tablet 2-in-1 models in Q4 2014, with many models to... Read more
Apple’s 2014 Back to School promotion: $100 g...
 Apple’s 2014 Back to School promotion includes a free $100 App Store Gift Card with the purchase of any new Mac (Mac mini excluded), or a $50 Gift Card with the purchase of an iPad or iPhone,... Read more

Jobs Board

Sr Software Lead Engineer, *Apple* Online S...
Sr Software Lead Engineer, Apple Online Store Publishing Systems Keywords: Company: Apple Job Code: E3PCAK8MgYYkw Location (City or ZIP): Santa Clara Status: Full Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
Sr. Product Leader, *Apple* Store Apps - Ap...
**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.