TweetFollow Us on Twitter

Quoting in the shell

Volume Number: 23 (2007)
Issue Number: 09
Column Tag: MacEnterprise

Quoting in the shell

Dealing with interesting characters in shell programming

By Philip Rinehart, Yale University

QUOTES?

This month, a rather interesting problem arose on the Macenterprise mailing list. How are quotes used and dealt with when scripting for the bash shell? It can be a particularly difficult problem, as OS X allows the use of non-standard characters and spaces in filenames. If quoting is not done properly, unexpected, and even disastrous results may occur. Anyone, remember iTunes 2.0 and destruction of hard drive data? It was a quoting problem! Thus, it is extremely important to quote things properly when shell programming, both for unexpected results, as well as properly sanitizing input.

Single Quotes?

Let's start the process by looking at the use of single quotes in bash shell scripting. The original question was about the use of a directory listing. The directory listing was then piped to a second command, which failed due to spaces in some of the directory names. The specific example:

   ls -F /Applications

Try it. Note how many Applications contain spaces in the name. If Adobe Acrobat 8 Professional were installed, any shell script attempting to perform an action on the path would fail. Why? As the shell interprets spaces as input separators, Adobe, Acrobat, 8 and Professional are all seen as individual paths. In our above example, if the chmod operation were being performed, "Adobe", "Acrobat", "8" and "Professional" would all be modified by chmod. This action would fail naturally, as each of these paths does not exist. Would using single quotes help?

Single quotes treat all information contained between single quotes as exact character data in the bash shell. In practice, single quotes are extraordinarily difficult to use as variables and are not expanded. Even the backslash character is not considered special within single quotes. In our example, this fact means that a variable cannot be used in single quotes when shell scripting. As a result, single quotes aren't going to solve the problem above, so a different solution is required.

Double Quotes?

What about the use of double quotes? Can they solve the problem? Hmm, it get's a bit closer this time. Again, let's use the above list command. When assigned to a variable, multiple words will be treated as a single word. Wow, what a mouthful! Here's a better example. Let's use Adobe Acrobat 8 Professional again. When talking about single quotes, remember the fact that each word is treated separately? With double quoting, the entire word remains intact, and is not split. So, if the variable test is assigned and then echoed out using double quotes, the entire word, Adobe Acrobat 8 Professional, is used. A brief shell script snippet to further reinforce this concept:

#!/bin/bash
test="Adobe Acrobat 8 Professional"
printf "Single word is $test\n"

Note, that when these commands are entered and run that the variable is correctly displayed with printf. Without the quotes, only the word Single is printed. Does this solution work for the problem encountered on the list? Well, not quite. It certainly gets somewhat closer, as spaces are correctly preserved with double quotes, but command substitutions with variables still see each space separated entry as a separate word. If single and double quotes won't solve the problem, what will? Can the field separator be altered, and will it solve the problem?

IFS

IFS? Never heard of it before? IFS stands for Internal Field Separator. It determines how bash interprets word boundaries. By default, the three special characters, space, tab, and newline are used. Using the Adobe Acrobat 8 example, it now should be somewhat clearer as to why each is treated as a separate word. bash reads each space in the name as a field separator. With the default value for IFS, command substitution treats each space as a delimiter. This can be changed. At the top of each shell script, set the IFS variable. Any character can be the field separator, from a semi-colon, space, to a comma. The Macenterprise list came up with the following solution for the problem when a directory listing contains spaces:

   
   IFS="\n"

The author inserted this line at the beginning of the shell script. Now that the IFS separator has been reset, spaces are no longer recognized as valid field separators. The problem is now gone, and the variable is correctly assigned. It is also useful to note that resetting IFS can be used to ensure that the IFS variable has not been improperly set before any shell script is run. As a security precaution when writing a shell script, just as the PATH variable is set, make it a habit to also set the IFS variable. If input is being read in, or parsed, any spoofing or accidental resetting of this variable becomes a non-issue.

USING FIND

Let's return to the original list command that the article opened with. The original question wanted to take a list of directories and then change their permissions. Jeff McCune from The Ohio State University suggested another method on the list. He used the shell find utility to accomplish the same thing in a more efficient and unix-y way. Let's start with the command in its full form:

find /Applications -type d -maxdepth 1 -mindepth 1 -print0 | xargs -0 chmod 750

O.K., that's a bit much! As the find utility may be new to some, let's break it down into its components. The second argument, /Applications, can be any directory that find will operate on. In this case, it is looking in the Applications directory. The next argument, -type, looks for directories, as specified by the option d. After that, the next two arguments instruct the find command not to descend into any of the Application packages. It is accomplished with the maxdepth and mindepth switches. The next option is where the real magic begins. The option, print0, prints the result to standard output, terminating the output with the ASCII NUL character. It will become clear why this option is important in a second. All the results from the command are printed without a terminating newline. Try just the first part of the command before the pipe character, both with -print and -print0. Note the difference, as it is critical to this commands proper function.

Now that the find results are complete, it is time to process them using xargs. xargs takes an argument list, and processes the results with any other command line utility. Remember the use of print0. As the NUL character terminates the output, xargs must use the -0 flag. Without it, xargs does not know how to interpret the results from the find command. This technique is also generally considered faster by Unix gurus, as it only executes the chmod command once, which may not be the case for a shell loop, or by using the –exec switch with the find utility.

As you can see, the problem presented on the list is not as straightforward as one might expect. As always, there is more than one way to solve the problem, but with a full understanding of the problem's intricacies, it should now be trivial to solve. Until next month, see you on the lists!


Philip Rinehart is co-chair of the steering committee leading the Mac OS X Enterprise Project (macenterprise.org) and is the Lead Mac Analyst at Yale University. He has been using Macintosh Computers since the days of the Macintosh SE, and Mac OS X since its Developer Preview Release. Before coming to Yale, he worked as a Unix system administrator for a dot-com company. He can be reached at: philip.rinehart@yale.edu.

The MacEnterprise project is a community of IT professionals sharing information and solutions to support Macs in an enterprise. We collaborate on the deployment, management, and integration of Mac OS X client and server computers into multi-platform computing environments.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Adium 1.5.10.3 - Popular instant messagi...
Adium is a fast and free instant messaging client which supports AIM, ICQ, Jabber, MSN, Yahoo!, Google Talk, Yahoo! Japan, Bonjour, Gadu-Gadu, Novell Groupwise, SIP/SIMPLE (Text), and Lotus Sametime... Read more
CleanMyMac 3.8.1 - $39.95
CleanMyMac makes space for the things you love. Sporting a range of ingenious new features, CleanMyMac lets you safely and intelligently scan and clean your entire system, delete large, unused files... Read more
BusyCal 3.1.7 - Powerful calendar app wi...
BusyCal is an award-winning desktop calendar that combines personal productivity features for individuals with powerful calendar sharing capabilities for families and workgroups. Its unique features... Read more
Lyn 1.8.9 - Lightweight image browser an...
Lyn is a fast, lightweight image browser and viewer designed for photographers, graphic artists, and Web designers. Featuring an extremely versatile and aesthetically pleasing interface, it delivers... Read more
Tweetbot 2.5 - Popular Twitter client.
Tweetbot is a full-featured OS X Twitter client with a lot of personality. Whether it's the meticulously-crafted interface, sounds and animation, or features like multiple timelines and column views... Read more
Monolingual 1.7.8 - Remove unwanted OS X...
Monolingual is a program for removing unnecesary language resources from OS X, in order to reclaim several hundred megabytes of disk space. If you use your computer in only one (human) language, you... Read more
Dash 4.0.3 - Instant search and offline...
Dash is an API documentation browser and code snippet manager. Dash helps you store snippets of code, as well as instantly search and browse documentation for almost any API you might use (for a full... Read more
Posterino 3.3.6 - Create posters, collag...
Posterino offers enhanced customization and flexibility including a variety of new, stylish templates featuring grids of identical or odd-sized image boxes. You can customize the size and shape of... Read more
Apple Numbers 4.1.1 - Apple's sprea...
With Apple Numbers, sophisticated spreadsheets are just the start. The whole sheet is your canvas. Just add dramatic interactive charts, tables, and images that paint a revealing picture of your data... Read more
Apple Pages 6.1.1 - Apple's word pr...
Apple Pages is a powerful word processor that gives you everything you need to create documents that look beautiful. And read beautifully. It lets you work seamlessly between Mac and iOS devices, and... Read more

Latest Forum Discussions

See All

SCRAP (Games)
SCRAP 1.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0 (iTunes) Description: That day, for no apparent reason, SCRAP decided to wake up and run. He had to, because his activation was a mistake the "Factory" could... | Read more »
The Bunker (Games)
The Bunker 1.1 Device: iOS Universal Category: Games Price: $3.99, Version: 1.1 (iTunes) Description: The critically acclaimed console hit "The Bunker" comes to iOS, The groundbreaking live-action thriller adventure set in a real... | Read more »
Die With Glory (Games)
Die With Glory 1.2.0 Device: iOS Universal Category: Games Price: $2.99, Version: 1.2.0 (iTunes) Description: Die with Glory is an epic adventure game where your goal is to die in glorious fashion. You must help Sigurd, a brave old... | Read more »
Get Ike in the new Fire Emblem: Heroes u...
One of the most popular Fire Emblem characters is finally available in a new update to Nintendo'sFire Emblem: Heroes. [Read more] | Read more »
Die With Glory in a new viking adventure...
If you're a fan of classic adventure games you'll do well to pick upDie With Glory, the gorgeous new title from Cloud Castle inc. Die With Glory updatesthe gameplay of the same kind of adventure classics such asMonkey Island for modern, mobile... | Read more »
Get up to speed with everything you need...
In case you haven’t heard, MU Origin just got a colossal new update with new all-server events, battle modes, and systems making their way to the land of MU. Here’s a handy guide to everything you need to know about the latest content. [Read... | Read more »
Minimalist puzzle game, Cuts, free on iO...
If you're looking for a gorgeous puzzle experience on iOS devices, developer Gamebra.in's aesthetically interesting puzzler, Cuts, is discounted to free on the iOS App Store right now. [Read more] | Read more »
Anime tactical RPG, War of Crown, comes...
If you're looking for another tactical RPG fix to go alongside your Fire Emblem Heroes campaigns check out Gamevil's newest, anime-inspired tactics RPG, War of Crown, which comes out tomorrow. [Read more] | Read more »
Fantasy MMORPG MU Origin adds new modes,...
MU Origin, Webzen’s highly popular fantasy MMORPG is getting ready to shake things up for the second time this year, as a new update makes its way to the Google Play and App Store from today. Introducing new systems, modes, and events, the land of... | Read more »
Blizzard is looking to hire a mobile dev...
A new thread on the popular video game rumor forum, NeoGAF, uncovered an interesting job listing over at Blizzard Entertainment. It appears the studio behindStarCraft, World of WarCraft, Hearthstone,andOverwatch is looking to bring on a new hire... | Read more »

Price Scanner via MacPrices.net

Digital Paper Tablet Offers Distraction Free...
I typically spend 8-10 hours a day gazing at the screens in my laptops and iPad, as tools of my livelihood, I don’t as a rule use electronic devices for pleasure reading. I subscribe to a daily... Read more
“Today at Apple” Bringing New Educational Ses...
Apple has announced plans to launch dozens of new educational sessions next month in all 495 Apple Stores ranging in topics from photo and video to music, coding, art and design, and more. The hands-... Read more
Smart Finance Free Comprehensive Personal Fin...
Moscow-based indie developer, Alexander Survillo has announced the release and immediate availability of Smart Finance: Personal Finance, Budget & Money 1.1.4, an update to his comprehensive... Read more
12-inch 1.1GHz Retina MacBooks on sale for $1...
B&H has 12″ 1.1GHz Retina MacBooks on sale for $100 off MSRP. Shipping is free, and B&H charges NY & NJ sales tax only: - 12″ 1.1GHz Space Gray Retina MacBook: $1199.99 $100 off MSRP - 12... Read more
13-inch 2.7GHz Retina MacBook Pro on sale for...
B&H Photo has the 13″ 2.7GHz Retina MacBook Pro on sale for $130 off MSRP. Shipping is free, and B&H charges NY & NJ tax only: - 13″ 2.7GHz/128GB Retina MacBook Pro (MF839LL/A): $1169 $... Read more
15-inch 2.2GHz Retina MacBook Pros available...
B&H Photo has the 15″ 2.2GHz Retina MacBook Pro available for $200 off MSRP including free shipping plus NY & NJ sales tax only: - 15″ 2.2GHz Retina MacBook Pro (MJLQ2LL/A): $1799.99 $200 off... Read more
13-inch Touch Bar MacBook Pros on sale for up...
B&H Photo has the 2016 Apple 13″ Touch Bar MacBook Pros in stock today for up to $150 off MSRP. Shipping is free, and B&H charges NY & NJ sales tax only: - 13″ 2.9GHz/512GB Touch Bar... Read more
Apple refurbished Apple TVs available for up...
Apple has Certified Refurbished 32GB and 64GB Apple TVs available for up to $30 off the cost of new models. Apple’s standard one-year warranty is included with each model, and shipping is free: -... Read more
12-inch 1.2GHz Retina MacBooks on sale for up...
B&H has 12″ 1.2GHz Retina MacBooks on sale for up to $160 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.2GHz Space Gray Retina MacBook: $1439.99 $160 off MSRP - 12″ 1... Read more
HyperX Ships Pulsefire FPS Gaming Mouse, Winn...
Your reporter is a longtime fan of gaming mice for general purpose coomnputing use, finding them typically superior in comfort and performance. HyperX, a division of Kingston Technology Company, Inc... Read more

Jobs Board

*Apple* Mac Computer Technician - GeekHampto...
…complex computer issues over the phone and in person? GeekHampton, Long Island's Apple Premium Service Provider, is looking for you! Come work with our crew Read more
Product Manager, *Apple* Platforms - Viacom...
…Product Manager to drive the execution of its iOS and AppleTV experiences. The Apple Platform Product Manager will be a leader in our Agile/Scrum environment and Read more
Geek Squad *Apple* Master Consultation Agen...
**500662BR** **Job Title:** Geek Squad Apple Master Consultation Agent **Location Number:** 000286-Canton-Store **Job Description:** **What does a Geek Squad Read more
*Apple* Mobile Master - Best Buy (United Sta...
**500710BR** **Job Title:** Apple Mobile Master **Location Number:** 000279-North Olmsted-Store **Job Description:** **What does a Best Buy Apple Mobile Master Read more
*Apple* Engineering Specialist - CSRA (Unite...
Apple Engineering Specialist All times are in Eastern Daylight Time Requisition ID Job Locations US DC Washington DC Posted Date Category Engineering Sciences Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.