TweetFollow Us on Twitter

Quoting in the shell

Volume Number: 23 (2007)
Issue Number: 09
Column Tag: MacEnterprise

Quoting in the shell

Dealing with interesting characters in shell programming

By Philip Rinehart, Yale University

QUOTES?

This month, a rather interesting problem arose on the Macenterprise mailing list. How are quotes used and dealt with when scripting for the bash shell? It can be a particularly difficult problem, as OS X allows the use of non-standard characters and spaces in filenames. If quoting is not done properly, unexpected, and even disastrous results may occur. Anyone, remember iTunes 2.0 and destruction of hard drive data? It was a quoting problem! Thus, it is extremely important to quote things properly when shell programming, both for unexpected results, as well as properly sanitizing input.

Single Quotes?

Let's start the process by looking at the use of single quotes in bash shell scripting. The original question was about the use of a directory listing. The directory listing was then piped to a second command, which failed due to spaces in some of the directory names. The specific example:

   ls -F /Applications

Try it. Note how many Applications contain spaces in the name. If Adobe Acrobat 8 Professional were installed, any shell script attempting to perform an action on the path would fail. Why? As the shell interprets spaces as input separators, Adobe, Acrobat, 8 and Professional are all seen as individual paths. In our above example, if the chmod operation were being performed, "Adobe", "Acrobat", "8" and "Professional" would all be modified by chmod. This action would fail naturally, as each of these paths does not exist. Would using single quotes help?

Single quotes treat all information contained between single quotes as exact character data in the bash shell. In practice, single quotes are extraordinarily difficult to use as variables and are not expanded. Even the backslash character is not considered special within single quotes. In our example, this fact means that a variable cannot be used in single quotes when shell scripting. As a result, single quotes aren't going to solve the problem above, so a different solution is required.

Double Quotes?

What about the use of double quotes? Can they solve the problem? Hmm, it get's a bit closer this time. Again, let's use the above list command. When assigned to a variable, multiple words will be treated as a single word. Wow, what a mouthful! Here's a better example. Let's use Adobe Acrobat 8 Professional again. When talking about single quotes, remember the fact that each word is treated separately? With double quoting, the entire word remains intact, and is not split. So, if the variable test is assigned and then echoed out using double quotes, the entire word, Adobe Acrobat 8 Professional, is used. A brief shell script snippet to further reinforce this concept:

#!/bin/bash
test="Adobe Acrobat 8 Professional"
printf "Single word is $test\n"

Note, that when these commands are entered and run that the variable is correctly displayed with printf. Without the quotes, only the word Single is printed. Does this solution work for the problem encountered on the list? Well, not quite. It certainly gets somewhat closer, as spaces are correctly preserved with double quotes, but command substitutions with variables still see each space separated entry as a separate word. If single and double quotes won't solve the problem, what will? Can the field separator be altered, and will it solve the problem?

IFS

IFS? Never heard of it before? IFS stands for Internal Field Separator. It determines how bash interprets word boundaries. By default, the three special characters, space, tab, and newline are used. Using the Adobe Acrobat 8 example, it now should be somewhat clearer as to why each is treated as a separate word. bash reads each space in the name as a field separator. With the default value for IFS, command substitution treats each space as a delimiter. This can be changed. At the top of each shell script, set the IFS variable. Any character can be the field separator, from a semi-colon, space, to a comma. The Macenterprise list came up with the following solution for the problem when a directory listing contains spaces:

   
   IFS="\n"

The author inserted this line at the beginning of the shell script. Now that the IFS separator has been reset, spaces are no longer recognized as valid field separators. The problem is now gone, and the variable is correctly assigned. It is also useful to note that resetting IFS can be used to ensure that the IFS variable has not been improperly set before any shell script is run. As a security precaution when writing a shell script, just as the PATH variable is set, make it a habit to also set the IFS variable. If input is being read in, or parsed, any spoofing or accidental resetting of this variable becomes a non-issue.

USING FIND

Let's return to the original list command that the article opened with. The original question wanted to take a list of directories and then change their permissions. Jeff McCune from The Ohio State University suggested another method on the list. He used the shell find utility to accomplish the same thing in a more efficient and unix-y way. Let's start with the command in its full form:

find /Applications -type d -maxdepth 1 -mindepth 1 -print0 | xargs -0 chmod 750

O.K., that's a bit much! As the find utility may be new to some, let's break it down into its components. The second argument, /Applications, can be any directory that find will operate on. In this case, it is looking in the Applications directory. The next argument, -type, looks for directories, as specified by the option d. After that, the next two arguments instruct the find command not to descend into any of the Application packages. It is accomplished with the maxdepth and mindepth switches. The next option is where the real magic begins. The option, print0, prints the result to standard output, terminating the output with the ASCII NUL character. It will become clear why this option is important in a second. All the results from the command are printed without a terminating newline. Try just the first part of the command before the pipe character, both with -print and -print0. Note the difference, as it is critical to this commands proper function.

Now that the find results are complete, it is time to process them using xargs. xargs takes an argument list, and processes the results with any other command line utility. Remember the use of print0. As the NUL character terminates the output, xargs must use the -0 flag. Without it, xargs does not know how to interpret the results from the find command. This technique is also generally considered faster by Unix gurus, as it only executes the chmod command once, which may not be the case for a shell loop, or by using the –exec switch with the find utility.

As you can see, the problem presented on the list is not as straightforward as one might expect. As always, there is more than one way to solve the problem, but with a full understanding of the problem's intricacies, it should now be trivial to solve. Until next month, see you on the lists!


Philip Rinehart is co-chair of the steering committee leading the Mac OS X Enterprise Project (macenterprise.org) and is the Lead Mac Analyst at Yale University. He has been using Macintosh Computers since the days of the Macintosh SE, and Mac OS X since its Developer Preview Release. Before coming to Yale, he worked as a Unix system administrator for a dot-com company. He can be reached at: philip.rinehart@yale.edu.

The MacEnterprise project is a community of IT professionals sharing information and solutions to support Macs in an enterprise. We collaborate on the deployment, management, and integration of Mac OS X client and server computers into multi-platform computing environments.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

djay Pro 1.2 - Transform your Mac into a...
djay Pro provides a complete toolkit for performing DJs. Its unique modern interface is built around a sophisticated integration with iTunes and Spotify, giving you instant access to millions of... Read more
Stacks 2.7.2 - New way to create pages i...
Stacks is a new way to create pages in RapidWeaver. It's a plugin designed to combine drag-and-drop simplicity with the power of fluid layout. Features Fluid Layout: Stacks lets you build pages... Read more
Vitamin-R 2.30 - Personal productivity t...
Vitamin-R creates the optimal conditions for your brain to work at its best by structuring your work into short bursts of distraction-free, highly focused activity alternating with opportunities for... Read more
HoudahSpot 4.0.10 - Advanced file-search...
HoudahSpot is a powerful yet accessible desktop search software. Use HoudahSpot to locate hard-to-find files and keep frequently used files within reach. HoudahSpot builds upon Spotlight, which comes... Read more
Mac DVDRipper Pro 5.0.6 - Copy, backup,...
Mac DVDRipper Pro is the DVD backup solution that lets you protect your DVDs from scratches, save your batteries by reading your movies from your hard disk, manage your collection with just a few... Read more
Logic Pro X 10.2 - Music creation and au...
Logic Pro X is the most advanced version of Logic ever. Sophisticated new tools for professional songwriting, editing, and mixing are built around a modern interface that's designed to get creative... Read more
VMware Fusion 8.0.0 - Run Windows apps a...
VMware Fusion 8 and Fusion 8 Pro--the latest versions of its virtualization software for running Windows on a Mac without rebooting--include full support for Windows 10, OS X El Capitan, and the... Read more
MYStuff Pro 2.0.21 - Create inventories...
MYStuff Pro is the most flexible way to create detail-rich inventories for your home or small business. Add items to MYStuff by dragging and dropping existing information, uploading new images, or... Read more
VueScan 9.5.23 - Scanner software with a...
VueScan is a scanning program that works with most high-quality flatbed and film scanners to produce scans that have excellent color fidelity and color balance. VueScan is easy to use, and has... Read more
Skype 7.11.653 - Voice-over-internet pho...
Skype allows you to talk to friends, family and co-workers across the Internet without the inconvenience of long distance telephone charges. Using peer-to-peer data transmission technology, Skype... Read more

Emo Chorus (Music)
Emo Chorus 1.0.0 Device: iOS Universal Category: Music Price: $1.99, Version: 1.0.0 (iTunes) Description: Realistic Choir simulator ranging from simple Chorus emulation to full ensemble Choir with 128 members. ### introductory offer... | Read more »
Forest Spirit (Games)
Forest Spirit 1.0.5 Device: iOS Universal Category: Games Price: $2.99, Version: 1.0.5 (iTunes) Description: | Read more »
Ski Safari 2 (Games)
Ski Safari 2 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: The world's most fantastical, fun, family-friendly skiing game is back and better than ever! Play as Sven's sister Evana, share... | Read more »
Lara Croft GO (Games)
Lara Croft GO 1.0.47768 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.47768 (iTunes) Description: Lara Croft GO is a turn based puzzle-adventure set in a long-forgotten world. Explore the ruins of an ancient... | Read more »
Whispering Willows (Games)
Whispering Willows 1.23 Device: iOS Universal Category: Games Price: $4.99, Version: 1.23 (iTunes) Description: **LAUNCH SALE 50% OFF** - Whispering Willows is on sale for 50% off ($4.99) until September 9th. | Read more »
Calvino Noir (Games)
Calvino Noir 1.1 Device: iOS iPhone Category: Games Price: $3.99, Version: 1.1 (iTunes) Description: The film noir stealth game. Calvino Noir is the exploratory, sneaking adventure through the 1930s European criminal underworld.... | Read more »
Angel Sword (Games)
Angel Sword 1.0 Device: iOS Universal Category: Games Price: $6.99, Version: 1.0 (iTunes) Description: Prepare to adventure in the most epic full scale multiplayer 3D RPG for mobile! Experience amazing detailed graphics in full HD.... | Read more »
infltr - Infinite Filters (Photography)
infltr - Infinite Filters 1.0 Device: iOS Universal Category: Photography Price: $1.99, Version: 1.0 (iTunes) Description: Say goodbye to the same old boring filters and filter life as you live it - in the moment. With infltr, you... | Read more »
You'll be Able to Make Your Own BAD...
Have you been playing BADLAND so much that you've run out content? Well that won't be a problem for much longer because Frogmind is giving us a level editor in an upcoming (and free) update. [Read more] | Read more »
Magic: The Gathering - Puzzle Quest is a...
It looks like we can add Magic: The Gatehring to the list of "stuff that has a Puzzle Quest spin-off." D3 Go! and Wizards of the Coast have officially announced that the popular collectible card game will be getting it's own match-3 puzzler later... | Read more »

Price Scanner via MacPrices.net

The ‘Book Mystique Still Magic After All Thes...
This column has been called The ‘Book Mystique for what has been a 15 year run so far, and I have no interest in changing the name. The reference is to what I’ve observed is a the near magical... Read more
15-inch Retina MacBook Pros on sale for up to...
B&H Photo has 2015 15″ Retina MacBook Pros on sale for up to $184 off MSRP including free shipping plus NY sales tax only: - 15″ 2.2GHz Retina MacBook Pro: $1815 $184 off - 15″ 2.5GHz Retina... Read more
11-inch 128GB MacBook Air on sale for $799, s...
Adorama has the 11″ 1.6GHz/128GB MacBook Air on sale for $799.99 including free shipping plus NY & NJ sales tax only. Their price is $100 off MSRP, and it’s the lowest price available for this... Read more
Apple Unlikely To Gain Market Share From Andr...
According to a new mobile phone forecast from the International Data Corporation (IDC) Worldwide Quarterly Mobile Phone Tracker, smartphone shipments are expected to grow 10.4% in 2015 to 1.44... Read more
Save up to $600 with Apple refurbished Mac Pr...
The Apple Store has Apple Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The... Read more
Kainos Platform Helps Healthcare Organization...
UK-based provider of IT services Kainos Group plc (Kainos) has announced the launch of its Kainos Evolve Mobile-Enabled Healthcare platform, which is designed to give healthcare providers and... Read more
Mac Pros on sale for up to $300 off MSRP
B&H Photo has Mac Pros on sale for up to $300 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3699, $... Read more
21-inch iMacs on sale for up to $120 off MSRP
B&H Photo has 21″ iMacs on sale for up to $100 off MSRP including free shipping plus NY sales tax only: - 21″ 1.4GHz iMac: $1029.99 $70 off - 21″ 2.7GHz iMac: $1199.99 $100 off - 21″ 2.9GHz iMac... Read more
12-inch 1.1GHz Gold MacBook on sale for $1199...
B&H Photo has the 12″ 1.1GHz Gold Retina MacBook on sale for $1199 including free shipping plus NY sales tax only. Their price is $100 off MSRP, and it’s the lowest price available for this model... Read more
for Meetings for iPad Helps Manage And Publis...
for Meetings is a meetings app for iPad that makes it easy to manage and publish your meeting notes. for Meetings for iPad enables recording of notes, agenda items, actions, remarks and more, and... Read more

Jobs Board

*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Software QA Engineer, *Apple* Pay Security...
Changing the world is all in a day039s work at Apple . If you love innovation, here039s your chance to make a career of it. You039ll work hard. But the job comes with Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.