TweetFollow Us on Twitter

Mac in the Shell: Dropbox in the Shell

Volume Number: 26
Issue Number: 03
Column Tag: Mac in the Shell

Mac in the Shell: Dropbox in the Shell

Or, Integrating Dropbox and bash!

by Edward Marczak

Welcome

Last month, I promised to get back to some shell topics. This month, I'll do just that. I've become a pretty big fan of Dropbox (http://getdropbox.com). However, it's rare that I access it directly in the GUI. This makes simplifying my life with Dropbox in a shell a pretty high priority. This article will look at how I've integrated Dropbox and my bash shell environment.

Why Dropbox?

If you're unfamiliar with Dropbox, it's a cloud-based storage solution that two-way syncs between the Dropbox servers and a folder on any machines that you've installed the Dropbox client on. Of course, like Apple's iDisk, you provide credentials, so it only syncs your files. Dropbox is cross-platform and has clients for Mac, Linux and Windows. Dropbox allows you to define where your local Dropbox folder is located. Unlike iDisk, which tries to act like it's magic and shield you from its workings, Dropbox acts much more naturally. You are asked to define a folder that is your Dropbox. That's it. From then on, anything that you put in that folder is kept in sync.

I don't want to spend too much time on why I love Dropbox. You're either using Dropbox already and know how good it is, you're thinking of trying it (go try it now at the link mentioned in the intro) or have decided that you don't care for it. If you're in the latter camp, perhaps this article can give you some ideas for similar challenges you may face.

I work on different computers all the time: I have several at work and a few at home. This was quite a challenge for me initially since I had long used the traditional one-laptop-for-everything setup. While I don't need identical environments on every machine I use, there are certain aspects I like to configure in a similar manner. There are also certain tools that I'd like to have access to and certain files that I want to tote about. I have a MobileMe account, so, why don't I just use iDisk for this?

Initially, I tried to use iDisk, but the first barrier was that iDisk is just unbearably slow. Even with a pretty fast connection to the Internet, sync times were just unacceptable. The sync routine for iDisk is also a bit messy. It would often complain of file conflicts for no reason that I could figure out. This is disruptive, as the conflict dialog just pops up over whatever you're doing. Constantly. Perhaps the biggest deal-breaker is that iDisk does not support symlinks. Well, if you keep a local copy of your iDisk, symlinks work fine locally, but they will not be synced up to MobileMe, and therefore won't come down to any other computers syncing to the same iDisk.

There were even more issues, but that was enough for me to start looking for alternatives. A number of friends were using Dropbox, so I gave it a try. Well, everything that I just mentioned is wrong with iDisk is just fine in Dropbox. First, it's fast. Second, it feels better integrated into Mac OS X than just about any third-party product that I've used, and possibly even more so than iDisk. It's absolutely invisible. Finally, my favorite: symlinks work just fine (even though I've modified my workflow and not using them much anymore).

Shell Integration

When I said earlier that Dropbox is very 'natural' and behaves as one would expect, that includes how it responds in a shell. While iDisk mounts a local diskimage, you can't define where that is located. (Well, OK...the disk image is stored in ~/Library/FileSync/(FS_identifier)/(MobileMe_username)_iDisk.sparsebundle and I suppose you could symlink it somewhere. But that doesn't stop the mount from appearing in /Volumes). The Dropbox folder, on the other hand, acts just like any folder (directory) on the system. You can view it in the GUI, or you can list it in a shell. You can 'cd' to it in a shell and manipulate it. Like you'd expect.

That all seems pretty good, right? How much more would you expect to do? Here are some ideas.

Where's Dropbox?

Since Dropbox, as a solution is intended for multiple computers, each computer needs its own client install. Nothing says that the location of the Dropbox folder must be the same on each. By default, the Dropbox installer offers to put the defined folder in your home directory. However, I often move that into /Users/Shared or other locations. I've certainly developed variances among my installs. This is pretty easy to solve using a common shell convention: environment variables.

A review on environment variables: like most shells, bash supports session-persistent in-memory variables. Variables can store string or numeric values. There's even an array type, but it's seldom used and fairly tricky ugly to handle. (Seriously, if you're trying to use arrays in bash you should really consider moving to a scripting language that had arrays in mind from the beginning. Ruby and Python are two excellent examples).

Two important characteristics about environment variables: 1) They're available in all shell contexts. That is, they're defined in each shell you create, and any that fork from that shell. 2) When the $ character is encountered, the shell performs variable expansion. This lets us use variables that we've defined to build up other strings and commands.

The first think that I want to know is this: where is the Dropbox folder on the machine I'm using? Conveniently, Dropbox stores all installation information in an sqlite3 database in a user's home directory. One of the rows of the database is the path to the current Dropbox folder. Excellent. Mac OS X loves sqlite3 and has the command-line binary built-in. Extracting the value of the path is easy:

sqlite3 ~/.dropbox/dropbox.db 'select value from config 
where key="dropbox_path"'
Vi9Wb2x1bWVzL2bbhWVzL1VzZXJzL1YoMTMlZC9nZXJtERMyb3Bib3gKcDEKLg==

Oh, what's this? Unsurprisingly, this value is not stored as plain text. Nor, though, is it encrypted. It's base64 encoded and 'pickled' for serialization. Looks like the Dropbox developers are fans of Python. Python supports a serialization library called 'pickle'. This causes us to run into a small, but surmountable issue: there are no tools that allow us to directly unpickle an object completely within bash. I wrote a small utility in Python that gets me all of this information, but I promised to focus on the shell only, and this is a good opportunity to explore more tools.

First, we'll use the sqlite3 command to get the information from the database. As shown above, the command needs to know which database to read and the SQL command to run against the database. The HOME variable is defined for us in a shell environment, and points to the home directory of the current user. So, the command I'll recommend (and shown in the complete example) uses the ${HOME} variable over the '~' character.

Next, we need to base64 decode the value we get out of the database. OS X doesn't have a single, explicit command that performs a base64 decode. Again, you could use a Python or Perl one-liner, but I prefer to use the multi-faceted openssl binary, which contains a base64 decode routine.

One the value is base64 decoded, you'll notice some extra characters at the beginning of the output, and one on a separate line. This is a result of the pickle serialization. Let's get rid of the separate, second line right away. To do this, I use head. The head utility is a counterpart to tail. The former will display the first n number of lines from the beginning of a file, while the latter will display the last n lines from the end of a file. We only want the first line, so, we pass that information in as an argument. Finally, there's that one extra character up front. We can trim that out with the cut command. We simply tell cut that we only want from character (the -c switch) 2 onward.

If you remember pipes, they come in handy here. A pipe connects the output of one binary to the input of another. Often, programs operate on files on-disk. For example, the head utility can be used like this:

head -3 filename.txt

That will display the first three lines of the file filename.txt. Well, Unix treats just about everything as a file-even input and output. In the case of using a pipe, you're taking the standard output (represented as a file as /dev/stdout) of one program and making it the input (/dev/stdin) of another application (one that reads standard in. Not all do).

Stringing it all together, the final command looks like this:

sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where 
key="dropbox_path"' | openssl base64 -d | head -1 | cut -c 2-

This displays where the current user's Dropbox folder is. More than display it, we want to store it for future use. Here's what I'd put in my .bash_profile script to accomplish this:

# Setup Dropbox Location - used in scripts
DB=$(sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where key="dropbox_path"'
| openssl base64 -d | head -1 | cut -c 2-)

This takes advantage of bash's command substitution. The $(COMMAND) construct allows bash to substitute COMMAND with its output. A simple example:

$ START_TIME=$(date)
$ echo $START_TIME
Fri Mar 12 08:31:31 EST 2010

In the Dropbox example, we're assigning the output of our string of commands-which ultimately outputs the Dropbox path-to the variable DB. From that point forward, we can reference this path as ${DB}

What's This All Good For?

Now we have the local Dropbox folder path. Whoopee. Where can you do anything with that? In other scripts, of course. I actually have several directories in my home directory that are actually stored in my Dropbox folder. This is possible thanks to the magic of symlinks.

A symlink is just a pointer to a file on disk. The ln command is used to create links. Long time Mac users can equate this to a Finder alias. But they're a little different. In essence, a symlink is just another name for a file. Mac OS X provides two kinds of links: hard and symbolic. A hard link effectively is the linked file. An example will make this clear:

$ echo "This is a test" > linktext.txt
$ ln linktext.txt hardlink.txt
$ rm linktext.txt 
$ cat hardlink.txt 
This is a test

What just happened here?

First, a file is created by echo-ing the text "This is a test" into linktext.txt. Then, the ln command is used to create a hard link, named hardlink.txt. Now, the name "hardlink.txt" uses the same inode as linktext.txt, making them the same file. To prove this, in the next step removes the original file. But there's still a pointer to that data on the filesystem: our hardlink, hardlink.txt. To prove this, we can use cat to dump the contents of hardlink.txt. Look at that, it's the same as our original file that we "deleted." Only when we remove hardlink.txt will this directory entry be freed up.

A symbolic link, used much more frequently, is a separate entry on disk that points to the original file. Unlike a hard link, deleting the original file will free the directory entry. This is much more akin to a Finder alias. However, a Finder alias is only understood by the Finder, and not by shell tools. A symlink is much more flexible, being understood by both the Finder and the shell. Let's look at an example:

$ echo "This is a test" > linktext.txt
$ ln -s linktext.txt symlink.txt
$ ls -l
total 16
-rw-r-r- 1 erm admin 15 Mar 12 18:36 linktext.txt
lrwxr-xr-x 1 erm admin 12 Mar 12 18:36 symlink.txt -> linktext.txt
$ cat symlink.txt
This is a test
$ rm linktext.txt 
$ cat symlink.txt 
cat: symlink.txt: No such file or directory

You'll see that unlike hard links, a symbolic link is separate from the original file. In this example, we create a file using echo to redirect text, then create a symbolic link to it. Note that we use the -s switch with ln. By default, ln creates a hard link. The -s switch instructs ln to create a symbolic link. If we try the same delete-and-use-it test that we did with the hard link, we're going to be disappointed. As shown, after removing the original file, the symlink points to nothing.

While it may seem the hard links are better in some respects, symlinks are actually a little more flexible. Since inodes are only unique to a given file system, hard links cannot span across file systems. Another limitation of hard links is that they can only point to files, not directories. A common use of symlinks is to use a common name to point to the latest distribution directory of an application or web directory. For example:

web-site-1.0/
web-site-2.0/
htdocs -> web-site-1.0

In this case, web-site-1.0 is the current site. Once web-site-2.0 is complete and ready to become the main site, all that needs to be done is to point the symlink to the new directory.

Other Ways to Integrate

Now that I have these basics out of the way and I have easy access to my Dropbox folder and a variable with the Dropbox folder path, what do I use it for? First, I have a script that configures a new machine to my liking.

In my Dropbox folder, I store my ~/bin directory. The idea is that I can symlink to it from my home directory. (I've actually changed how I do this-read until the end of this article to read how I now handle this). Inside of ~/bin, I keep a setup script that configures a new machine for my use. One thing that this script relies on: the $DB variable-it needs to know where I've configured the Dropbox folder to be. On (most) any new machine, my workflow is something like this:

1. Create login account

2. Install Dropbox

3. Run ${DB}/bin/setup_new.sh - the setup script

What does this setup script do? I'll just generalize here as the script is incredibly specific to the way I work. However, the important thing is that it can bootstrap my environment just from the Dropbox folder.

First, it symlinks ~/bin folder back to itself. Since we already know the path, this is pretty easy:

ln -s ${DB}/bin ${HOME}/bin

A nicer way to handle this involves some logic and error checking:

#!/bin/bash
# Setup links to bin folder
ERROR=0
if [ -d ${HOME}/bin ]; then
  mv ${HOME}/bin ${HOME}/bin.old
  if [ $? != 0 ]; then
    echo "ERROR: Could not move current ~/bin directory"
    ERROR=1
  fi
fi
if [ -f ${HOME}/bin ]; then
  echo "ERROR: ~/bin already exists - not touching"
  ERROR=1
fi
if [[ $ERROR == 0 ]]; then
  echo "Linking ${HOME}/bin -> ${DB}/bin"
  ln -s ${DB}/bin ${HOME}/bin
fi

Conditional statements will be looked at in more detail in a future column.

The second thing my setup_new.sh script does is to call another script that runs several defaults commands that tweak things just the way I like them. (Yes, I like seeing all files in the Finder, and no, I do not want a warning when I empty the trash). I keep it in s separate script just because I like the modularity and possibility of easily running it without needing to run the entire setup_main.sh script.

Summary and Epilogue

This article demonstrates one way to integrate your shell experience with a perhaps not-so-obvious GUI element. In this case, once your shell knows where the Dropbox folder is, it can perform any manner of syncing action. Which reminds me: I said I'd talk a little bit about how I'm doing things differently now.

I did start off with the same system that I mention here-just symlinking files and folders over to certain locations in my Dropbox folder. In fact, I still do that with certain files and folders, but have begun a hybrid strategy. For my ~/bin and ~/dev folders, I just store my git repository for each on Dropbox. This way, the new machine setup scrip can just check out bin and dev into the new home folder. Any changes in ~/bin and ~/dev can get checked in when ready. Those changes will then be staged on Dropbox, ready for checkout by any other of my Dropbox clients.

If that sounds a little convoluted, well, it works really well for certain workflows, and not so much for others. That's why I have a hybrid approach. However, even files and folders that are symlinked directly into Dropbox are kept under version control (git).

Media of the month: "User Interface Design for Programmers" by Joel Spolsky (http://apress.com/book/view/1893115941). This is a bit of an oldie, but still a goodie. This book is a great read to get you outside of the tech mindset to help you design applications so they have an interface that humans can use.

See you next month with more back-to-the-shell topics.


Ed Marczak is the Executive Editor of MacTech Magazine. He has written the Mac in the Shell column since 2004.

 
AAPL
$99.02
Apple Inc.
+1.35
MSFT
$43.97
Microsoft Corpora
-0.53
GOOG
$590.60
Google Inc.
+1.58

MacTech Search:
Community Search:

Software Updates via MacUpdate

OS X Yosemite Wallpaper 1.0 - Desktop im...
OS X Yosemite Wallpaper is the gorgeous new background image for Apple's upcoming OS X 10.10 Yosemite. This wallpaper is available for all screen resolutions with a source file that measures 5,418... Read more
Acorn 4.4 - Bitmap image editor. (Demo)
Acorn is a new image editor built with one goal in mind - simplicity. Fast, easy, and fluid, Acorn provides the options you'll need without any overhead. Acorn feels right, and won't drain your bank... Read more
Bartender 1.2.20 - Organize your menu ba...
Bartender lets you organize your menu bar apps. Features: Lets you tidy your menu bar apps how you want. See your menu bar apps when you want. Hide the apps you need to run, but do not need to... Read more
TotalFinder 1.6.2 - Adds tabs, hotkeys,...
TotalFinder is a universally acclaimed navigational companion for your Mac. Enhance your Mac's Finder with features so smart and convenient, you won't believe you ever lived without them. Tab-based... Read more
Vienna 3.0.0 RC 2 :be5265e: - RSS and At...
Vienna is a freeware and Open-Source RSS/Atom newsreader with article storage and management via a SQLite database, written in Objective-C and Cocoa, for the OS X operating system. It provides... Read more
VLC Media Player 2.1.5 - Popular multime...
VLC Media Player is a highly portable multimedia player for various audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, MP3, OGG, ...) as well as DVDs, VCDs, and various streaming protocols. It... Read more
Default Folder X 4.6.7 - Enhances Open a...
Default Folder X attaches a toolbar to the right side of the Open and Save dialogs in any OS X-native application. The toolbar gives you fast access to various folders and commands. You just click... Read more
TinkerTool 5.3 - Expanded preference set...
TinkerTool is an application that gives you access to additional preference settings Apple has built into Mac OS X. This allows to activate hidden features in the operating system and in some of the... Read more
Audio Hijack Pro 2.11.0 - Record and enh...
Audio Hijack Pro drastically changes the way you use audio on your computer, giving you the freedom to listen to audio when you want and how you want. Record and enhance any audio with Audio Hijack... Read more
Intermission 1.1.1 - Pause and rewind li...
Intermission allows you to pause and rewind live audio from any application on your Mac. Intermission will buffer up to 3 hours of audio, allowing users to skip through any assortment of audio... Read more

Latest Forum Discussions

See All

Traps n’ Gemstones Review
Traps n’ Gemstones Review By Campbell Bird on July 28th, 2014 Our Rating: :: CASTLEVANIA JONESUniversal App - Designed for iPhone and iPad Fight mummies, dig tunnels, and ride a runaway minecart to discover ancient secrets in this... | Read more »
The Phantom PI Mission Apparition Review
The Phantom PI Mission Apparition Review By Jordan Minor on July 28th, 2014 Our Rating: :: GHOSTS BUSTEDUniversal App - Designed for iPhone and iPad The Phantom PI is an exceedingly clever and well-crafted adventure game.   | Read more »
More Stubies Are Coming Your Way in a Ne...
More Stubies Are Coming Your Way in a New Update Posted by Jessica Fisher on July 28th, 2014 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
The Great Prank War Review
The Great Prank War Review By Nadia Oxford on July 28th, 2014 Our Rating: :: PRANKING IS SERIOUS BUSINESSUniversal App - Designed for iPhone and iPad Though short, The Great Prank War offers an interesting and fun mix of action and... | Read more »
Marvel Contest of Champions Announced at...
Marvel Contest of Champions Announced at Comic-Con Posted by Jennifer Allen on July 28th, 2014 [ permalink ] Announced over the weekend at San Diego Comic-Con was the fairly exciting looking Marvel Contest of Champions. | Read more »
Teenage Mutant Ninja Turtles Review
Teenage Mutant Ninja Turtles Review By Jennifer Allen on July 28th, 2014 Our Rating: :: DULL SWIPINGUniversal App - Designed for iPhone and iPad The pizza power is weak when it comes to this Teenage Mutant Ninja Turtles game.   | Read more »
Exploration Focused Puzzle Game Beatbudd...
Exploration Focused Puzzle Game Beatbuddy Set to Make Transition from PC to iOS this September Posted by Jennifer Allen on July 28th, 2014 [ permalink ] | Read more »
PlanetHD
PlanetHD By Nadia Oxford on July 28th, 2014 Our Rating: :: SPACE MADNESSUniversal App - Designed for iPhone and iPad PlanetHD will keep players busy for a while, though its unpredictable physics are a handful to deal with.   | Read more »
This Week at 148Apps: July 21-25, 2014
Another Week of Expert App Reviews   At 148Apps, we help you sort through the great ocean of apps to find the ones we think you’ll like and the ones you’ll need. Our top picks become Editor’s Choice, our stamp of approval for apps with that little... | Read more »
Reddme for iPhone - The Reddit Client (...
Reddme for iPhone - The Reddit Client 1.0 Device: iOS iPhone Category: News Price: $.99, Version: 1.0 (iTunes) Description: Reddme for iPhone is an iOS 7-optimized Reddit client that offers a refreshing new way to experience Reddit... | Read more »

Price Scanner via MacPrices.net

13-inch 2.5GHz MacBook Pro on sale for $1099,...
Best Buy has the 13″ 2.5GHz MacBook Pro available for $1099.99 on their online store. Choose free shipping or free instant local store pickup (if available). Their price is $100 off MSRP. Price is... Read more
Roundup of Apple refurbished MacBook Pros, th...
The Apple Store has Apple Certified Refurbished 13″ and 15″ MacBook Pros available for up to $400 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free. Their prices... Read more
Record Mac Shipments In Q2/14 Confound Analys...
A Seeking Alpha Trefis commentary notes that Apple’s fiscal Q3 2014 results released July 22, beat market predictions on earnings, although revenues were slightly lower than anticipated. Apple’s Mac’... Read more
Intel To Launch Core M Silicon For Use In Not...
Digitimes’ Monica Chen and Joseph Tsai, report that Intel will launch 14nm-based Core M series processors specifically for use in fanless notebook/tablet 2-in-1 models in Q4 2014, with many models to... Read more
Apple’s 2014 Back to School promotion: $100 g...
 Apple’s 2014 Back to School promotion includes a free $100 App Store Gift Card with the purchase of any new Mac (Mac mini excluded), or a $50 Gift Card with the purchase of an iPad or iPhone,... Read more
iMacs on sale for $150 off MSRP, $250 off for...
Best Buy has iMacs on sale for up to $160 off MSRP for a limited time. Choose free home shipping or free instant local store pickup (if available). Prices are valid for online orders only, in-store... Read more
Mac minis on sale for $100 off MSRP, starting...
Best Buy has Mac minis on sale for $100 off MSRP. Choose free shipping or free instant local store pickup. Prices are for online orders only, in-store prices may vary: 2.5GHz Mac mini: $499.99 2.3GHz... Read more
Global Tablet Market Grows 11% in Q2/14 Notwi...
Worldwide tablet sales grew 11.0 percent year over year in the second quarter of 2014, with shipments reaching 49.3 million units according to preliminary data from the International Data Corporation... Read more
New iPhone 6 Models to Have Staggered Release...
Digitimes’ Cage Chao and Steve Shen report that according to unnamed sources in Apple’s upstream iPhone supply chain, the new 5.5-inch iPhone will be released several months later than the new 4.7-... Read more
New iOS App Helps People Feel Good About thei...
Mobile shoppers looking for big savings at their favorite stores can turn to the Goodshop app, a new iOS app with the latest coupons and deals at more than 5,000 online stores. In addition to being a... Read more

Jobs Board

Sr Software Lead Engineer, *Apple* Online S...
Sr Software Lead Engineer, Apple Online Store Publishing Systems Keywords: Company: Apple Job Code: E3PCAK8MgYYkw Location (City or ZIP): Santa Clara Status: Full Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
Sr. Product Leader, *Apple* Store Apps - Ap...
**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.