TweetFollow Us on Twitter

Mac in the Shell: Dropbox in the Shell

Volume Number: 26
Issue Number: 03
Column Tag: Mac in the Shell

Mac in the Shell: Dropbox in the Shell

Or, Integrating Dropbox and bash!

by Edward Marczak

Welcome

Last month, I promised to get back to some shell topics. This month, I'll do just that. I've become a pretty big fan of Dropbox (http://getdropbox.com). However, it's rare that I access it directly in the GUI. This makes simplifying my life with Dropbox in a shell a pretty high priority. This article will look at how I've integrated Dropbox and my bash shell environment.

Why Dropbox?

If you're unfamiliar with Dropbox, it's a cloud-based storage solution that two-way syncs between the Dropbox servers and a folder on any machines that you've installed the Dropbox client on. Of course, like Apple's iDisk, you provide credentials, so it only syncs your files. Dropbox is cross-platform and has clients for Mac, Linux and Windows. Dropbox allows you to define where your local Dropbox folder is located. Unlike iDisk, which tries to act like it's magic and shield you from its workings, Dropbox acts much more naturally. You are asked to define a folder that is your Dropbox. That's it. From then on, anything that you put in that folder is kept in sync.

I don't want to spend too much time on why I love Dropbox. You're either using Dropbox already and know how good it is, you're thinking of trying it (go try it now at the link mentioned in the intro) or have decided that you don't care for it. If you're in the latter camp, perhaps this article can give you some ideas for similar challenges you may face.

I work on different computers all the time: I have several at work and a few at home. This was quite a challenge for me initially since I had long used the traditional one-laptop-for-everything setup. While I don't need identical environments on every machine I use, there are certain aspects I like to configure in a similar manner. There are also certain tools that I'd like to have access to and certain files that I want to tote about. I have a MobileMe account, so, why don't I just use iDisk for this?

Initially, I tried to use iDisk, but the first barrier was that iDisk is just unbearably slow. Even with a pretty fast connection to the Internet, sync times were just unacceptable. The sync routine for iDisk is also a bit messy. It would often complain of file conflicts for no reason that I could figure out. This is disruptive, as the conflict dialog just pops up over whatever you're doing. Constantly. Perhaps the biggest deal-breaker is that iDisk does not support symlinks. Well, if you keep a local copy of your iDisk, symlinks work fine locally, but they will not be synced up to MobileMe, and therefore won't come down to any other computers syncing to the same iDisk.

There were even more issues, but that was enough for me to start looking for alternatives. A number of friends were using Dropbox, so I gave it a try. Well, everything that I just mentioned is wrong with iDisk is just fine in Dropbox. First, it's fast. Second, it feels better integrated into Mac OS X than just about any third-party product that I've used, and possibly even more so than iDisk. It's absolutely invisible. Finally, my favorite: symlinks work just fine (even though I've modified my workflow and not using them much anymore).

Shell Integration

When I said earlier that Dropbox is very 'natural' and behaves as one would expect, that includes how it responds in a shell. While iDisk mounts a local diskimage, you can't define where that is located. (Well, OK...the disk image is stored in ~/Library/FileSync/(FS_identifier)/(MobileMe_username)_iDisk.sparsebundle and I suppose you could symlink it somewhere. But that doesn't stop the mount from appearing in /Volumes). The Dropbox folder, on the other hand, acts just like any folder (directory) on the system. You can view it in the GUI, or you can list it in a shell. You can 'cd' to it in a shell and manipulate it. Like you'd expect.

That all seems pretty good, right? How much more would you expect to do? Here are some ideas.

Where's Dropbox?

Since Dropbox, as a solution is intended for multiple computers, each computer needs its own client install. Nothing says that the location of the Dropbox folder must be the same on each. By default, the Dropbox installer offers to put the defined folder in your home directory. However, I often move that into /Users/Shared or other locations. I've certainly developed variances among my installs. This is pretty easy to solve using a common shell convention: environment variables.

A review on environment variables: like most shells, bash supports session-persistent in-memory variables. Variables can store string or numeric values. There's even an array type, but it's seldom used and fairly tricky ugly to handle. (Seriously, if you're trying to use arrays in bash you should really consider moving to a scripting language that had arrays in mind from the beginning. Ruby and Python are two excellent examples).

Two important characteristics about environment variables: 1) They're available in all shell contexts. That is, they're defined in each shell you create, and any that fork from that shell. 2) When the $ character is encountered, the shell performs variable expansion. This lets us use variables that we've defined to build up other strings and commands.

The first think that I want to know is this: where is the Dropbox folder on the machine I'm using? Conveniently, Dropbox stores all installation information in an sqlite3 database in a user's home directory. One of the rows of the database is the path to the current Dropbox folder. Excellent. Mac OS X loves sqlite3 and has the command-line binary built-in. Extracting the value of the path is easy:

sqlite3 ~/.dropbox/dropbox.db 'select value from config 
where key="dropbox_path"'
Vi9Wb2x1bWVzL2bbhWVzL1VzZXJzL1YoMTMlZC9nZXJtERMyb3Bib3gKcDEKLg==

Oh, what's this? Unsurprisingly, this value is not stored as plain text. Nor, though, is it encrypted. It's base64 encoded and 'pickled' for serialization. Looks like the Dropbox developers are fans of Python. Python supports a serialization library called 'pickle'. This causes us to run into a small, but surmountable issue: there are no tools that allow us to directly unpickle an object completely within bash. I wrote a small utility in Python that gets me all of this information, but I promised to focus on the shell only, and this is a good opportunity to explore more tools.

First, we'll use the sqlite3 command to get the information from the database. As shown above, the command needs to know which database to read and the SQL command to run against the database. The HOME variable is defined for us in a shell environment, and points to the home directory of the current user. So, the command I'll recommend (and shown in the complete example) uses the ${HOME} variable over the '~' character.

Next, we need to base64 decode the value we get out of the database. OS X doesn't have a single, explicit command that performs a base64 decode. Again, you could use a Python or Perl one-liner, but I prefer to use the multi-faceted openssl binary, which contains a base64 decode routine.

One the value is base64 decoded, you'll notice some extra characters at the beginning of the output, and one on a separate line. This is a result of the pickle serialization. Let's get rid of the separate, second line right away. To do this, I use head. The head utility is a counterpart to tail. The former will display the first n number of lines from the beginning of a file, while the latter will display the last n lines from the end of a file. We only want the first line, so, we pass that information in as an argument. Finally, there's that one extra character up front. We can trim that out with the cut command. We simply tell cut that we only want from character (the -c switch) 2 onward.

If you remember pipes, they come in handy here. A pipe connects the output of one binary to the input of another. Often, programs operate on files on-disk. For example, the head utility can be used like this:

head -3 filename.txt

That will display the first three lines of the file filename.txt. Well, Unix treats just about everything as a file-even input and output. In the case of using a pipe, you're taking the standard output (represented as a file as /dev/stdout) of one program and making it the input (/dev/stdin) of another application (one that reads standard in. Not all do).

Stringing it all together, the final command looks like this:

sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where 
key="dropbox_path"' | openssl base64 -d | head -1 | cut -c 2-

This displays where the current user's Dropbox folder is. More than display it, we want to store it for future use. Here's what I'd put in my .bash_profile script to accomplish this:

# Setup Dropbox Location - used in scripts
DB=$(sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where key="dropbox_path"'
| openssl base64 -d | head -1 | cut -c 2-)

This takes advantage of bash's command substitution. The $(COMMAND) construct allows bash to substitute COMMAND with its output. A simple example:

$ START_TIME=$(date)
$ echo $START_TIME
Fri Mar 12 08:31:31 EST 2010

In the Dropbox example, we're assigning the output of our string of commands-which ultimately outputs the Dropbox path-to the variable DB. From that point forward, we can reference this path as ${DB}

What's This All Good For?

Now we have the local Dropbox folder path. Whoopee. Where can you do anything with that? In other scripts, of course. I actually have several directories in my home directory that are actually stored in my Dropbox folder. This is possible thanks to the magic of symlinks.

A symlink is just a pointer to a file on disk. The ln command is used to create links. Long time Mac users can equate this to a Finder alias. But they're a little different. In essence, a symlink is just another name for a file. Mac OS X provides two kinds of links: hard and symbolic. A hard link effectively is the linked file. An example will make this clear:

$ echo "This is a test" > linktext.txt
$ ln linktext.txt hardlink.txt
$ rm linktext.txt 
$ cat hardlink.txt 
This is a test

What just happened here?

First, a file is created by echo-ing the text "This is a test" into linktext.txt. Then, the ln command is used to create a hard link, named hardlink.txt. Now, the name "hardlink.txt" uses the same inode as linktext.txt, making them the same file. To prove this, in the next step removes the original file. But there's still a pointer to that data on the filesystem: our hardlink, hardlink.txt. To prove this, we can use cat to dump the contents of hardlink.txt. Look at that, it's the same as our original file that we "deleted." Only when we remove hardlink.txt will this directory entry be freed up.

A symbolic link, used much more frequently, is a separate entry on disk that points to the original file. Unlike a hard link, deleting the original file will free the directory entry. This is much more akin to a Finder alias. However, a Finder alias is only understood by the Finder, and not by shell tools. A symlink is much more flexible, being understood by both the Finder and the shell. Let's look at an example:

$ echo "This is a test" > linktext.txt
$ ln -s linktext.txt symlink.txt
$ ls -l
total 16
-rw-r-r- 1 erm admin 15 Mar 12 18:36 linktext.txt
lrwxr-xr-x 1 erm admin 12 Mar 12 18:36 symlink.txt -> linktext.txt
$ cat symlink.txt
This is a test
$ rm linktext.txt 
$ cat symlink.txt 
cat: symlink.txt: No such file or directory

You'll see that unlike hard links, a symbolic link is separate from the original file. In this example, we create a file using echo to redirect text, then create a symbolic link to it. Note that we use the -s switch with ln. By default, ln creates a hard link. The -s switch instructs ln to create a symbolic link. If we try the same delete-and-use-it test that we did with the hard link, we're going to be disappointed. As shown, after removing the original file, the symlink points to nothing.

While it may seem the hard links are better in some respects, symlinks are actually a little more flexible. Since inodes are only unique to a given file system, hard links cannot span across file systems. Another limitation of hard links is that they can only point to files, not directories. A common use of symlinks is to use a common name to point to the latest distribution directory of an application or web directory. For example:

web-site-1.0/
web-site-2.0/
htdocs -> web-site-1.0

In this case, web-site-1.0 is the current site. Once web-site-2.0 is complete and ready to become the main site, all that needs to be done is to point the symlink to the new directory.

Other Ways to Integrate

Now that I have these basics out of the way and I have easy access to my Dropbox folder and a variable with the Dropbox folder path, what do I use it for? First, I have a script that configures a new machine to my liking.

In my Dropbox folder, I store my ~/bin directory. The idea is that I can symlink to it from my home directory. (I've actually changed how I do this-read until the end of this article to read how I now handle this). Inside of ~/bin, I keep a setup script that configures a new machine for my use. One thing that this script relies on: the $DB variable-it needs to know where I've configured the Dropbox folder to be. On (most) any new machine, my workflow is something like this:

1. Create login account

2. Install Dropbox

3. Run ${DB}/bin/setup_new.sh - the setup script

What does this setup script do? I'll just generalize here as the script is incredibly specific to the way I work. However, the important thing is that it can bootstrap my environment just from the Dropbox folder.

First, it symlinks ~/bin folder back to itself. Since we already know the path, this is pretty easy:

ln -s ${DB}/bin ${HOME}/bin

A nicer way to handle this involves some logic and error checking:

#!/bin/bash
# Setup links to bin folder
ERROR=0
if [ -d ${HOME}/bin ]; then
  mv ${HOME}/bin ${HOME}/bin.old
  if [ $? != 0 ]; then
    echo "ERROR: Could not move current ~/bin directory"
    ERROR=1
  fi
fi
if [ -f ${HOME}/bin ]; then
  echo "ERROR: ~/bin already exists - not touching"
  ERROR=1
fi
if [[ $ERROR == 0 ]]; then
  echo "Linking ${HOME}/bin -> ${DB}/bin"
  ln -s ${DB}/bin ${HOME}/bin
fi

Conditional statements will be looked at in more detail in a future column.

The second thing my setup_new.sh script does is to call another script that runs several defaults commands that tweak things just the way I like them. (Yes, I like seeing all files in the Finder, and no, I do not want a warning when I empty the trash). I keep it in s separate script just because I like the modularity and possibility of easily running it without needing to run the entire setup_main.sh script.

Summary and Epilogue

This article demonstrates one way to integrate your shell experience with a perhaps not-so-obvious GUI element. In this case, once your shell knows where the Dropbox folder is, it can perform any manner of syncing action. Which reminds me: I said I'd talk a little bit about how I'm doing things differently now.

I did start off with the same system that I mention here-just symlinking files and folders over to certain locations in my Dropbox folder. In fact, I still do that with certain files and folders, but have begun a hybrid strategy. For my ~/bin and ~/dev folders, I just store my git repository for each on Dropbox. This way, the new machine setup scrip can just check out bin and dev into the new home folder. Any changes in ~/bin and ~/dev can get checked in when ready. Those changes will then be staged on Dropbox, ready for checkout by any other of my Dropbox clients.

If that sounds a little convoluted, well, it works really well for certain workflows, and not so much for others. That's why I have a hybrid approach. However, even files and folders that are symlinked directly into Dropbox are kept under version control (git).

Media of the month: "User Interface Design for Programmers" by Joel Spolsky (http://apress.com/book/view/1893115941). This is a bit of an oldie, but still a goodie. This book is a great read to get you outside of the tech mindset to help you design applications so they have an interface that humans can use.

See you next month with more back-to-the-shell topics.


Ed Marczak is the Executive Editor of MacTech Magazine. He has written the Mac in the Shell column since 2004.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Apple Security Update 2015-005 - For OS...
Apple Security Update 2015-005 is recommended for all users and improves the security of OS X. For detailed information about the security content of this update, please visit: http://support.apple.... Read more
Apple HP Printer Drivers 3.1 - For OS X...
Apple HP Printer Drivers includes the latest HP printing and scanning software for OS X Lion or later. For information about supported printer models, see this page. Version 3.1: The latest printing... Read more
Epson Printer Drivers 3.1 - For OS X 10....
Epson Printer Drivers installs the latest software for your EPSON printer or scanner for OS X Yosemite, OS X Mavericks, OS X Mountain Lion, and OS X Lion. For more information about printing and... Read more
Xcode 6.4 - Integrated development envir...
Xcode provides everything developers need to create great applications for Mac, iPhone, and iPad. Xcode brings user interface design, coding, testing, and debugging into a united workflow. The Xcode... Read more
OS X Yosemite 10.10.4 - Apple's lat...
OS X Yosemite is Apple's newest operating system for Mac. An elegant design that feels entirely fresh, yet inherently familiar. The apps you use every day, enhanced with new features. And a... Read more
Dash 3.0.2 - Instant search and offline...
Dash is an API Documentation Browser and Code Snippet Manager. Dash helps you store snippets of code, as well as instantly search and browse documentation for almost any API you might use (for a full... Read more
FontExplorer X Pro 5.0 - Font management...
FontExplorer X Pro is optimized for professional use; it's the solution that gives you the power you need to manage all your fonts. Now you can more easily manage, activate and organize your... Read more
Typinator 6.6 - Speedy and reliable text...
Typinator turbo-charges your typing productivity. Type a little. Typinator does the rest. We've all faced projects that require repetitive typing tasks. With Typinator, you can store commonly used... Read more
Arq 4.12.1 - Online backup to Google Dri...
Arq is super-easy online backup for the Mac. Back up to your own Google Drive storage (15GB free storage), your own Amazon Glacier ($.01/GB per month storage) or S3, or any SFTP server. Arq backs up... Read more
Gutenprint 5.2.11-pre1 - Quality drivers...
Gutenprint is a suite of printer drivers that may be used with most common UNIX print spooling systems, including CUPS, lpr, LPRng, or others. Gutenprint currently supports over 2000 printer models.... Read more

Vector 2 is Officially a Thing and it...
Vector is a pretty cool parkour-driven runner that's gotten a pretty decent following since it first came out - although personally I think more people could stand to show it some love. Anyway, Nekki has announced that a sequel isofficially on its... | Read more »
This Week at 148Apps:June 22-26, 2015
June's Summer Journey Continues With 148Apps How do you know what apps are worth your time and money? Just look to the review team at 148Apps. We sort through the chaos and find the apps you're looking for. The ones we love become Editor’s Choice,... | Read more »
LEGO® Minifigures Online (Games)
LEGO® Minifigures Online 1.0.1 Device: iOS iPhone Category: Games Price: $4.99, Version: 1.0.1 (iTunes) Description: | Read more »
World of Tanks Blitz celebrates its firs...
Today marks the first anniversary of the launch of World of Tanks Blitz, the mobile version of the PC tank battler, World of Tanks. World of Tanks Blitz launched on iOS and Android on June 26th last year and to celebrate, Wargaming is giving all... | Read more »
Heroes and Castles 2 Has its Own Standal...
Heroes and Castles 2 is a third-person castle defense game from the same team behind Block Fortress and Bug Heroes. It's cool, it's fun, and now it has its very own free version. [Read more] | Read more »
Formula Cartoon All-Stars Lets You Race...
Ever want to pit your favorite characters from shows like Steven Universe, Adventure Time, and Regular Show against each other in a not quite death race? Well once upon a time you could, but Formula All Stars Touch N' Go doesn't exist anymore. Hope... | Read more »
Retype - Typography Photo Editor (Photo...
Retype - Typography Photo Editor 1.0 Device: iOS Universal Category: Photography Price: $2.99, Version: 1.0 (iTunes) Description: Retype is built out of passion for great typography and it's all about adding text to photo with style... | Read more »
Hungry Shark Evolution Celebrates Shark...
Shark Week is almost here, as is Independence Day, so naturally Hungry Shark Evolution is going to get in on the action. Yes, even the fireworks. [Read more] | Read more »
The New Trivia Crack Will Feature a Musi...
It's official: iHeartMedia (you may know them from iHeartRadio) will be in charge of providing music-related questions for Trivia Crack's upcoming sequel. Also Trivia Crack is getting a sequel. [Read more] | Read more »
Toca Life: City (Education)
Toca Life: City 1.0 Device: iOS Universal Category: Education Price: $2.99, Version: 1.0 (iTunes) Description: Welcome to Toca Life: City, a metropolis filled with everyday fun! Customize characters, explore exciting locations and... | Read more »

Price Scanner via MacPrices.net

OtterBox Releases New Symmetry Series Metalli...
Otterbox’s new Symmetry Series of smartphone cases flaunts the best of both both street style and street smarts with their new metallic finishes and trusted OtterBox protection for iPhone 6 and... Read more
MacBook Airs on sale for up to $75 off MSRP
Save up to $75 on the purchase of a new 2015 13″ or 11″ 1.6GHz MacBook Air at the following resellers. Shipping is free with each model: 11" 128GB MSRP $899 11" 256GB... Read more
Apple’s Education discount saves up to $300 o...
Purchase a new Mac or iPad at The Apple Store for Education and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free,... Read more
Save up to $600 with Apple refurbished Mac Pr...
The Apple Store has Apple Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The... Read more
Mac Pros on sale for up to $260 off MSRP
B&H Photo has Mac Pros on sale for up to $260 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3719.99... Read more
Save up to $400 on 2014 15-inch Retina MacBoo...
B&H Photo has previous-generation 2014 15″ Retina MacBook Pros on sale for up to $400 off original MSRP. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.2GHz Retina MacBook Pro... Read more
15-inch Retina MacBook Pros on sale for up to...
B&H Photo has new 2015 15″ Retina MacBook Pros on sale for up to $125 off MSRP including free shipping plus NY sales tax only: - 15″ 2.2GHz Retina MacBook Pro: $1899.99 $100 off - 15″ 2.5GHz... Read more
College Student Deals: Additional $100 off Ma...
Take an additional $100 off all MacBooks and iMacs at Best Buy Online with their College Students Deals Savings, valid through July 11, 2015. Anyone with a valid .EDU email address can take advantage... Read more
Apple refurbished Time Capsules available for...
The Apple Store has certified refurbished Time Capsules available for $100 off MSRP. Apple’s one-year warranty is included with each Time Capsule, and shipping is free: - 2TB Time Capsule: $199, $100... Read more
Newsweek Launches iPhone App
The venerable weekly news magazine Newsweek, owned by IBT Media, has announced the launch of its first iPhone app. The new app is available through Apple’s App Store and will allow consumers to read... Read more

Jobs Board

*Apple* TV Live Streaming Frameworks Test En...
**Job Summary** Work and contribute towards the engineering of Apple 's state-of-the-art products involving video, audio, and graphics in Interactive Media Group (IMG) at Read more
Project Manager, WW *Apple* Fulfillment Ope...
…a senior project manager / business analyst to work within our Worldwide Apple Fulfillment Operations and the Business Process Re-engineering team. This role will work Read more
Senior Data Scientist, *Apple* Retail - Onl...
**Job Summary** Apple Retail - Online sells Apple products to customers around the world. In addition to selling Apple products with unique services such as iPad Read more
*Apple* Music Producer - Apple (United State...
**Job Summary** Apple Music seeks a Producer to help shepherd some of the most important content and editorial initiatives within the music app, with a particular focus Read more
Sr. Technical Services Consultant, *Apple*...
**Job Summary** Apple Professional Services (APS) has an opening for a senior technical position that contributes to Apple 's efforts for strategic and transactional Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.