TweetFollow Us on Twitter

Mac in the Shell: Dropbox in the Shell

Volume Number: 26
Issue Number: 03
Column Tag: Mac in the Shell

Mac in the Shell: Dropbox in the Shell

Or, Integrating Dropbox and bash!

by Edward Marczak

Welcome

Last month, I promised to get back to some shell topics. This month, I'll do just that. I've become a pretty big fan of Dropbox (http://getdropbox.com). However, it's rare that I access it directly in the GUI. This makes simplifying my life with Dropbox in a shell a pretty high priority. This article will look at how I've integrated Dropbox and my bash shell environment.

Why Dropbox?

If you're unfamiliar with Dropbox, it's a cloud-based storage solution that two-way syncs between the Dropbox servers and a folder on any machines that you've installed the Dropbox client on. Of course, like Apple's iDisk, you provide credentials, so it only syncs your files. Dropbox is cross-platform and has clients for Mac, Linux and Windows. Dropbox allows you to define where your local Dropbox folder is located. Unlike iDisk, which tries to act like it's magic and shield you from its workings, Dropbox acts much more naturally. You are asked to define a folder that is your Dropbox. That's it. From then on, anything that you put in that folder is kept in sync.

I don't want to spend too much time on why I love Dropbox. You're either using Dropbox already and know how good it is, you're thinking of trying it (go try it now at the link mentioned in the intro) or have decided that you don't care for it. If you're in the latter camp, perhaps this article can give you some ideas for similar challenges you may face.

I work on different computers all the time: I have several at work and a few at home. This was quite a challenge for me initially since I had long used the traditional one-laptop-for-everything setup. While I don't need identical environments on every machine I use, there are certain aspects I like to configure in a similar manner. There are also certain tools that I'd like to have access to and certain files that I want to tote about. I have a MobileMe account, so, why don't I just use iDisk for this?

Initially, I tried to use iDisk, but the first barrier was that iDisk is just unbearably slow. Even with a pretty fast connection to the Internet, sync times were just unacceptable. The sync routine for iDisk is also a bit messy. It would often complain of file conflicts for no reason that I could figure out. This is disruptive, as the conflict dialog just pops up over whatever you're doing. Constantly. Perhaps the biggest deal-breaker is that iDisk does not support symlinks. Well, if you keep a local copy of your iDisk, symlinks work fine locally, but they will not be synced up to MobileMe, and therefore won't come down to any other computers syncing to the same iDisk.

There were even more issues, but that was enough for me to start looking for alternatives. A number of friends were using Dropbox, so I gave it a try. Well, everything that I just mentioned is wrong with iDisk is just fine in Dropbox. First, it's fast. Second, it feels better integrated into Mac OS X than just about any third-party product that I've used, and possibly even more so than iDisk. It's absolutely invisible. Finally, my favorite: symlinks work just fine (even though I've modified my workflow and not using them much anymore).

Shell Integration

When I said earlier that Dropbox is very 'natural' and behaves as one would expect, that includes how it responds in a shell. While iDisk mounts a local diskimage, you can't define where that is located. (Well, OK...the disk image is stored in ~/Library/FileSync/(FS_identifier)/(MobileMe_username)_iDisk.sparsebundle and I suppose you could symlink it somewhere. But that doesn't stop the mount from appearing in /Volumes). The Dropbox folder, on the other hand, acts just like any folder (directory) on the system. You can view it in the GUI, or you can list it in a shell. You can 'cd' to it in a shell and manipulate it. Like you'd expect.

That all seems pretty good, right? How much more would you expect to do? Here are some ideas.

Where's Dropbox?

Since Dropbox, as a solution is intended for multiple computers, each computer needs its own client install. Nothing says that the location of the Dropbox folder must be the same on each. By default, the Dropbox installer offers to put the defined folder in your home directory. However, I often move that into /Users/Shared or other locations. I've certainly developed variances among my installs. This is pretty easy to solve using a common shell convention: environment variables.

A review on environment variables: like most shells, bash supports session-persistent in-memory variables. Variables can store string or numeric values. There's even an array type, but it's seldom used and fairly tricky ugly to handle. (Seriously, if you're trying to use arrays in bash you should really consider moving to a scripting language that had arrays in mind from the beginning. Ruby and Python are two excellent examples).

Two important characteristics about environment variables: 1) They're available in all shell contexts. That is, they're defined in each shell you create, and any that fork from that shell. 2) When the $ character is encountered, the shell performs variable expansion. This lets us use variables that we've defined to build up other strings and commands.

The first think that I want to know is this: where is the Dropbox folder on the machine I'm using? Conveniently, Dropbox stores all installation information in an sqlite3 database in a user's home directory. One of the rows of the database is the path to the current Dropbox folder. Excellent. Mac OS X loves sqlite3 and has the command-line binary built-in. Extracting the value of the path is easy:

sqlite3 ~/.dropbox/dropbox.db 'select value from config 
where key="dropbox_path"'
Vi9Wb2x1bWVzL2bbhWVzL1VzZXJzL1YoMTMlZC9nZXJtERMyb3Bib3gKcDEKLg==

Oh, what's this? Unsurprisingly, this value is not stored as plain text. Nor, though, is it encrypted. It's base64 encoded and 'pickled' for serialization. Looks like the Dropbox developers are fans of Python. Python supports a serialization library called 'pickle'. This causes us to run into a small, but surmountable issue: there are no tools that allow us to directly unpickle an object completely within bash. I wrote a small utility in Python that gets me all of this information, but I promised to focus on the shell only, and this is a good opportunity to explore more tools.

First, we'll use the sqlite3 command to get the information from the database. As shown above, the command needs to know which database to read and the SQL command to run against the database. The HOME variable is defined for us in a shell environment, and points to the home directory of the current user. So, the command I'll recommend (and shown in the complete example) uses the ${HOME} variable over the '~' character.

Next, we need to base64 decode the value we get out of the database. OS X doesn't have a single, explicit command that performs a base64 decode. Again, you could use a Python or Perl one-liner, but I prefer to use the multi-faceted openssl binary, which contains a base64 decode routine.

One the value is base64 decoded, you'll notice some extra characters at the beginning of the output, and one on a separate line. This is a result of the pickle serialization. Let's get rid of the separate, second line right away. To do this, I use head. The head utility is a counterpart to tail. The former will display the first n number of lines from the beginning of a file, while the latter will display the last n lines from the end of a file. We only want the first line, so, we pass that information in as an argument. Finally, there's that one extra character up front. We can trim that out with the cut command. We simply tell cut that we only want from character (the -c switch) 2 onward.

If you remember pipes, they come in handy here. A pipe connects the output of one binary to the input of another. Often, programs operate on files on-disk. For example, the head utility can be used like this:

head -3 filename.txt

That will display the first three lines of the file filename.txt. Well, Unix treats just about everything as a file-even input and output. In the case of using a pipe, you're taking the standard output (represented as a file as /dev/stdout) of one program and making it the input (/dev/stdin) of another application (one that reads standard in. Not all do).

Stringing it all together, the final command looks like this:

sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where 
key="dropbox_path"' | openssl base64 -d | head -1 | cut -c 2-

This displays where the current user's Dropbox folder is. More than display it, we want to store it for future use. Here's what I'd put in my .bash_profile script to accomplish this:

# Setup Dropbox Location - used in scripts
DB=$(sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where key="dropbox_path"'
| openssl base64 -d | head -1 | cut -c 2-)

This takes advantage of bash's command substitution. The $(COMMAND) construct allows bash to substitute COMMAND with its output. A simple example:

$ START_TIME=$(date)
$ echo $START_TIME
Fri Mar 12 08:31:31 EST 2010

In the Dropbox example, we're assigning the output of our string of commands-which ultimately outputs the Dropbox path-to the variable DB. From that point forward, we can reference this path as ${DB}

What's This All Good For?

Now we have the local Dropbox folder path. Whoopee. Where can you do anything with that? In other scripts, of course. I actually have several directories in my home directory that are actually stored in my Dropbox folder. This is possible thanks to the magic of symlinks.

A symlink is just a pointer to a file on disk. The ln command is used to create links. Long time Mac users can equate this to a Finder alias. But they're a little different. In essence, a symlink is just another name for a file. Mac OS X provides two kinds of links: hard and symbolic. A hard link effectively is the linked file. An example will make this clear:

$ echo "This is a test" > linktext.txt
$ ln linktext.txt hardlink.txt
$ rm linktext.txt 
$ cat hardlink.txt 
This is a test

What just happened here?

First, a file is created by echo-ing the text "This is a test" into linktext.txt. Then, the ln command is used to create a hard link, named hardlink.txt. Now, the name "hardlink.txt" uses the same inode as linktext.txt, making them the same file. To prove this, in the next step removes the original file. But there's still a pointer to that data on the filesystem: our hardlink, hardlink.txt. To prove this, we can use cat to dump the contents of hardlink.txt. Look at that, it's the same as our original file that we "deleted." Only when we remove hardlink.txt will this directory entry be freed up.

A symbolic link, used much more frequently, is a separate entry on disk that points to the original file. Unlike a hard link, deleting the original file will free the directory entry. This is much more akin to a Finder alias. However, a Finder alias is only understood by the Finder, and not by shell tools. A symlink is much more flexible, being understood by both the Finder and the shell. Let's look at an example:

$ echo "This is a test" > linktext.txt
$ ln -s linktext.txt symlink.txt
$ ls -l
total 16
-rw-r-r- 1 erm admin 15 Mar 12 18:36 linktext.txt
lrwxr-xr-x 1 erm admin 12 Mar 12 18:36 symlink.txt -> linktext.txt
$ cat symlink.txt
This is a test
$ rm linktext.txt 
$ cat symlink.txt 
cat: symlink.txt: No such file or directory

You'll see that unlike hard links, a symbolic link is separate from the original file. In this example, we create a file using echo to redirect text, then create a symbolic link to it. Note that we use the -s switch with ln. By default, ln creates a hard link. The -s switch instructs ln to create a symbolic link. If we try the same delete-and-use-it test that we did with the hard link, we're going to be disappointed. As shown, after removing the original file, the symlink points to nothing.

While it may seem the hard links are better in some respects, symlinks are actually a little more flexible. Since inodes are only unique to a given file system, hard links cannot span across file systems. Another limitation of hard links is that they can only point to files, not directories. A common use of symlinks is to use a common name to point to the latest distribution directory of an application or web directory. For example:

web-site-1.0/
web-site-2.0/
htdocs -> web-site-1.0

In this case, web-site-1.0 is the current site. Once web-site-2.0 is complete and ready to become the main site, all that needs to be done is to point the symlink to the new directory.

Other Ways to Integrate

Now that I have these basics out of the way and I have easy access to my Dropbox folder and a variable with the Dropbox folder path, what do I use it for? First, I have a script that configures a new machine to my liking.

In my Dropbox folder, I store my ~/bin directory. The idea is that I can symlink to it from my home directory. (I've actually changed how I do this-read until the end of this article to read how I now handle this). Inside of ~/bin, I keep a setup script that configures a new machine for my use. One thing that this script relies on: the $DB variable-it needs to know where I've configured the Dropbox folder to be. On (most) any new machine, my workflow is something like this:

1. Create login account

2. Install Dropbox

3. Run ${DB}/bin/setup_new.sh - the setup script

What does this setup script do? I'll just generalize here as the script is incredibly specific to the way I work. However, the important thing is that it can bootstrap my environment just from the Dropbox folder.

First, it symlinks ~/bin folder back to itself. Since we already know the path, this is pretty easy:

ln -s ${DB}/bin ${HOME}/bin

A nicer way to handle this involves some logic and error checking:

#!/bin/bash
# Setup links to bin folder
ERROR=0
if [ -d ${HOME}/bin ]; then
  mv ${HOME}/bin ${HOME}/bin.old
  if [ $? != 0 ]; then
    echo "ERROR: Could not move current ~/bin directory"
    ERROR=1
  fi
fi
if [ -f ${HOME}/bin ]; then
  echo "ERROR: ~/bin already exists - not touching"
  ERROR=1
fi
if [[ $ERROR == 0 ]]; then
  echo "Linking ${HOME}/bin -> ${DB}/bin"
  ln -s ${DB}/bin ${HOME}/bin
fi

Conditional statements will be looked at in more detail in a future column.

The second thing my setup_new.sh script does is to call another script that runs several defaults commands that tweak things just the way I like them. (Yes, I like seeing all files in the Finder, and no, I do not want a warning when I empty the trash). I keep it in s separate script just because I like the modularity and possibility of easily running it without needing to run the entire setup_main.sh script.

Summary and Epilogue

This article demonstrates one way to integrate your shell experience with a perhaps not-so-obvious GUI element. In this case, once your shell knows where the Dropbox folder is, it can perform any manner of syncing action. Which reminds me: I said I'd talk a little bit about how I'm doing things differently now.

I did start off with the same system that I mention here-just symlinking files and folders over to certain locations in my Dropbox folder. In fact, I still do that with certain files and folders, but have begun a hybrid strategy. For my ~/bin and ~/dev folders, I just store my git repository for each on Dropbox. This way, the new machine setup scrip can just check out bin and dev into the new home folder. Any changes in ~/bin and ~/dev can get checked in when ready. Those changes will then be staged on Dropbox, ready for checkout by any other of my Dropbox clients.

If that sounds a little convoluted, well, it works really well for certain workflows, and not so much for others. That's why I have a hybrid approach. However, even files and folders that are symlinked directly into Dropbox are kept under version control (git).

Media of the month: "User Interface Design for Programmers" by Joel Spolsky (http://apress.com/book/view/1893115941). This is a bit of an oldie, but still a goodie. This book is a great read to get you outside of the tech mindset to help you design applications so they have an interface that humans can use.

See you next month with more back-to-the-shell topics.


Ed Marczak is the Executive Editor of MacTech Magazine. He has written the Mac in the Shell column since 2004.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Freeway Pro 7.0.3 - Drag-and-drop Web de...
Freeway Pro lets you build websites with speed and precision... without writing a line of code! With its user-oriented drag-and-drop interface, Freeway Pro helps you piece together the website of... Read more
Cloud 3.3.0 - File sharing from your men...
Cloud is simple file sharing for the Mac. Drag a file from your Mac to the CloudApp icon in the menubar and we take care of the rest. A link to the file will automatically be copied to your clipboard... Read more
Cyberduck 4.6.5 - FTP and SFTP browser....
Cyberduck is a robust FTP/FTP-TLS/SFTP browser for the Mac whose lack of visual clutter and cleverly intuitive features make it easy to use. Support for external editors and system technologies such... Read more
Firefox 36.0 - Fast, safe Web browser. (...
Firefox for Mac offers a fast, safe Web browsing experience. Browse quickly, securely, and effortlessly. With its industry-leading features, Firefox is the choice of Web development professionals and... Read more
Thunderbird 31.5.0 - Email client from M...
As of July 2012, Thunderbird has transitioned to a new governance model, with new features being developed by the broader free software and open source community, and security fixes and improvements... Read more
VOX 2.4 - Music player that supports man...
VoxIt just sounds better! The beauty is in its simplicity, yet behind the minimal exterior lies a powerful music player with a ton of features & support for all audio formats you should ever need... Read more
A Better Finder Rename 9.46 - File, phot...
A Better Finder Rename is the most complete renaming solution available on the market today. That's why, since 1996, tens of thousands of hobbyists, professionals and businesses depend on A Better... Read more
WALTR 1.0.9 - Drag-and-drop any media fi...
WALTR is designed to make it easy to upload and convert any music or video file to an iPad or iPhone format for native playback. It supports a huge variety of media file types, including MP3, MP4,... Read more
Default Folder X 4.6.14 - Enhances Open...
Default Folder X attaches a toolbar to the right side of the Open and Save dialogs in any OS X-native application. The toolbar gives you fast access to various folders and commands. You just click on... Read more
Boom 2 1.1 - System-wide pro audio app f...
Boom 2 is a system-wide volume booster and equalizer app that is designed especially for OS X 10.10 Yosemite. It comes with a smart interface, self-calibrates itself according to your Mac, offers... Read more

Legacy Quest is an Upcoming Rouge-like T...
Legacy Quest is an Upcoming Rouge-like That’ll Kill the Whole Family Posted by Jessica Fisher on February 26th, 2015 [ permalink ] Nexon Co. | Read more »
Grudgeball: Enter the Chaosphere Review
Grudgeball: Enter the Chaosphere Review By Jordan Minor on February 26th, 2015 Our Rating: :: MUSCLE MENUniversal App - Designed for iPhone and iPad Regular Show gets an above average game.   | Read more »
Action RPG League of Angels – Fire Raide...
Gaia is being invaded by the Devil Prince and the demonic Devil Army at his disposal, and it’s up to you and your avatar to defeat him in League of Angels – Fire Raiders. Raise a mighty army from hundreds of recruitable angel heroes and take the... | Read more »
Burn Rubber on the Ice With a New Cars:...
Burn Rubber on the Ice With a New Cars: Fast as Lightning Update Posted by Jessica Fisher on February 26th, 2015 [ permalink ] Universal App - Designed for iPhone and iPad | Read more »
AdVenture Capitalist Review
AdVenture Capitalist Review By Jordan Minor on February 26th, 2015 Our Rating: :: DAS KAPITALUniversal App - Designed for iPhone and iPad An inadvertent Marxist manifesto.   | Read more »
Monster vs Sheep Review
Monster vs Sheep Review By Jennifer Allen on February 25th, 2015 Our Rating: :: SAMEY FUNUniversal App - Designed for iPhone and iPad What Monster vs Sheep lacks in variety it makes up for with stress relieving fun. At least, for a... | Read more »
Is Your Face Ready for the New Outwitter...
Is Your Face Ready for the New Outwitters 2.0 Trailer? Posted by Jessica Fisher on February 25th, 2015 [ permalink ] One Man Left Studios has announced that their turn-based strategy game, | Read more »
HowToFormat Review
HowToFormat Review By Jennifer Allen on February 25th, 2015 Our Rating: :: USEFUL TIPSiPhone App - Designed for the iPhone, compatible with the iPad Making a presentation and want to get it just right? HowToFormat teaches you how... | Read more »
Thermo Diem Review
Thermo Diem Review By Jennifer Allen on February 25th, 2015 Our Rating: :: GETS TO THE POINTUniversal App - Designed for iPhone and iPad Want to know whether it’s warmer or colder tomorrow? That’s precisely what Thermo Diem will... | Read more »
Tap Heroes Review
Tap Heroes Review By Jennifer Allen on February 25th, 2015 Our Rating: :: TAPPING & GRINDINGUniversal App - Designed for iPhone and iPad Tap like crazy, grind like crazy, in this tapping game that overcomplicates the mix.   | Read more »

Price Scanner via MacPrices.net

New Travel Health App “My Travel Health” iOS...
Rochester, Minnesota based Travel Health and Wellness LLC has announced that its new iOS app help safeguard the user’s health when traveling abroad — “My Travel Health” is now available on the Apple... Read more
Sale! MacBook Airs for up to $115 off MSRP
B&H Photo has MacBook Airs on sale for up to $100 off MSRP. Shipping is free, and B&H charges NY sales tax only: - 11″ 128GB MacBook Air: $799 100 off MSRP - 11″ 256GB MacBook Air: $999 $100... Read more
15-inch 2.0GHz Retina MacBook Pro (refurbishe...
The Apple Store has Apple Certified Refurbished previous-generation 15″ 2.0GHz Retina MacBook Pros available for $1489 including free shipping plus Apple’s standard one-year warranty. Their price is... Read more
Wither The iPad mini? End Of The Road Imminen...
AppleDailyReport’s Dennis Sellers predicts that the iPad mini is going to be left to wither on the vine, as it were, and then just allowed to fade away — a casualty of the IPhone 6 Plus and other... Read more
Android and iOS Duopoly Owns 96.3% of Smartph...
IDC reports that Android and iOS inched closer to total domination of the worldwide smartphone market in both the fourth quarter (4Q14) and the calendar year 2014 (CY14). According to data from the... Read more
13-inch 2.4GHz Retina MacBook Pro available f...
MacMall has the 2013 13″ 2.4GHz/128GB Retina MacBook Pro available for $999.99 for a limited time. Shipping is free. Their price is $300 off original MSRP, and it’s the only sub-$1000 new Retina... Read more
Save up to $300 on a new Mac, $30 on an iPad,...
Purchase a new Mac or iPad at The Apple Store for Education and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free,... Read more
Mac minis available for up to $75 off MSRP
MacMall has Mac minis on sale for up to $75 off MSRP including free shipping. Their prices are the lowest available for these models from any reseller: - 1.4GHz Mac mini: $459.99 $40 off - 2.6GHz Mac... Read more
WaterField Unveils Versatile Padded Gear Pouc...
San Francisco manufacturer WaterField Design’s new Padded Gear Pouch is a light and handy-sized, yet protective, organizer for every kind of take-along gear: technology, travel, toiletries,... Read more
College Student Deals: Additional $50 off Mac...
Take an additional $50 off all MacBooks and iMacs at Best Buy Online with their College Students Deals Savings, valid through April 11, 2015. Anyone with a valid .EDU email address can take advantage... Read more

Jobs Board

*Apple* Solutions Consultant - Retail Sales...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
*Apple* Solutions Consultant (ASC)- Retail S...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
Sr. Technical Services Consultant, *Apple*...
**Job Summary** Apple Professional Services (APS) has an opening for a senior technical position that contributes to Apple 's efforts for strategic and transactional Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.