TweetFollow Us on Twitter

Mac in the Shell: Dropbox in the Shell

Volume Number: 26
Issue Number: 03
Column Tag: Mac in the Shell

Mac in the Shell: Dropbox in the Shell

Or, Integrating Dropbox and bash!

by Edward Marczak

Welcome

Last month, I promised to get back to some shell topics. This month, I'll do just that. I've become a pretty big fan of Dropbox (http://getdropbox.com). However, it's rare that I access it directly in the GUI. This makes simplifying my life with Dropbox in a shell a pretty high priority. This article will look at how I've integrated Dropbox and my bash shell environment.

Why Dropbox?

If you're unfamiliar with Dropbox, it's a cloud-based storage solution that two-way syncs between the Dropbox servers and a folder on any machines that you've installed the Dropbox client on. Of course, like Apple's iDisk, you provide credentials, so it only syncs your files. Dropbox is cross-platform and has clients for Mac, Linux and Windows. Dropbox allows you to define where your local Dropbox folder is located. Unlike iDisk, which tries to act like it's magic and shield you from its workings, Dropbox acts much more naturally. You are asked to define a folder that is your Dropbox. That's it. From then on, anything that you put in that folder is kept in sync.

I don't want to spend too much time on why I love Dropbox. You're either using Dropbox already and know how good it is, you're thinking of trying it (go try it now at the link mentioned in the intro) or have decided that you don't care for it. If you're in the latter camp, perhaps this article can give you some ideas for similar challenges you may face.

I work on different computers all the time: I have several at work and a few at home. This was quite a challenge for me initially since I had long used the traditional one-laptop-for-everything setup. While I don't need identical environments on every machine I use, there are certain aspects I like to configure in a similar manner. There are also certain tools that I'd like to have access to and certain files that I want to tote about. I have a MobileMe account, so, why don't I just use iDisk for this?

Initially, I tried to use iDisk, but the first barrier was that iDisk is just unbearably slow. Even with a pretty fast connection to the Internet, sync times were just unacceptable. The sync routine for iDisk is also a bit messy. It would often complain of file conflicts for no reason that I could figure out. This is disruptive, as the conflict dialog just pops up over whatever you're doing. Constantly. Perhaps the biggest deal-breaker is that iDisk does not support symlinks. Well, if you keep a local copy of your iDisk, symlinks work fine locally, but they will not be synced up to MobileMe, and therefore won't come down to any other computers syncing to the same iDisk.

There were even more issues, but that was enough for me to start looking for alternatives. A number of friends were using Dropbox, so I gave it a try. Well, everything that I just mentioned is wrong with iDisk is just fine in Dropbox. First, it's fast. Second, it feels better integrated into Mac OS X than just about any third-party product that I've used, and possibly even more so than iDisk. It's absolutely invisible. Finally, my favorite: symlinks work just fine (even though I've modified my workflow and not using them much anymore).

Shell Integration

When I said earlier that Dropbox is very 'natural' and behaves as one would expect, that includes how it responds in a shell. While iDisk mounts a local diskimage, you can't define where that is located. (Well, OK...the disk image is stored in ~/Library/FileSync/(FS_identifier)/(MobileMe_username)_iDisk.sparsebundle and I suppose you could symlink it somewhere. But that doesn't stop the mount from appearing in /Volumes). The Dropbox folder, on the other hand, acts just like any folder (directory) on the system. You can view it in the GUI, or you can list it in a shell. You can 'cd' to it in a shell and manipulate it. Like you'd expect.

That all seems pretty good, right? How much more would you expect to do? Here are some ideas.

Where's Dropbox?

Since Dropbox, as a solution is intended for multiple computers, each computer needs its own client install. Nothing says that the location of the Dropbox folder must be the same on each. By default, the Dropbox installer offers to put the defined folder in your home directory. However, I often move that into /Users/Shared or other locations. I've certainly developed variances among my installs. This is pretty easy to solve using a common shell convention: environment variables.

A review on environment variables: like most shells, bash supports session-persistent in-memory variables. Variables can store string or numeric values. There's even an array type, but it's seldom used and fairly tricky ugly to handle. (Seriously, if you're trying to use arrays in bash you should really consider moving to a scripting language that had arrays in mind from the beginning. Ruby and Python are two excellent examples).

Two important characteristics about environment variables: 1) They're available in all shell contexts. That is, they're defined in each shell you create, and any that fork from that shell. 2) When the $ character is encountered, the shell performs variable expansion. This lets us use variables that we've defined to build up other strings and commands.

The first think that I want to know is this: where is the Dropbox folder on the machine I'm using? Conveniently, Dropbox stores all installation information in an sqlite3 database in a user's home directory. One of the rows of the database is the path to the current Dropbox folder. Excellent. Mac OS X loves sqlite3 and has the command-line binary built-in. Extracting the value of the path is easy:

sqlite3 ~/.dropbox/dropbox.db 'select value from config 
where key="dropbox_path"'
Vi9Wb2x1bWVzL2bbhWVzL1VzZXJzL1YoMTMlZC9nZXJtERMyb3Bib3gKcDEKLg==

Oh, what's this? Unsurprisingly, this value is not stored as plain text. Nor, though, is it encrypted. It's base64 encoded and 'pickled' for serialization. Looks like the Dropbox developers are fans of Python. Python supports a serialization library called 'pickle'. This causes us to run into a small, but surmountable issue: there are no tools that allow us to directly unpickle an object completely within bash. I wrote a small utility in Python that gets me all of this information, but I promised to focus on the shell only, and this is a good opportunity to explore more tools.

First, we'll use the sqlite3 command to get the information from the database. As shown above, the command needs to know which database to read and the SQL command to run against the database. The HOME variable is defined for us in a shell environment, and points to the home directory of the current user. So, the command I'll recommend (and shown in the complete example) uses the ${HOME} variable over the '~' character.

Next, we need to base64 decode the value we get out of the database. OS X doesn't have a single, explicit command that performs a base64 decode. Again, you could use a Python or Perl one-liner, but I prefer to use the multi-faceted openssl binary, which contains a base64 decode routine.

One the value is base64 decoded, you'll notice some extra characters at the beginning of the output, and one on a separate line. This is a result of the pickle serialization. Let's get rid of the separate, second line right away. To do this, I use head. The head utility is a counterpart to tail. The former will display the first n number of lines from the beginning of a file, while the latter will display the last n lines from the end of a file. We only want the first line, so, we pass that information in as an argument. Finally, there's that one extra character up front. We can trim that out with the cut command. We simply tell cut that we only want from character (the -c switch) 2 onward.

If you remember pipes, they come in handy here. A pipe connects the output of one binary to the input of another. Often, programs operate on files on-disk. For example, the head utility can be used like this:

head -3 filename.txt

That will display the first three lines of the file filename.txt. Well, Unix treats just about everything as a file-even input and output. In the case of using a pipe, you're taking the standard output (represented as a file as /dev/stdout) of one program and making it the input (/dev/stdin) of another application (one that reads standard in. Not all do).

Stringing it all together, the final command looks like this:

sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where 
key="dropbox_path"' | openssl base64 -d | head -1 | cut -c 2-

This displays where the current user's Dropbox folder is. More than display it, we want to store it for future use. Here's what I'd put in my .bash_profile script to accomplish this:

# Setup Dropbox Location - used in scripts
DB=$(sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where key="dropbox_path"'
| openssl base64 -d | head -1 | cut -c 2-)

This takes advantage of bash's command substitution. The $(COMMAND) construct allows bash to substitute COMMAND with its output. A simple example:

$ START_TIME=$(date)
$ echo $START_TIME
Fri Mar 12 08:31:31 EST 2010

In the Dropbox example, we're assigning the output of our string of commands-which ultimately outputs the Dropbox path-to the variable DB. From that point forward, we can reference this path as ${DB}

What's This All Good For?

Now we have the local Dropbox folder path. Whoopee. Where can you do anything with that? In other scripts, of course. I actually have several directories in my home directory that are actually stored in my Dropbox folder. This is possible thanks to the magic of symlinks.

A symlink is just a pointer to a file on disk. The ln command is used to create links. Long time Mac users can equate this to a Finder alias. But they're a little different. In essence, a symlink is just another name for a file. Mac OS X provides two kinds of links: hard and symbolic. A hard link effectively is the linked file. An example will make this clear:

$ echo "This is a test" > linktext.txt
$ ln linktext.txt hardlink.txt
$ rm linktext.txt 
$ cat hardlink.txt 
This is a test

What just happened here?

First, a file is created by echo-ing the text "This is a test" into linktext.txt. Then, the ln command is used to create a hard link, named hardlink.txt. Now, the name "hardlink.txt" uses the same inode as linktext.txt, making them the same file. To prove this, in the next step removes the original file. But there's still a pointer to that data on the filesystem: our hardlink, hardlink.txt. To prove this, we can use cat to dump the contents of hardlink.txt. Look at that, it's the same as our original file that we "deleted." Only when we remove hardlink.txt will this directory entry be freed up.

A symbolic link, used much more frequently, is a separate entry on disk that points to the original file. Unlike a hard link, deleting the original file will free the directory entry. This is much more akin to a Finder alias. However, a Finder alias is only understood by the Finder, and not by shell tools. A symlink is much more flexible, being understood by both the Finder and the shell. Let's look at an example:

$ echo "This is a test" > linktext.txt
$ ln -s linktext.txt symlink.txt
$ ls -l
total 16
-rw-r-r- 1 erm admin 15 Mar 12 18:36 linktext.txt
lrwxr-xr-x 1 erm admin 12 Mar 12 18:36 symlink.txt -> linktext.txt
$ cat symlink.txt
This is a test
$ rm linktext.txt 
$ cat symlink.txt 
cat: symlink.txt: No such file or directory

You'll see that unlike hard links, a symbolic link is separate from the original file. In this example, we create a file using echo to redirect text, then create a symbolic link to it. Note that we use the -s switch with ln. By default, ln creates a hard link. The -s switch instructs ln to create a symbolic link. If we try the same delete-and-use-it test that we did with the hard link, we're going to be disappointed. As shown, after removing the original file, the symlink points to nothing.

While it may seem the hard links are better in some respects, symlinks are actually a little more flexible. Since inodes are only unique to a given file system, hard links cannot span across file systems. Another limitation of hard links is that they can only point to files, not directories. A common use of symlinks is to use a common name to point to the latest distribution directory of an application or web directory. For example:

web-site-1.0/
web-site-2.0/
htdocs -> web-site-1.0

In this case, web-site-1.0 is the current site. Once web-site-2.0 is complete and ready to become the main site, all that needs to be done is to point the symlink to the new directory.

Other Ways to Integrate

Now that I have these basics out of the way and I have easy access to my Dropbox folder and a variable with the Dropbox folder path, what do I use it for? First, I have a script that configures a new machine to my liking.

In my Dropbox folder, I store my ~/bin directory. The idea is that I can symlink to it from my home directory. (I've actually changed how I do this-read until the end of this article to read how I now handle this). Inside of ~/bin, I keep a setup script that configures a new machine for my use. One thing that this script relies on: the $DB variable-it needs to know where I've configured the Dropbox folder to be. On (most) any new machine, my workflow is something like this:

1. Create login account

2. Install Dropbox

3. Run ${DB}/bin/setup_new.sh - the setup script

What does this setup script do? I'll just generalize here as the script is incredibly specific to the way I work. However, the important thing is that it can bootstrap my environment just from the Dropbox folder.

First, it symlinks ~/bin folder back to itself. Since we already know the path, this is pretty easy:

ln -s ${DB}/bin ${HOME}/bin

A nicer way to handle this involves some logic and error checking:

#!/bin/bash
# Setup links to bin folder
ERROR=0
if [ -d ${HOME}/bin ]; then
  mv ${HOME}/bin ${HOME}/bin.old
  if [ $? != 0 ]; then
    echo "ERROR: Could not move current ~/bin directory"
    ERROR=1
  fi
fi
if [ -f ${HOME}/bin ]; then
  echo "ERROR: ~/bin already exists - not touching"
  ERROR=1
fi
if [[ $ERROR == 0 ]]; then
  echo "Linking ${HOME}/bin -> ${DB}/bin"
  ln -s ${DB}/bin ${HOME}/bin
fi

Conditional statements will be looked at in more detail in a future column.

The second thing my setup_new.sh script does is to call another script that runs several defaults commands that tweak things just the way I like them. (Yes, I like seeing all files in the Finder, and no, I do not want a warning when I empty the trash). I keep it in s separate script just because I like the modularity and possibility of easily running it without needing to run the entire setup_main.sh script.

Summary and Epilogue

This article demonstrates one way to integrate your shell experience with a perhaps not-so-obvious GUI element. In this case, once your shell knows where the Dropbox folder is, it can perform any manner of syncing action. Which reminds me: I said I'd talk a little bit about how I'm doing things differently now.

I did start off with the same system that I mention here-just symlinking files and folders over to certain locations in my Dropbox folder. In fact, I still do that with certain files and folders, but have begun a hybrid strategy. For my ~/bin and ~/dev folders, I just store my git repository for each on Dropbox. This way, the new machine setup scrip can just check out bin and dev into the new home folder. Any changes in ~/bin and ~/dev can get checked in when ready. Those changes will then be staged on Dropbox, ready for checkout by any other of my Dropbox clients.

If that sounds a little convoluted, well, it works really well for certain workflows, and not so much for others. That's why I have a hybrid approach. However, even files and folders that are symlinked directly into Dropbox are kept under version control (git).

Media of the month: "User Interface Design for Programmers" by Joel Spolsky (http://apress.com/book/view/1893115941). This is a bit of an oldie, but still a goodie. This book is a great read to get you outside of the tech mindset to help you design applications so they have an interface that humans can use.

See you next month with more back-to-the-shell topics.


Ed Marczak is the Executive Editor of MacTech Magazine. He has written the Mac in the Shell column since 2004.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Apple GarageBand 10.1.4 - Complete recor...
The new GarageBand is a whole music creation studio right inside your Mac -- complete with keyboard, synths, orchestral and percussion instruments, presets for guitar and voice, an entirely... Read more
Alfred 3.2.1 - Quick launcher for apps a...
Alfred is an award-winning productivity application for OS X. Alfred saves you time when you search for files online or on your Mac. Be more productive with hotkeys, keywords, and file actions at... Read more
Adobe Lightroom 6.8 - Import, develop, a...
Adobe Lightroom is available as part of Adobe Creative Cloud for as little as $9.99/month bundled with Photoshop CC as part of the photography package. Lightroom 6 is also available for purchase as a... Read more
Backblaze 4.2.0.990 - Online backup serv...
Backblaze is an online backup service designed from the ground-up for the Mac. With unlimited storage available for $5 per month, as well as a free 15-day trial, peace of mind is within reach with... Read more
OmniPlan Pro 3.6 - Professional-grade pr...
With OmniPlan Pro, you can create logical, manageable project plans with Gantt charts, schedules, summaries, milestones, and critical paths. Break down the tasks needed to make your project a success... Read more
OmniPlan 3.6 - Robust project management...
With OmniPlan, you can create logical, manageable project plans with Gantt charts, schedules, summaries, milestones, and critical paths. Break down the tasks needed to make your project a success,... Read more
Little Snitch 3.7.1 - Alerts you about o...
Little Snitch gives you control over your private outgoing data. Track background activity As soon as your computer connects to the Internet, applications often have permission to send any... Read more
iMazing 2.1.3 - Complete iOS device mana...
iMazing (was DiskAid) is the ultimate iOS device manager with capabilities far beyond what iTunes offers. With iMazing and your iOS device (iPhone, iPad, or iPod), you can: Copy music to and from... Read more
AppDelete 4.3.1 - $7.99
AppDelete is an uninstaller that will remove not only applications but also widgets, preference panes, plugins, and screensavers along with their associated files. Without AppDelete these associated... Read more
FileZilla 3.23.0.2 - Fast and reliable F...
FileZilla (ported from Windows) is a fast and reliable FTP client and server with lots of useful features and an intuitive interface. Version 3.23.0.2: Bug Fixes and Minor Changes Speed up icon... Read more

Latest Forum Discussions

See All

Galaxy on Fire 3 and four other fantasti...
Galaxy on Fire 3 - Manticore brings the series back for another round of daring space battles. It's familiar territory for folks who are familiar with the franchise. If you've beaten the game and are looking to broaden your horizons, might we... | Read more »
The best apps for your holiday gift exch...
What's that, you say? You still haven't started your holiday shopping? Don't beat yourself up over it -- a lot of people have been putting it off, too. It's become easier and easier to procrastinate gift shopping thanks to a number of apps that... | Read more »
Toca Hair Salon 3 (Education)
Toca Hair Salon 3 1.0 Device: iOS Universal Category: Education Price: $2.99, Version: 1.0 (iTunes) Description: | Read more »
Winter comes to Darkwood as Seekers Note...
MyTona, based in the chilly Siberian city of Yakutsk, has brought a little festive fun to its hidden object game Seekers Notes: Hidden Mystery. The Christmas update introduces some new inhabitants to players, and with them a chance to win plenty of... | Read more »
Bully: Anniversary Edition (Games)
Bully: Anniversary Edition 1.03.1 Device: iOS Universal Category: Games Price: $6.99, Version: 1.03.1 (iTunes) Description: *** PLEASE NOTE: This game is officially supported on the following devices: iPhone 5 and newer, iPod Touch... | Read more »
PINE GROVE (Games)
PINE GROVE 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: A pine grove where there are no footsteps of people due to continuous missing cases. The case is still unsolved and nothing has... | Read more »
Niantic teases new Pokémon announcement...
After rumors started swirling yesterday, it turns out there is an official Pokémon GO update on its way. We’ll find out what’s in store for us and our growing Pokémon collections tomorrow during the Starbucks event, but Niantic will be revealing... | Read more »
3 reasons why Nicki Minaj: The Empire is...
Nicki Minaj is as business-savvy as she is musically talented and she’s proved that by launching her own game. Designed by Glu, purveyors of other fine celebrity games like cult favorite Kim Kardashian: Hollywood, Nicki Minaj: The Empire launched... | Read more »
Clash of Clans is getting its own animat...
Riding on its unending wave of fame and success, Clash of Clans is getting an animated web series based on its Clash-A-Rama animated shorts.As opposed to the current shorts' 60 second run time, the new and improved Clash-A-Rama will be comprised of... | Read more »
Leaks hint at Pokémon GO and Starbucks C...
Leaked images from a hub for Starbucks employees suggests that a big Pokémon GO event with the coffee giant could begin this very week. The images appeared on Reddit and hint at some exciting new things to come for Niantic's smash hit game. | Read more »

Price Scanner via MacPrices.net

Holiday sale: 12-inch Retina MacBook for $100...
B&H has 12″ Retina MacBooks on sale for $100 off MSRP as part of their Holiday sale. Shipping is free, and B&H charges NY sales tax only: - 12″ 1.1GHz Space Gray Retina MacBook: $1199 $100... Read more
Apple refurbished 13-inch MacBook Airs availa...
Apple has Certified Refurbished 13″ MacBook Airs available starting at $849. An Apple one-year warranty is included with each MacBook, and shipping is free: - 13″ 1.6GHz/8GB/128GB MacBook Air: $849 $... Read more
Apple refurbished iMacs available for up to $...
Apple has Certified Refurbished 2015 21″ & 27″ iMacs available for up to $350 off MSRP. Apple’s one-year warranty is standard, and shipping is free. The following models are available: - 21″ 3.... Read more
Apple’s Education discount saves up to $300 o...
Purchase a new Mac or iPad using Apple’s Education Store and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free: -... Read more
Back in stock: Apple refurbished Mac minis fr...
Apple has Certified Refurbished Mac minis available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: - 1.4GHz Mac mini: $419 $80 off MSRP - 2.6GHz Mac... Read more
Twenty-Five Years Of Apple Laptops – A person...
Among many other things, the often tumultuous 16th year of the new century marked the 25th anniversary of Apple laptop computers, not counting the optimistically named 16-pound Mac Portable of 1989.... Read more
Landlordy iOS App Adds Support For Appliances...
Riga, Latvia based E-protect SIA is releasing major update (version 1.8) to its Landlordy app for managing rental business financials on the go. Landlordy is iPhone and iPad app designed for self-... Read more
Holiday sale, Apple iMacs for up to $200 off...
B&H Photo has 21″ and 27″ Apple iMacs on sale for up to $200 off MSRP including free shipping plus NY sales tax only: - 27″ 3.3GHz iMac 5K: $2099 $200 off MSRP - 27″ 3.2GHz/1TB Fusion iMac 5K: $... Read more
Holiday sale: Mac minis for $50 to $100 off M...
B&H Photo has Mac minis on sale for up to $100 off MSRP free shipping plus NY sales tax only: - 1.4GHz Mac mini: $449 $50 off MSRP - 2.6GHz Mac mini: $629 $70 off MSRP - 2.8GHz Mac mini: $899 $... Read more
Mac Pros on sale for up to $300 off MSRP, ref...
B&H Photo has Mac Pros on sale for up to $300 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3699, $... Read more

Jobs Board

*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Retail - Multiple Positions- Trumbul...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions- Philade...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Retail - Multiple Positions- San Ant...
Job Description:SalesSpecialist - Retail Customer Service and SalesTransform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Products Tester Needed - Apple (Unit...
…we therefore look forward to put out products to quality test for durability. Apple leads the digital music revolution with its iPods and iTunes online store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.