TweetFollow Us on Twitter

Mac in the Shell: Dropbox in the Shell

Volume Number: 26
Issue Number: 03
Column Tag: Mac in the Shell

Mac in the Shell: Dropbox in the Shell

Or, Integrating Dropbox and bash!

by Edward Marczak

Welcome

Last month, I promised to get back to some shell topics. This month, I'll do just that. I've become a pretty big fan of Dropbox (http://getdropbox.com). However, it's rare that I access it directly in the GUI. This makes simplifying my life with Dropbox in a shell a pretty high priority. This article will look at how I've integrated Dropbox and my bash shell environment.

Why Dropbox?

If you're unfamiliar with Dropbox, it's a cloud-based storage solution that two-way syncs between the Dropbox servers and a folder on any machines that you've installed the Dropbox client on. Of course, like Apple's iDisk, you provide credentials, so it only syncs your files. Dropbox is cross-platform and has clients for Mac, Linux and Windows. Dropbox allows you to define where your local Dropbox folder is located. Unlike iDisk, which tries to act like it's magic and shield you from its workings, Dropbox acts much more naturally. You are asked to define a folder that is your Dropbox. That's it. From then on, anything that you put in that folder is kept in sync.

I don't want to spend too much time on why I love Dropbox. You're either using Dropbox already and know how good it is, you're thinking of trying it (go try it now at the link mentioned in the intro) or have decided that you don't care for it. If you're in the latter camp, perhaps this article can give you some ideas for similar challenges you may face.

I work on different computers all the time: I have several at work and a few at home. This was quite a challenge for me initially since I had long used the traditional one-laptop-for-everything setup. While I don't need identical environments on every machine I use, there are certain aspects I like to configure in a similar manner. There are also certain tools that I'd like to have access to and certain files that I want to tote about. I have a MobileMe account, so, why don't I just use iDisk for this?

Initially, I tried to use iDisk, but the first barrier was that iDisk is just unbearably slow. Even with a pretty fast connection to the Internet, sync times were just unacceptable. The sync routine for iDisk is also a bit messy. It would often complain of file conflicts for no reason that I could figure out. This is disruptive, as the conflict dialog just pops up over whatever you're doing. Constantly. Perhaps the biggest deal-breaker is that iDisk does not support symlinks. Well, if you keep a local copy of your iDisk, symlinks work fine locally, but they will not be synced up to MobileMe, and therefore won't come down to any other computers syncing to the same iDisk.

There were even more issues, but that was enough for me to start looking for alternatives. A number of friends were using Dropbox, so I gave it a try. Well, everything that I just mentioned is wrong with iDisk is just fine in Dropbox. First, it's fast. Second, it feels better integrated into Mac OS X than just about any third-party product that I've used, and possibly even more so than iDisk. It's absolutely invisible. Finally, my favorite: symlinks work just fine (even though I've modified my workflow and not using them much anymore).

Shell Integration

When I said earlier that Dropbox is very 'natural' and behaves as one would expect, that includes how it responds in a shell. While iDisk mounts a local diskimage, you can't define where that is located. (Well, OK...the disk image is stored in ~/Library/FileSync/(FS_identifier)/(MobileMe_username)_iDisk.sparsebundle and I suppose you could symlink it somewhere. But that doesn't stop the mount from appearing in /Volumes). The Dropbox folder, on the other hand, acts just like any folder (directory) on the system. You can view it in the GUI, or you can list it in a shell. You can 'cd' to it in a shell and manipulate it. Like you'd expect.

That all seems pretty good, right? How much more would you expect to do? Here are some ideas.

Where's Dropbox?

Since Dropbox, as a solution is intended for multiple computers, each computer needs its own client install. Nothing says that the location of the Dropbox folder must be the same on each. By default, the Dropbox installer offers to put the defined folder in your home directory. However, I often move that into /Users/Shared or other locations. I've certainly developed variances among my installs. This is pretty easy to solve using a common shell convention: environment variables.

A review on environment variables: like most shells, bash supports session-persistent in-memory variables. Variables can store string or numeric values. There's even an array type, but it's seldom used and fairly tricky ugly to handle. (Seriously, if you're trying to use arrays in bash you should really consider moving to a scripting language that had arrays in mind from the beginning. Ruby and Python are two excellent examples).

Two important characteristics about environment variables: 1) They're available in all shell contexts. That is, they're defined in each shell you create, and any that fork from that shell. 2) When the $ character is encountered, the shell performs variable expansion. This lets us use variables that we've defined to build up other strings and commands.

The first think that I want to know is this: where is the Dropbox folder on the machine I'm using? Conveniently, Dropbox stores all installation information in an sqlite3 database in a user's home directory. One of the rows of the database is the path to the current Dropbox folder. Excellent. Mac OS X loves sqlite3 and has the command-line binary built-in. Extracting the value of the path is easy:

sqlite3 ~/.dropbox/dropbox.db 'select value from config 
where key="dropbox_path"'
Vi9Wb2x1bWVzL2bbhWVzL1VzZXJzL1YoMTMlZC9nZXJtERMyb3Bib3gKcDEKLg==

Oh, what's this? Unsurprisingly, this value is not stored as plain text. Nor, though, is it encrypted. It's base64 encoded and 'pickled' for serialization. Looks like the Dropbox developers are fans of Python. Python supports a serialization library called 'pickle'. This causes us to run into a small, but surmountable issue: there are no tools that allow us to directly unpickle an object completely within bash. I wrote a small utility in Python that gets me all of this information, but I promised to focus on the shell only, and this is a good opportunity to explore more tools.

First, we'll use the sqlite3 command to get the information from the database. As shown above, the command needs to know which database to read and the SQL command to run against the database. The HOME variable is defined for us in a shell environment, and points to the home directory of the current user. So, the command I'll recommend (and shown in the complete example) uses the ${HOME} variable over the '~' character.

Next, we need to base64 decode the value we get out of the database. OS X doesn't have a single, explicit command that performs a base64 decode. Again, you could use a Python or Perl one-liner, but I prefer to use the multi-faceted openssl binary, which contains a base64 decode routine.

One the value is base64 decoded, you'll notice some extra characters at the beginning of the output, and one on a separate line. This is a result of the pickle serialization. Let's get rid of the separate, second line right away. To do this, I use head. The head utility is a counterpart to tail. The former will display the first n number of lines from the beginning of a file, while the latter will display the last n lines from the end of a file. We only want the first line, so, we pass that information in as an argument. Finally, there's that one extra character up front. We can trim that out with the cut command. We simply tell cut that we only want from character (the -c switch) 2 onward.

If you remember pipes, they come in handy here. A pipe connects the output of one binary to the input of another. Often, programs operate on files on-disk. For example, the head utility can be used like this:

head -3 filename.txt

That will display the first three lines of the file filename.txt. Well, Unix treats just about everything as a file-even input and output. In the case of using a pipe, you're taking the standard output (represented as a file as /dev/stdout) of one program and making it the input (/dev/stdin) of another application (one that reads standard in. Not all do).

Stringing it all together, the final command looks like this:

sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where 
key="dropbox_path"' | openssl base64 -d | head -1 | cut -c 2-

This displays where the current user's Dropbox folder is. More than display it, we want to store it for future use. Here's what I'd put in my .bash_profile script to accomplish this:

# Setup Dropbox Location - used in scripts
DB=$(sqlite3 ${HOME}/.dropbox/dropbox.db 'select value from config where key="dropbox_path"'
| openssl base64 -d | head -1 | cut -c 2-)

This takes advantage of bash's command substitution. The $(COMMAND) construct allows bash to substitute COMMAND with its output. A simple example:

$ START_TIME=$(date)
$ echo $START_TIME
Fri Mar 12 08:31:31 EST 2010

In the Dropbox example, we're assigning the output of our string of commands-which ultimately outputs the Dropbox path-to the variable DB. From that point forward, we can reference this path as ${DB}

What's This All Good For?

Now we have the local Dropbox folder path. Whoopee. Where can you do anything with that? In other scripts, of course. I actually have several directories in my home directory that are actually stored in my Dropbox folder. This is possible thanks to the magic of symlinks.

A symlink is just a pointer to a file on disk. The ln command is used to create links. Long time Mac users can equate this to a Finder alias. But they're a little different. In essence, a symlink is just another name for a file. Mac OS X provides two kinds of links: hard and symbolic. A hard link effectively is the linked file. An example will make this clear:

$ echo "This is a test" > linktext.txt
$ ln linktext.txt hardlink.txt
$ rm linktext.txt 
$ cat hardlink.txt 
This is a test

What just happened here?

First, a file is created by echo-ing the text "This is a test" into linktext.txt. Then, the ln command is used to create a hard link, named hardlink.txt. Now, the name "hardlink.txt" uses the same inode as linktext.txt, making them the same file. To prove this, in the next step removes the original file. But there's still a pointer to that data on the filesystem: our hardlink, hardlink.txt. To prove this, we can use cat to dump the contents of hardlink.txt. Look at that, it's the same as our original file that we "deleted." Only when we remove hardlink.txt will this directory entry be freed up.

A symbolic link, used much more frequently, is a separate entry on disk that points to the original file. Unlike a hard link, deleting the original file will free the directory entry. This is much more akin to a Finder alias. However, a Finder alias is only understood by the Finder, and not by shell tools. A symlink is much more flexible, being understood by both the Finder and the shell. Let's look at an example:

$ echo "This is a test" > linktext.txt
$ ln -s linktext.txt symlink.txt
$ ls -l
total 16
-rw-r-r- 1 erm admin 15 Mar 12 18:36 linktext.txt
lrwxr-xr-x 1 erm admin 12 Mar 12 18:36 symlink.txt -> linktext.txt
$ cat symlink.txt
This is a test
$ rm linktext.txt 
$ cat symlink.txt 
cat: symlink.txt: No such file or directory

You'll see that unlike hard links, a symbolic link is separate from the original file. In this example, we create a file using echo to redirect text, then create a symbolic link to it. Note that we use the -s switch with ln. By default, ln creates a hard link. The -s switch instructs ln to create a symbolic link. If we try the same delete-and-use-it test that we did with the hard link, we're going to be disappointed. As shown, after removing the original file, the symlink points to nothing.

While it may seem the hard links are better in some respects, symlinks are actually a little more flexible. Since inodes are only unique to a given file system, hard links cannot span across file systems. Another limitation of hard links is that they can only point to files, not directories. A common use of symlinks is to use a common name to point to the latest distribution directory of an application or web directory. For example:

web-site-1.0/
web-site-2.0/
htdocs -> web-site-1.0

In this case, web-site-1.0 is the current site. Once web-site-2.0 is complete and ready to become the main site, all that needs to be done is to point the symlink to the new directory.

Other Ways to Integrate

Now that I have these basics out of the way and I have easy access to my Dropbox folder and a variable with the Dropbox folder path, what do I use it for? First, I have a script that configures a new machine to my liking.

In my Dropbox folder, I store my ~/bin directory. The idea is that I can symlink to it from my home directory. (I've actually changed how I do this-read until the end of this article to read how I now handle this). Inside of ~/bin, I keep a setup script that configures a new machine for my use. One thing that this script relies on: the $DB variable-it needs to know where I've configured the Dropbox folder to be. On (most) any new machine, my workflow is something like this:

1. Create login account

2. Install Dropbox

3. Run ${DB}/bin/setup_new.sh - the setup script

What does this setup script do? I'll just generalize here as the script is incredibly specific to the way I work. However, the important thing is that it can bootstrap my environment just from the Dropbox folder.

First, it symlinks ~/bin folder back to itself. Since we already know the path, this is pretty easy:

ln -s ${DB}/bin ${HOME}/bin

A nicer way to handle this involves some logic and error checking:

#!/bin/bash
# Setup links to bin folder
ERROR=0
if [ -d ${HOME}/bin ]; then
  mv ${HOME}/bin ${HOME}/bin.old
  if [ $? != 0 ]; then
    echo "ERROR: Could not move current ~/bin directory"
    ERROR=1
  fi
fi
if [ -f ${HOME}/bin ]; then
  echo "ERROR: ~/bin already exists - not touching"
  ERROR=1
fi
if [[ $ERROR == 0 ]]; then
  echo "Linking ${HOME}/bin -> ${DB}/bin"
  ln -s ${DB}/bin ${HOME}/bin
fi

Conditional statements will be looked at in more detail in a future column.

The second thing my setup_new.sh script does is to call another script that runs several defaults commands that tweak things just the way I like them. (Yes, I like seeing all files in the Finder, and no, I do not want a warning when I empty the trash). I keep it in s separate script just because I like the modularity and possibility of easily running it without needing to run the entire setup_main.sh script.

Summary and Epilogue

This article demonstrates one way to integrate your shell experience with a perhaps not-so-obvious GUI element. In this case, once your shell knows where the Dropbox folder is, it can perform any manner of syncing action. Which reminds me: I said I'd talk a little bit about how I'm doing things differently now.

I did start off with the same system that I mention here-just symlinking files and folders over to certain locations in my Dropbox folder. In fact, I still do that with certain files and folders, but have begun a hybrid strategy. For my ~/bin and ~/dev folders, I just store my git repository for each on Dropbox. This way, the new machine setup scrip can just check out bin and dev into the new home folder. Any changes in ~/bin and ~/dev can get checked in when ready. Those changes will then be staged on Dropbox, ready for checkout by any other of my Dropbox clients.

If that sounds a little convoluted, well, it works really well for certain workflows, and not so much for others. That's why I have a hybrid approach. However, even files and folders that are symlinked directly into Dropbox are kept under version control (git).

Media of the month: "User Interface Design for Programmers" by Joel Spolsky (http://apress.com/book/view/1893115941). This is a bit of an oldie, but still a goodie. This book is a great read to get you outside of the tech mindset to help you design applications so they have an interface that humans can use.

See you next month with more back-to-the-shell topics.


Ed Marczak is the Executive Editor of MacTech Magazine. He has written the Mac in the Shell column since 2004.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

ExpanDrive 5.4.1 - Access cloud storage...
ExpanDrive builds cloud storage in every application, acts just like a USB drive plugged into your Mac. With ExpanDrive, you can securely access any remote file server directly from the Finder or... Read more
Espionage 3.6.6 - Simple, state-of-the-a...
Espionage offers state-of-the-art encryption and plausible deniability for your confidential data. Sometimes, encrypting your data isn't enough to protect it. That's why Espionage 3 goes beyond data... Read more
Pinegrow Web Designer 2.94 - Mockup and...
Pinegrow Web Designer is desktop app that lets you mockup and design webpages faster with multi-page editing, CSS and LESS styling, and smart components for Bootstrap, Foundation, Angular JS, and... Read more
1Password 6.3.3 - Powerful password mana...
1Password is a password manager that uniquely brings you both security and convenience. It is the only program that provides anti-phishing protection and goes beyond password management by adding Web... Read more
Sublime Text 3126 - Sophisticated text e...
Sublime Text is a sophisticated text editor for code, markup, and prose. You'll love the slick user interface, extraordinary features, and amazing performance. Features Goto Anything. Use Goto... Read more
ForkLift 3.0 Beta 2 - Powerful file mana...
ForkLift is a powerful file manager and ferociously fast FTP client clothed in a clean and versatile UI that offers the combination of absolute simplicity and raw power expected from a well-executed... Read more
OmniFocus 2.7.1 - GTD task manager with...
OmniFocus helps you manage your tasks the way that you want, freeing you to focus your attention on the things that matter to you most. Capturing tasks and ideas is always a keyboard shortcut away in... Read more
CleanApp 5.1.1 - Application deinstaller...
CleanApp is an application deinstaller and archiver.... Your hard drive gets fuller day by day, but do you know why? CleanApp 5 provides you with insights how to reclaim disk space. There are... Read more
Together 3.6.1 - Store and organize all...
Together helps you organize your Mac, giving you the ability to store, edit and preview your files in a single clean, uncluttered interface. Features Smart storage. With simple drag-and-drop... Read more
Cloud 4.1.1 - File sharing from your men...
Cloud is simple file sharing for the Mac. Drag a file from your Mac to the CloudApp icon in the menubar and we take care of the rest. A link to the file will automatically be copied to your clipboard... Read more

Zip—Zap (Games)
Zip—Zap 1.01 Device: iOS Universal Category: Games Price: $1.99, Version: 1.01 (iTunes) Description: Touch to contract.Release to let go.Bring the clumsy mechanical beings home. · · · over 100 levelsno adsno in-app-purchases Zip—... | Read more »
Paperback: The Game (Games)
Paperback: The Game 1.0 Device: iOS Universal Category: Games Price: $3.99, Version: 1.0 (iTunes) Description: You are an author trying to finish kitschy paperback novels. Complete Westerns, Science Fiction, Romance or even a Crime... | Read more »
How to Rule With a Firm Hand in My Majes...
My Majesty is a kingdom management sim not unlike August’s magisterial hit, Reigns. It’s essentially a reskin of developer Tigrido’s previous management sim, Dictator. As supreme ruler of the land, you must consult with a number of subjects to... | Read more »
Our 5 Favorite iMessage Sticker Packs
At long last, iMessage joins the ranks of messaging apps the likes of LINE and Whatsapp, adding an impressive collection of stickers. They’re a great way to add a little something extra to your daily conversations. [Read more] | Read more »
How to get past Vulture Island's tr...
Vulture Island is a colorful and quirky mish-mash of platforming and puzzles. It’s creative and fresh, but sometimes the game can throw a curveball at you, leaving you stuck as to how you should progress. These tips will help you explore smoothly... | Read more »
The new Clash of Kings is just for Weste...
If you’ve played the original Clash of Kings, you’ll probably recognise the city building, alliance forging and strategic battles in Clash of Kings: The West. What sets this version apart is that it’s tailor made for a Western audience and the... | Read more »
Frost - Survival card game (Games)
Frost - Survival card game 1.12.1 Device: iOS Universal Category: Games Price: $3.99, Version: 1.12.1 (iTunes) Description: *Warning: the game will work on iPhone 5C and above and iPad Pro / 4. Other devices are not supported* | Read more »
How to build and care for your team in D...
Before you hit the trail and become a dog sledding legend, there’s actually a fair bit of prep work to be done. In Dog Sled Saga, you’re not only racing, you’re also building and caring for a team of furry friends. There’s a lot to consider—... | Read more »
How to win every race in Dog Sled Saga
If I had to guess, I’d say Dog Sled Saga is the most adorable racing game on the App Store right now. It’s a dog sled racing sim full of adorable, loyal puppies. Just look at those fluffy little tails wagging. Behind that cute, pixelated facade is... | Read more »
Let the war games commence in Gunship Ba...
Buzz Lightyear famously said, “This isn’t flying, this is falling – with style!” In the case of Gunship Battle: Second War, though, this really is flying - with style! The flight simulator app from Joycity puts you in control of 20 faithfully... | Read more »

Price Scanner via MacPrices.net

15-inch 2.2GHz Retina MacBook Pro on sale for...
B&H Photo has the 2015 15″ 2.2GHz Retina MacBook Pro (MJLQ2LL/A) on sale for $1799, including free shipping plus NY sales tax only. Amazon also has the 2015 15″ 2.2GHz Retina MacBook Pro (... Read more
Toughbook Celebrates 20 Years of Ruggedized M...
Panasonic System Communications Company of North America, Division of Panasonic Corporation of North America (Panasonic) today celebrates the 20th anniversary of its industry-leading Toughbook mobile... Read more
12-inch 1.1GHz Gray Retina MacBook on sale fo...
B&H Photo has the 2016 12″ 1.1GHz Gray Retina MacBook on sale for $1199.99 including free shipping plus NY sales tax only. Their price is $100 off MSRP. Read more
13-inch 2.5GHz MacBook Pro (Apple refurbished...
Apple has Certified Refurbished 13″ 2.5GHz MacBook Pros available for $829, or $270 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 13″ 2.5GHz MacBook Pros... Read more
Save 30% on Camera Guard’s Secure Protection...
To celebrate the release of macOS Sierra, Miami-based security solutions company, ProtectStar has announced a special 30% discount on Camera Guard Professional for Mac 2016. This innovative security... Read more
DVDFab Special Deal – Get a 1-Year Free Licen...
Beijing, China based specialist in the field of DVD, Blu-ray and video backup solutions, Fengtao Software has launched its Autumn Special Deals 2016, giving a 1-year free license of a randomly picked... Read more
21-inch iMacs on sale for up to $120 off MSRP
B&H Photo has 21″ iMacs on sale for up to $120 off MSRP including free shipping plus NY sales tax only: - 21″ 3.1GHz iMac 4K: $1379 $120 off MSRP - 21″ 2.8GHz iMac: $1199.99 $100 off MSRP - 21″ 1... Read more
13-inch 2.7GHz/256GB Retina MacBook Pro on sa...
Amazon.com has the 13″ 2.7GHz/256GB Retina Apple MacBook Pro on sale for $151 off MSRP including free shipping: - 13″ 2.7GHz/256GB Retina MacBook Pro (sku MF840LL/A): $1348 $151 off MSRP Read more
Apple TVs on sale for up to $50 off MSRP
Best Buy has 32GB and 64GB Apple TVs on sale for $40-$50 off MSRP on their online store. Choose free shipping or free local store pickup (if available). Sale prices for online orders only, in-store... Read more
Apple refurbished 13-inch Retina MacBook Pros...
Apple has Certified Refurbished 13″ Retina MacBook Pros available for up to $270 off the cost of new models. An Apple one-year warranty is included with each model, and shipping is free: - 13″ 2.7GHz... Read more

Jobs Board

*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Restaurant Manager (Neighborhood Captain) - A...
…in every aspect of daily operation. WHY YOU'LL LIKE IT: You'll be the Big Apple . You'll solve problems. You'll get to show your ability to handle the stress and Read more
Sr. *Apple* Mac Engineer - Net2Source Inc....
…staffing, training and technology. We have following position open with our client. Sr. Apple Mac Engineer6+ Months CTH Start date : 19th Sept Travelling Job If Read more
*Apple* Retail - Multiple Positions-Norfolk,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Restaurant Manager (Neighborhood Captain) - A...
…in every aspect of daily operation. WHY YOU'LL LIKE IT: You'll be the Big Apple . You'll solve problems. You'll get to show your ability to handle the stress and Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.