Sparkleshare for dummies.

Of course, there is, I guess, an argument that Sparkleshare isn't for dummies.

What is this document?

It's an explanation of how I set up SparkleShare so that it does pretty much everything that Dropbox does, but mostly slightly better (for my purposes), and sometimes slightly worse. The bottom line is that I now have a SparkleShare directory on multiple computers, all synched, and hence I have a very easy way of passing files and directories between multiple computers. Moreover it's quicker than Dropbox (for me), and instead of a free 5 gig limit, crucially, I have a "free" 1 terabyte limit (so I can use it to do things like backing up my partner's laptop without her ever having to connect it to an external hard drive). Note that the terabyte itself wasn't free -- I didn't have one handy, so had to buy an external hard drive. But, crucially, I am not paying any monthly fees. Also, I can only have one Dropbox, but I can have several SparkleShare directories, some shared with my family, some with work colleagues, and some just for me. There are some catches, but I'll come to those later.

Is this for me?

If you've never used Dropbox, you should try that first, to see if it works for you (it works for me, but for one problem it didn't work for me, because I needed a lot more than 5 gigs but didn't want to pay a monthly fee). And if you've never used unix, then this document below almost certainly is not for you.

Ingredients

Here is what I needed, and this will be one of the reasons that SparkleShare isn't for everyone: I needed, but fortunately had, root access to an (Ubuntu) Linux machine, connected to the internet, running an ssh daemon, and which was always on 24/7. I also bought a 1 Terabyte external hard drive to use as storage space -- but if you have oodles of free space on your own Linux machine then you can just use that instead. I also needed an understanding of some basic commands of the unix operating system, and apt-get, and the ability to open a terminal so I can type these commands in.

If you also have root access to an Ubuntu, or Debian, or RedHat Linux machine that is internet-enabled and on 24/7, then SparkleShare might be the thing for you too. However, because I knew essentially nothing about git or github, and the documentation on the SparkleShare website was (in my opinion) rather terse (I write this on 17th April 2013 and of course this may change in the future) the whole thing initially looked a bit daunting. But I've got it working now. Next I'll explain what I did, and then I'll explain my understanding of what is actually going on.

Making it all work.

The idea is that we'll first set up the Ubuntu Linux machine (with the Terabyte external hard drive) as the host, and then we'll set up various devices (a Windows machine, and a Linux machine (which may or may not be the host!)) as clients. Note that, unsurprisingly, you have to set up the host first before you set up the clients.

Setting up the host.

Host Step 1 (optional): sort out your external drive.

My host will be my Ubuntu Linux machine which I leave on 24/7 and which it's possible to ssh into. I don't want my stuff to be publically viewable by others, and I don't want to pay monthly fees, and I don't have much space left on my Ubuntu Linux machine, so I bought a terabyte hard drive, and this is where the files will actually be stored at the host end. One reason that using my own linux machine is is better (for me) than the choice used by Dropbox is that now all the computers I'm going to use as clients and the host are in the UK; with Dropbox every file is stored on central servers in the USA, so the to-ing and fro-ing takes longer.

Now unsurprisingly, my new terabyte hard drive was formatted in some Windows format, so the first thing I did was I reformatted the filesystem so that it was ext4. Note that this erases all the data on the drive, so it should be the first thing you do! There are lots of ways of doing this; google is your friend here. Note that if you decide not to reformat your drive, then there may or may not be problems later on. Note also that if you are not careful with formatting tools then you can end up reformatting your computer's hard drive (instead of your new external hard drive) and losing everything. So be careful.

The reason I wanted to use ext4 as the filesystem on my external hard drive is that later on there will be a new user created on the linux box called "storage", and various files will need the right permissions for this "storage" chap, and permissions on a multi-user system is one of the main things that ext4 does but FAT32, or whatever the usual filesystem format for these drives is, does not. I don't know what would have happened if I had not reformatted the drive. Maybe it would be still possible to get it to work, if you know more about things like /etc/fstab than I do.

OK so let's assume you've managed to either reformat your terabyte hard drive and get it plugged into your linux box, or that you've decided that your linux box will have enough space and will be fine on its own.

Host Step 2: set up the host.

This part was easy, because the nice folk at SparkleShare have done all the work for you. On their home page, no less, there is a description of how to set up the host. On 17th April 2013 the first few commands it said were this:

# Run all of the following commands as the root user on
# a Debian or Red Hat based Linux system
su

# Fetch the Dazzle script
curl https://raw.github.com/hbons/Dazzle/master/dazzle.sh \
  --output /usr/bin/dazzle && chmod +x /usr/bin/dazzle

# Run the initial setup
dazzle setup

Note that "su" doesn't work, by default, on my Ubuntu system -- but "sudo su -" does :-) and so that's what I did instead. I then copied and pasted those other two commands, and all the work was done for me. Note that we're going to be running a lot of commands as root, and I find it easier to use "sudo su -" and then actually being root for a while, than just using sudo again and again.

Host Step 3: let's make some "projects"

What is a project? You can think of your Dropbox directory as one "project". But actually I want two projects. I want to have one directory that only I can use on my various machine, and I want to have another one that all my family can use on all their multiple machines. So (still logged in as root) let's create some projects.

dazzle create FamilyShare

It's as easy as that! We've got a new project called FamilyShare. IMPORTANT THING: your linux machine just printed out something you need to remember: something like this:

Project "FamilyShare" was successfully created.
To link up a SparkleShare client, enter the following
details into the "Add Hosted Project..." dialog: 

      Address: ssh://storage@12.34.56.78:22
  Remote Path: /home/storage/FamilyShare

You'll need that later. Now let's do another one!

dazzle create MeShare

[again, remember what the output was, those last few lines of it at least.] Of course, you can create as many as you like. And if you ever want to create more later, e.g. a new project called "blah" that you want to share between yourself and two people you just met on the internet, just log back in as root and type "dazzle create blah", or, even easier, just type "sudo dazzle create blah" as a normal user.

Host Step 4 (optional): let's move our projects onto the external drive.

Somewhere on my linux box I now have two directories, called "FamilyShare" and "MeShare". Where are they? Well what has happened, when you ran "dazzle setup", was that a new user called "storage" has been created, and in his home directory, which is probably "/home/storage", there are now directories called FamilyShare and MeShare. I don't want them there though, I want them on my external hard drive! My external hard drive is mounted on /media/fishandchips; of course if you have any sense then yours will probably be mounted somewhere else. Here's what I did then (still as root!). First I created the directory where I actually wanted the files to be hosted:

cd /media/fishandchips/
mkdir SparkleShare
chown storage.storage SparkleShare/
chmod 700 SparkleShare/
cd SparkleShare/

[note of course that you need to change directory instead into the place where your external hard drive is mounted, so that first line needs changing at your end.]

And then, in that new SparkleShare directory, I'm going to move each of my projects, and in place of where they originally were I'm going to put a link to where I've moved them. So, for example, for my family's directory (a.k.a. project) FamilyShare, I would do this (still within /media/fishandchips/SparkleShare and still as root):

mv ~storage/FamilyShare/ .
ln -s /media/fishandchips/SparkleShare/FamilyShare ~storage/FamilyShare
chown storage.storage ~storage/FamilyShare

[noting of course that the fishandchips again needs to be changed to the place where you want to store your files, so probably your external hard drive if you're doing this step]. And of course I repeat this for any other project I want to move onto the external hard drive. I don't know whether I need those symlinks. I do know though, that this is the point where I am a bit unsure about what would have happened if the external hard drive's file system was not ext4; I think that for FAT32 the commands would "work" but would basically just be ignored, and then permissions would be all wrong, and probably it would be possible to sort it all out but I went for the easy option of reformatting the drive to a filesystem that plays better with unix, because I'm using it on a unix machine. Note also that things might have been a bit easier if I had actually been logged in as user "storage" above; then I wouldn't have had to keep chmodding and chowning everything; however by default it's not possible to log in as user storage. One can make it possible to log in as this user, but I am not sure of the ramifications of doing this, and what I did above seems to work fine.

OK, we're now done with the host! As far as I can see, you don't now need to start some daemon or whatever -- the job of the host is just to be there 24/7 and to provide ssh access. There may be more to it than this, but that's all that I understand.

Setting up clients.

This is where the programs available to download at SparkleShare.org come in. First, log into the computer that you want your new shared SparkleShare directory in.

Client Step 1: Download the appropriate program.

Easy, huh? Linux, mac or windows. As I said above, get the program from SparkleShare.org. Here's a hint: if you're running Ubuntu like me, then take the trouble to download the gzipped tarfile from the Sparkleshare website and follow the instructions there (it involves installing a bunch of stuff via apt-get and then compiling sparkleshare itself; the instructions are here if you're interested). The reason I recommend this, rather than typing "sudo apt-get install sparkleshare" and then skipping straight to step 3, is that, at the time of writing, building from source will give you stable version 1.0.0 of Sparkleshare, whereas going via apt-get will only give you version 0.8.2. You can see what apt-get will give you by typing the following:

sudo apt-get -s install sparkleshare

This will not actually install anything; it will just tell you which version of sparkleshare apt-get would install if you asked it.

Client Step 2: Install it!

This is well-documented on the SparkleShare website. So read the instructions there. Note that for a unix machine it involves some gunzipping, untarring, apt-getting, sudoing and so on; plenty of work there. Follow your nose.

For a Windows machine it just involves downloading an installation file from the sparkleshare website, and running it as an administrator. Follow your nose. Note though that, as far as I can see, actually running sparkleshare itself does not require administrative privileges, you'll just need them for the installation process.

Client Step 3: Get SparkleShare running on the client machine.

On a Windows machine, you can just search for a program called sparkleshare.exe and run it. If you were paying attention to where you actually installed SparkleShare, you can navigate there and you'll find the exe in that directory. Double-click!

On a linux machine, type "sparkleshare start" (you don't have to be root, or use sudo).

Using either of these methods, you'll now have a SparkleShare directory, probably in C:\Users\[your login] on a Windows machine, and in /home/[your login] on a linux machine. Go into it and you'll find a file called something like "Fred's link code.txt" ("Fred" will be replaced by whatever name you gave sparkleshare when you set it up). That file is a text file and it contains what looks like four or five lines of garbage. Don't delete it -- those lines are really important for the next step :-)

Client Step 4: Tell the host about the client.

OK, back to the host machine! The host needs to be told to allow the client in. So, as root on the host linux machine, type

dazzle link

(or "sudo dazzle link" if you're not root). The host will ask you for the client's link code. That's where those 4-5 lines of garbage from step 3 go. I used Dropbox to move the link code over ;-)

Client Step 5: Tell the client about the host.

Now back to the client machine! You've started the sparkleshare program, so somewhere there will be a little sparkleshare icon. On a Windows 7 machine it will be in that little icon tray down in the bottom right, near your Dropbox icon. On my Ubuntu 12.04 machine it's in the little bar on the top. It looks like a directory with a star in. On Windows it's white, on linux it's orange. Go figure. Anyway, find it, click on it, and go for "SparkleShare->Add Hosted Project" and we find ourselves faced with a window and a question or two. Where's our project hosted? On our own server -- that's the point. Now SparkleShare wants us to choose an "address" and a "remote path". That was the stuff that was printed out when we created the project way back in Step 3 of setting up the host.

OK so for "address" I'm going to put

 ssh://storage@12.34.56.78:22

(of course you'll need to change the IP address to the one of your host), and for "Remote Path" I'm going to put...well...I'm going to put

/media/fishandchips/SparkleShare/FamilyShare

because although SparkleShare created the FamilyShare project in /home/storage, I didn't want it there, I wanted it on my external hard drive. If you didn't move the project, of course you just type in the path to where SparkleShare put it.

Client Step 6: adding more computers.

Just repeat steps 1 to 5 as many times as you like, on as many computers as you like. Your project can appear on several computers -- indeed, this is the point.

We're done! Well, it worked for me.

Sparkleshare v Dropbox: Advantages and disadvantages.

Note that I am writing this as a not-particuarly-savvy end-user.

First here are some reasons why I like my Sparkleshare more than my Dropbox.

One Dropbox is problematic for me.

I have a nice dropbox set-up, used by me and my entire family, including my kids. But I can't quite trust my kids (e.g. one of them recently deleted some files belonging to another one of them, out of malice), and hence I can't quite trust my dropbox set-up. Furthermore, there are some files which I would like to pass between home and work computers, but which contain "confidential" work stuff (e.g. references I am in the middle of writing) -- I don't want them to be publically available to my children. But I can't (as far as I know) have two Dropboxes running on our home Windows machine. This issue is solved with SparkleShare.

Dropbox only gives me 5 gigs for free.

When your son is developing a game for his phone and the next thing you know, there are 20 apk files in your dropbox and they're all 10 megs, and you don't want to nag, and then a week later there are 30 apk's, and then your partner has a 1 gigabyte video that she has made at work and wants to pass to someone else and she decides that she'll use the Dropbox to do this -- when you and your family are all active Dropbox users -- then 5 gigabytes suddenly doesn't seem like such a large amount. Of course I can fix this by paying a monthly fee to Dropbox. But I can also fix this by buying a terabyte hard drive (a one-off investment) and using SparkleShare, for free.

Dropbox can see my files.

Well, google can read my email. Do I really care? I'm not sure I do, to be honest -- but with SparkleShare no-one can see my files except the people I decide I want to share them with.

The Dropbox servers are a long way from me.

They're in the USA, in fact. So if I want to move a file from the downstairs Windows computer to the upstairs unix computer, it is somehow a little counterintuitive to move it via the USA (I am in the UK). Not only is it counterintuitive, but in practice it's much slower than just moving the files via a computer that is only physically a few miles away.

Now what about the disadvantages?

Sparkleshare seems harder to use on phones.

I have an Android phone and I use ES File Explorer to navigate around it. ES File Explorer treats my Dropbox directory as just another directory on the phone, and hence I have seamless integration between Dropbox and my phone. There was a SparkleShare Android app, but apparently (17th April 2013) it no longer works. No doubt another one will come. Until it does, one simple solution is "plug the phone into a computer where SparkleShare is running". Eew. Wires. Another solution is "get a git client running on your phone.". I need to invest some time in finding out how this works **TODO**.

Version control has its disadvantages too.

Under the hood SparkleShare is using Git. Git has version control. So imagine in theory that you have an encrypted 1 gig file, and you want to change one byte of it. You unencrypt it, change the byte, and re-encrypt it. Git then carefully remembers both the new version (which it makes publically visible to you) and the old version (which is harder to find, but still there). In short, you are now using 2 gigs of space to store one gig of file. That may or may not be what you want.

I cannot imagine this being an issue for me, and here is the fix I envisage if it does become an issue: just create a new project, and copy all the files over, and then physically delete the files in the old project on the host machine.

My host machine is less reliable than Dropbox's.

One day my external hard drive will fail. One day my computer will crash. Or its ssh daemon will randomly die. One day someone will come around to test all the sockets in my office and they'll unplug my computer and then not plug it back in again, and it will be the day I've just left to go on holiday, or to a conference. Hence one day my SparkleShare directories won't work, at least temporarily. And what if that's the day that my kid desperately needs it to work for some reason? Dropbox is going to be much more stable. This can be fixed by using something like Github, but then your projects become publically-viewable, unless you pay -- and if you pay then you might find life easier just paying Dropbox.

What I'm trying to say is that there are swings and roundabouts here. Do you want SparkleShare or Dropbox? Personally I'm currently using both. Time will tell which one I'll ultimately find more useful.

Finally, many thanks to hbons for answering several questions of mine on IRC.