iizip - Hacking Together Your Own Dropbox

Ben Dechran, Sputnik Agency (Australia)

• works for a marketing agency, lots of photoshop files thrown around
• file server which works great most of the time unless ...
  • people forget to put the files on the file server
  • people forget to rename a file before putting it on the file server and delete the old file
  • people don't put a file on the file server and go on vacation
• dropbox nice solution, but has issues
  • works well for small companies, small groups, unconferences, etc.
• problem #1 -- storage space
  • in our case, needed 3TB of space
• problem #2 -- need internet connection
  • if connection goes down, you have the files but you lose the sync
• problem #3 -- dependent upon third party
  • what if they go down?
• problem #4 -- all or nothing
  • every person replicates everything
  • dropbox supposedly doing selective sync
• problem #5 -- user management
  • users can't be managed by your company's infrastructure
• problem #6 - big bad world
  • security and privacy
  • it's encrypted, but you're sending data out to someone else's servers
• other options
  • ifolder - open source project by novell, written in .net
    • tricky to set up
    • slow
    • packaged for windows, mac, opensuse
• decided to roll my own--needed ...
  • local files
  • monitor for changes
  • versioning
  • conflict resolution
  • transparent -- install and forget
• attempted solutions
  • tortoisesvn -- people forgot to check in, etc.
  • cron job to push changes to svn
    • some of the changes were hard to detect
    • if someone deletes a directory on their hard drive, deletes the metadata
  • tried git -- since it's distributed it solved some of the issues with svn
• for synchronization
  • inotify, inotifywait -- in the linux kernel
  • notify-send -- could use this for bubble type notifications in dropbox
• for conflict resolution
  • grep the git status
• iizip is the combination of all these various tools
  • currently the project is a bunch of scripts; eventual goal is to have a package that would be deployed/installed on local machines
  • iizip-init -- creates git repository and iizip directories
  • the scripts mostly map to the git commands
    • e.g. send checks in locally and also pushes to remote repository
• someone in the audience suggested looking into using couchdb in similar fashion to how ubuntu one works
  • http://arstechnica.com/open-source/guides/2009/12/code-tutorial-make-your-application-sync-with-ubuntu-one.ars
• what's in the pipeline?
  • partial checkout so not everyone has to have every file locally
    • idea of subscribing to specific directories--IMAP already does this
  • push changes from other machines
    • currently there's a cron job that runs on the local machine
    • dropbox immediately pushes changes to all machines
    • considered using xmpp to handle this, but IMAP has persistent connections and also supports directories
  • multi-user
    • currently can't run more than once on the same machine since the directories are hard-coded
• unix tools philosophy -- many small tools linked together to achieve a larger goal
  • also makes it easier to port to other operating systems
• unison -- would in theory do all of this so worth checking out
• potential issues -- git doesn't track empty directories
  • if it's empty you may not care