Thursday, June 3, 2010

iizip - Hacking Together Your Own Dropbox

Ben Dechran, Sputnik Agency (Australia)
  • works for a marketing agency, lots of photoshop files thrown around
  • file server which works great most of the time unless ...
    • people forget to put the files on the file server
    • people forget to rename a file before putting it on the file server and delete the old file
    • people don't put a file on the file server and go on vacation
  • dropbox nice solution, but has issues
    • works well for small companies, small groups, unconferences, etc.
  • problem #1 -- storage space
    • in our case, needed 3TB of space
  • problem #2 -- need internet connection
    • if connection goes down, you have the files but you lose the sync
  • problem #3 -- dependent upon third party
    • what if they go down?
  • problem #4 -- all or nothing
    • every person replicates everything
    • dropbox supposedly doing selective sync
  • problem #5 -- user management
    • users can't be managed by your company's infrastructure
  • problem #6 - big bad world
    • security and privacy
    • it's encrypted, but you're sending data out to someone else's servers
  • other options
    • ifolder - open source project by novell, written in .net
      • tricky to set up
      • slow
      • packaged for windows, mac, opensuse
  • decided to roll my own--needed ...
    • local files
    • monitor for changes
    • versioning
    • conflict resolution
    • transparent -- install and forget
  • attempted solutions
    • tortoisesvn -- people forgot to check in, etc.
    • cron job to push changes to svn
      • some of the changes were hard to detect
      • if someone deletes a directory on their hard drive, deletes the metadata
    • tried git -- since it's distributed it solved some of the issues with svn
  • for synchronization
    • inotify, inotifywait -- in the linux kernel
    • notify-send -- could use this for bubble type notifications in dropbox
  • for conflict resolution
    • grep the git status
  • iizip is the combination of all these various tools
    • currently the project is a bunch of scripts; eventual goal is to have a package that would be deployed/installed on local machines
    • iizip-init -- creates git repository and iizip directories
    • the scripts mostly map to the git commands
      • e.g. send checks in locally and also pushes to remote repository
  • someone in the audience suggested looking into using couchdb in similar fashion to how ubuntu one works
  • what's in the pipeline?
    • partial checkout so not everyone has to have every file locally
      • idea of subscribing to specific directories--IMAP already does this
    • push changes from other machines
      • currently there's a cron job that runs on the local machine
      • dropbox immediately pushes changes to all machines
      • considered using xmpp to handle this, but IMAP has persistent connections and also supports directories
    • multi-user
      • currently can't run more than once on the same machine since the directories are hard-coded
  • unix tools philosophy -- many small tools linked together to achieve a larger goal
    • also makes it easier to port to other operating systems
  • unison -- would in theory do all of this so worth checking out
  • potential issues -- git doesn't track empty directories
    • if it's empty you may not care

No comments: