Tuesday, March 22, 2011

Subversion + Apache Installation and Repository Migration HOWTO

I knew this day would eventually come. Nearly five years after I fired up our current Subversion server, it was time to bid farewell to our trusty Fedora Core 4 physical server and move to a shiny new VM running Ubuntu Server 10.04 LTS. Setting up the original SVN server was my first actual task when I started my current job so it was nice to look back at just how well things had run all this time. I don't think we had a single outage, repository corruption, or any problems whatsoever during that time. Sure, the time may have drifted rather severely since we couldn't ever get that thing talking to an NTP server for whatever reason, but other than that it ran more or less perfectly for nearly five years. What's a few hours among friends anyway?

In fact that box ran so well it was really far off my radar, so I got pretty lazy about doing patches and updates. As of today the uptime on that thing is 1135 days--a little over three years for the math-challenged among us--so honestly it's a bit sad to see the thing go. But you can't stop progress, and honestly I'm completely loving having all our servers virtualized.

My initial thought when faced with this migration was to take the lazy approach and start fresh on the new server with an empty repository. (Seen those "Hoarders" shows on TLC? I'm exactly the opposite.) Several of my co-workers, however, wanted to keep the revision and commit histories for their projects so I looked into what would be involved since I'd never migrated an SVN repository before.

Luckily there are tons of resources available, but since I just went through this I thought I'd document it for my own benefit if not for the benefit of others. Starting with a bare-bones install of Ubuntu Server 10.04 (and no offense to the Red Hat world--love you guys and use your stuff too ...), here's how I set up Apache and Subversion and migrated our 8+ GB of SVN history from Subversion 1.2 to Subversion 1.6.6.

Install Apache and Subversion

Every time I set up a new GNU/Linux server I'm reminded of how damn simple it is. I SSHd in and ran:

sudo apt-get install apache2 subversion libapache2-svn

And with that Apache 2.2, Subversion 1.6.6, and the necessary SVN hooks into Apache are up and running. I feel like I should type more here to make it seem more involved and like I had to work hard to do this but, well, that's the beauty of GNU/Linux.

Create Users and Groups

I'm not convinced all of this was absolutely necessary, but I followed directions from a couple of different sources and did the following, and the end result seems to be working how I want.

First, create a subversion group:

sudo groupadd subversion

Next, add your user and the Apache user (www-data) to the subversion group:

sudo usermod -a -G subversion YOURUSER
sudo usermod -a -G subversion www-data

You'll also want to create additional user accounts that you need on the box here (sudo useradd -m USERNAME) and add them to the subversion group.

Create a New SVN Repository

In our setup we take the "one big repository for everything" approach so if you need to configure things differently you'll have to look elsewhere.

First, create a directory to hold the repository:

sudo mkdir /home/svn
sudo mkdir /home/svn/repos

Next, create the repository:

sudo svnadmin create /home/svn/repos

Finally, set the ownership and permissions on the SVN directories:

sudo chown -R www-data:subversion /home/svn/repos
sudo chmod -R g+rws /home/svn/repos

Configure Apache to Work With Subversion

In our setup since we're behind a firewall we use basic HTTP authentication so again, if you need SSL or some different setup you'll have to look elsewhere.

Apache has a module called dav_svn that allows Apache to work with Subversion, so you'll need to edit the file /etc/apache2/mods-enabled/dav_svn.conf to get things humming along.

We don't want anonymous read access to our repository so our final file winds up looking like this (I'll omit the large comment block at the top of the file):

<Location /svn>

  # Uncomment this to enable the repository
  DAV svn

  # Set this to the path to your repository
  SVNPath /home/svn/repos

  # ommitting lots of comments here ...

  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile /etc/apache2/dav_svn.passwd
  Require valid-user

  # rest of the file is comments ...

The only real change here from basically uncommenting what was commented out in the file is that I put Require valid-user outside the commented-out <LimitExcept> block since we don't want any anonymous access whatsoever.

Create Basic HTTP Passwords

Next we'll create the SVN password file referenced in the dav_svn.conf file above.

Type the following to navigate to the /etc/apache2 directory and create the .passwd file and add the first user to it.

cd /etc/apache2
sudo htpasswd -cm dav_svn.passwd YOUR_USER_NAME

You'll be prompted twice to enter a password.

For all subsequent users you need to add to the SVN password file omit the -c flag (the -m flag indicates to use an MD5 hash when storing the password):

sudo htpasswd -m dav_svn.passwd USER_NAME

All the Apache configuration is done at this point so restart Apache:

sudo service apache2 restart

If all these steps were done correctly, you can go to http://server/svn in a browser and you should be prompted to log in, at which point you'll see your SVN repository at revision 0 since there's nothing in it yet.

Back Up Your Existing SVN Repository

Shout out to this post by Steve Smith and the SVN red bean book for giving me the pieces to put together here.

On your old server, back up your existing repository (in my case the old repository was located at /svn/repos):

sudo svnadmin dump /svn/repos > myoldrepos.svn_dump

I don't think the .svn_dump extension has any meaning whatsoever; I just followed what Steve had in his post.

Creating this file may take a while. As I said above our final file wound up being over 8GB, so be patient since you don't get incremental updates in the terminal while the dump file is being created. (Remember, in GNU/Linux land no news is good news.)

Next, since I had to move this file from one server to another I tarred and gzipped it which cut the file size in half:

tar cvzf svndump.tgz myoldrepos.svn_dump

Again be patient because this may take a while.

Restore Old SVN Repository to New Repository

Step one is of course to move the dump of the old repository to the new server. I simply SFTPd from our new server to the old one and transferred the file that way, which took about 3 hours for a file that was a little over 4GB in size.

Once the file was on the new server, I expanded it:

tar xvf svndump.tgz

Now for the magic--restore the SVN dump from your old server into your new repository:

sudo svnadmin load /home/svn/repos < myoldrepos.svn_dump

At this point you'll see every commit in your old repository being replayed into the new one, so as you can imagine this will take a while. We had about 9200 commits in our 8GB repo and it probably took about an hour for the load to complete.

Provided all goes well you should then be able to hit http://server/svn in your browser and instead of seeing revision 0, you'll now see the same revision number that was in place when you executed the dump on the old server.

Repoint Eclipse Projects to the New Subversion Server

The last step in this process for Eclipse users is point your existing Eclipse/SVN projects to the new SVN server. I use Subclipse so if you use another plugin you'll want to look into the process for your plugin of choice.

  1. Backup all your stuff. I didn't have any issues, but you've been warned.

  2. Go into the Subversion Repository Exploring perspective

  3. Add the new SVN server as a repository

  4. Right-click the old SVN repository and click "Relocate"

  5. You'll see a warning saying "this probably isn't what you want to do," but in this case it is. You'll also see a list of your projects that will be repointed. Click next.

  6. In the next box, enter the URL to your OLD svn server in the top box (full path to the repo, e.g. http://oldserver/svn), and enter the URL to your NEW svn server in the bottom box (e.g. http://newserver/svn). Click finish.

  7. After a few seconds you'll see your old repository location disappear from the list in the SVN Repository Explorer.

Next, verify that your projects got repointed successfully.

  1. Change into a perspective that contains the project explorer

  2. Right click on one of your projects that got repointed and choose "Properties"

  3. Click on "Subversion." You should see the new SVN server paths in the URL and Repository Root fields.

With that you're all set on a new SVN server. Note that I was moving from SVN 1.2 to SVN 1.6.6 and even with the version changes had no issues doing the dump/load process.

Now to work up the courage to shut down a server with 1135 days of uptime that did its job so valiantly. God's speed, little server. :-)

1 comment:

chiefwalker504 said...

A website for classifieds and reviews,our main focus is in mover reviews,auto transport reviews,auto relocation reviews,transport reviews,auto transport,car transport reviewsandcar haulingandvehicle shipping.