CVS to Subversion Migration – a war story

Advertisements

One of the things that was annoying us at Wotif for some time has been how slow our CVS repository had been getting. Doing a tag after a release, for example, could take up to 15 minutes, CruiseControl was taking up to 45 minutes to do a scan, that sort of thing. Basically, the server we were using was almost literally having the hard disks melt… a 4GB CVS repository was being hammered almost continually. So, we finally bit the bullet a few weeks ago, and moved to Subversion, and I thought I’d share the experience.


We’d originally planned a slow transition to Subversion. However, after a necessary OS upgrade on the server, the CVS performance absolutely tanked – we’d lost some hard disk tuning settings. So it became a bit rushed. Quite literally, the migration got done over night.

h2. Creating the Subversion repositories

We used the “cvs2svn”:http://cvs2svn.tigris.org tool to do the actual migration. I was pleasantly surprised at how fast it was – creating the Subversion dumps took about 3 hours to do (which, considering it was 4 years of history, and 4GB in size, is damn good). We’d decided to create a few repositories, one for each important module in our CVS repository, so part of the process involved shuffling directories around. In the end, the largest repository came in with about 6300 revisions.

h2. Setting up the server

I’d decided to use the Apache mod_svn module as a front end, running over https. I followed the instructions in “Pragmatic Version Control With Subversion”:http://www.pragmaticprogrammer.com/titles/svn/ on this, and had it up and running within a few minutes.

We’d previously experimented with running ssh+svn, but this had a limitation in that you needed everyone to have a shell account and write access to the repository; our IT admins had objections to that (which was fair enough – they need to look after the box, after all). Using Apache meant that we could use one of the many Apache authentication options – that said, I went with the simplest and used the htpasswd file.

h2. Setting up FishEye

We use “FishEye”:http://cenqua.com/fisheye at Wotif, and it supports Subversion. Setting up Subversion access in FishEye was fairly easy – we just followed the instructions in the doco.

One thing we were not prepared for was how long FishEye took to “slurp” in the SVN repositories. The largest repository took nearly 30 hours – all up, it was a bit over 40 hours. The culprit was that we used the JavaSVN client, rather than the native JavaHL bindings; we also had FishEye set up on another box, rather than the server where Subversion was located. This made the calls a little slow… by contrast, I set up a FishEye instance on a box with a copy of the repositories, using JavaHL and the file:/// protocol (as recommended by Cenqua), and it took about 45 minutes to do the full index. Just a little faster… 😉 and we’ll look at relocating our FishEye instance soon.

h2. CruiseControl

Replacing the CVS setup in “CruiseControl”:http://cruisecontrol.sourceforge.net with Subversion was trivial, and we were very happy that the 45 minute “checking for modifications” period dropped to about a second. 🙂

One mildly annoying thing was that the Subversion revision number isn’t shown in the reporting web application; it is in the log file, but not shown in the app. We could have changed our stylesheet, but we went for something a little different – we wrote a Subversion-based label incrementer. Now the labels in CruiseControl have the repository number built in. This seems to be working well.

(Previously, we had a custom Publisher that did a CVS tag – this was no longer needed)

h2. IDE configurations

We use both IntelliJ and Eclipse in our environment. IntelliJ has Subversion built in, with about the same level of support as their CVS support – that is to say, not very much. Eclipse doesn’t have any native Subversion support, but there are two plugins – “Supclipse”:http://subclipse.tigris.org and “Subversive”:http://www.polarion.org/index.php?page=overview&project=subversive. I’m personally using Subclipse, but a couple of other people are trying out Subversive; we’ll see how it goes over time.

I can say that we’re not really happy with the Subversion plugins for Eclipse – they just aren’t as good as the CVS support. I really hope that the Eclipse project decides to make SVN support standard in the next major release of Eclipse, with the same level of support as CVS – particularly the preview merge support.

h2. Pain points so far

The biggest pain point we’ve had so far has been merges. We use parallel development of projects in a few different branches, with a “main line” that we regularly sync up with. With CVS, we’d got this down fairly pat, and the excellent support in Eclipse for CVS merges made the process fairly smooth. We’re still figuring out the best way to do this in Subversion – the process I’ve settled on for now is using the command line, saving the output, and then rectifying conflicts (using the “Edit Concflicts” feature in Subclipse) and fixing any “skipped” files. I then examine the result using the Synchronise view in Eclipse before committing. I’ve done two merges under Subversion now, and the second one wasn’t _too_ painful. We’ll see how this progresses over time.

We also had a nasty problem with a Java keystore file that was in our repository but NOT marked binary; this caused a couple of our branches to be unusuable until we fixed it.

h2. Overall results

Personally, I’m happy with the result of the move. We had the expected effect on the speeds of tagging, branch, logging, etc – Subversion is really good in this area. We aren’t yet as skilled wtih Subversion as we were with CVS, but that will come in time.

Subversion is proving to be a good successor to CVS – it certainly has picked up a lot of momentum over the last two years. It could use a few features (I’d really like merges to be recorded in the history, ala ClearCase), but overall, it’s pretty impressive.

Author: Robert Watkins

My name is Robert Watkins. I am a software developer and have been for over 18 years now. I currently work for people, but my opinions here are in no way endorsed by them (which is cool; their opinions aren’t endorsed by me either). My main professional interests are in Java development, using Agile methods, with a historical focus on building web based applications. I’m also a Mac-fan and love my iPhone, which I’m currently learning how to code for. I live and work in Brisbane, Australia, but I grew up in the Northern Territory, and still find Brisbane too cold (after 16 years here). I’m married, with two children and one cat. My politics are socialist in tendency, my religious affiliation is atheist (aka “none of the above”), my attitude is condescending and my moral standing is lying down.

3 thoughts on “CVS to Subversion Migration – a war story”

  1. If you can do the merge from the command line, you should be able to do it using Subclipse. One thing that seems to throw some people off is selecting revisions. The way Subclipse and TortoiseSVN do this is designed to make it easier for you. When you select revisions, instead of trying to select the number you want to use for the merge, you select the revisions you want to merge. You can select a range. Subclipse will then input the revision numbers you would need to enter to merge those revisions.

    Of course you can also just type the numbers yourself if you know them.

    I personally greatly prefer the Subversion merge to the way CVS does it. I like having Subversion do the merge and do all the updates to my working copy. Subclipse puts problem markers on any file that has conflicts, so you can just use the Problems view to find the files that need manual editing. The quick fix option from that view provides shortcuts to all of the actions you would need to do. For example if one of the files is a JAR, you would likely just want to take one version or the other and the quick fix lets you do this easily.

    Mark

  2. Hi,

    We are setting up a website similar to that of wotif.com – does anyone know who the software developers were? and if they would be interested in quoting on our business idea after signing a NDA and reviewing a systems requirements doc.

    Wotif is one of the most comparable websites to our business idea. We need an architect and software development team to rollout and develop the site asap.

    Let me know of any names or suggestions. We are currently obtaining a number of quotes but I am impressed with the wotif site and would love to speak to their developers – thanks A

  3. Well, I’m one of the developers for Wotif.com. Unfortunately, it would be a breach of contract to assist a direct competitor, so if Wotif.com is really comparable (ie, you’re building a website to provide access to a mass-market of hotels), then neither I nor any other Wotif.com developer would be able to help.

    I can point out that the Wotif site has taken something like 60-75 developer years to get to its current form, and our customer facing site is just the tip of the iceberg, so to speak.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s