Building Dependent Maven Projects in Bamboo

For the last year or so, I’ve been using Atlassian’s Bamboo (in the OnDemand variant) for our team’s build server. And, mostly, it’s an awesome tool. Some parts, however, are a little rough around the edges. Building dependent projects is one of them.

The Problem

First, let me describe the way I want a tool like this to work:

1. I should be able to have projects. Let’s call them “Project A” and “Project B”.
2. These projects should be able to have a dependency relationship. In particular, Project B should be dependent on Project A – it should need an artefact produced by Project A in order to build.
3. When Project B builds, it should automatically get the most recent version of that artefact produced by Project A.
4. When Project A builds, it should trigger a new build of Project B (which, in turn, should get the latest version of the artefact from Project A).
5. Project A and Project B should not have to build on the same server. In particular, because the OnDemand version of Bamboo only uses “Elastic Agents” – EC2 instances running inside Amazon Web Services – they should be able to build on clean images.
6. This should support branching. When building a branch of Project B, if there is a corresponding branch for Project A, that artefact should be used by preference. However, if there is no branch, then the “master” copy of the Project A artefact should be used instead.
7. I should be able to do this without altering the POMs, and without additional infrastructure outside of the build server itself.

The first few are reasonably common. When I was at Wotif, we had this set up with Jenkins using their Maven Dependency Analyser plugin. That worked okay – but we used a build server per project, with long-lived project branches. With my current team, we only have one build server, and we’re using short-lived Feature Branches – so a build server per project is out.

Requirement 5 starts to become an issue. This means you can’t do a simply ‘mvn install’ and use the local repository as a way of sharing artefacts between builds – the builds could easily be running on different VMs. So we start to need something more sophisticated.

Requirement 6 also rules out simply using an external Maven repository server, which you deploy SNAPSHOT builds to – because there is no server that supports this structure. You could do it with a hierarchy of servers – so that the build queries the branch-specific server first, which will fallback to the master one afterwards – but this isn’t something you’d want to manage by hand.

The Solution

Bamboo, as it turns out, has great support for branch management. In particular, you can get it to spawn a branch build automatically, using a simple naming convention for your branch names. (We use ‘story/XYZ’). This automated-branch-management is very powerful, and is a step-up from the branch management seen in Jenkins (which can handle merging of branches, but can’t easily manage separate branches). TeamCity looks like it has a similar feature, but then TeamCity doesn’t have a hosted solution. Oh, and TeamCity sucks, too – that’s just my opinion, of course.

To top this off, Bamboo has two other features that come into play: Dependency Analysis, and Artefact Sharing. These features are very powerful – but also a little crippled.

Dependency Analysis

Let’s start with the dependency analysis. You can enable this by adding the Maven Dependency Processor to your project’s tasks – at the end. I’m not sure, but I believe you need to add it to both the child and parent projects – Project B and Project A, respectively. You then need to enable the automatic dependency management, and then the projects will sort out their dependency relationships over the course of a couple of builds. After this, they’ll handle requirements 2, 4, and 6 (as listed above) automatically.

This isn’t perfect. For starters, the dependency processor analyses both the incoming dependencies of the project, and the outgoing artefacts of the project. Seems a bit silly to me. Also, why isn’t this just an option on the Maven 3 builder – why need a second task for this? But it does work.

Artefact Sharing

Artefact sharing, however, is crippled. Seriously so. So much so that I couldn’t believe it didn’t work – especially as there was (at the time) documentation implying that it did. However, I got confirmation: Bamboo 5 does not support automated sharing of Maven artefacts. Bummer. But you can fake it – it’s more complex, and you need to do it by hand, but it does work, and it solves Requirements 2, 5 and 6.

Here’s how you do it. First, from the parent project (Project A), you need to publish both the pom.xml and the artefact. For my projects, I do this as a second Maven 3 task, with a command like this:

clean help:effective-pom -Doutput=target/projectA-api.pom.xml package -Djar.finalName=projectA-api

This occurs after the main build, so I know all the tests have passed. I generate the effective pom because, in my case, I’m publishing a sub-module (the API module from ProjectA), and because it means both of my artefacts are in the same directory. I also run this from with the projectA-api directory – not from the project root. I strip the version number so that I don’t need to worry about it in the artefact definitions later.

(If I was publishing multiple dependencies in this fashion, I would need to do this for each module – painful)

You then need to declare both the effective pom.xml and the binary artefact as an artefact for the job. For example, I defined an artefact group called ‘projectA-api’, with a location of ‘projectA-api/target’, and a copy-pattern of ‘projectA-api*’.

Once this is declared, subsequent builds will publish the artefact. Published artefacts will be available as long as the build record is available, but only the latest artefact (per branch!) will be able to be used by dependent projects.

Having defined the dependency in the parent project, it is necessary to use it in the child project. Again, this needs to be configured manually – it is not automatic. We start by adding an ‘Artifact download’ task to the job, just after the ‘Source code checkout’. In this, you download the artefact from the parent project, and put it in a sub-directory – I called that ‘dependencies’, but feel free to use whatever you want.

After that, you add a Maven 3 task – before the main one for your project – to install the pom and artefact, using a command like this:

install:install-file -DpomFile=projectA-api.pom.xml -Dfile=projectA-api.jar

(Note that the correct version number will be determined from the pom file – that’s why it wasn’t necessary to include it in the artefact file)

You can then proceed to build. Bamboo will then download the artefacts – from the branch of Project A with the same name, if that exists, or from the default build for Project A – and then build it. And the requirements set out above are now met.

Summary

This does work. This is also very far from perfect. It strikes me as insane that you have to go through this rigmarole to tell Bamboo information it already knows (from the Maven Dependency Processor step that set up the build dependencies) in order to accomplish something that is required for the OnDemand environment. Without artefact sharing, there is no point in having dependent builds – you would always need to build Project A in order to build Project B (because the newly clean VM for Project B wouldn’t have the artefact from Project A). It’s clearly not impossible to do – I just described a straight forward if manually laborious method to achieve it. So why not just build it in?

In fact, if you would like to see a proper implementation of Maven artefact sharing between dependent projects, I encourage you to vote for it at https://jira.atlassian.com/browse/BAM-13881. The more votes the merrier.