AiL – Subnets! Public Subnets! Wow, they are free, and we can au-to-mate!

(to the tune of the Ewoks Victory Song. Now it can be stuck in your head, and not just mine)

Picking up from last time, I needed to start creating things inside my Virtual Private Cloud, or VPC. The first things to create are subnets – public subnets, in particular. Without a public subnet, nothing that I run in the VPC can be accessed from the internet – nor can they access the internet in turn.

Continue reading “AiL – Subnets! Public Subnets! Wow, they are free, and we can au-to-mate!”

AiL – Baby’s First CloudFormation Stack

As previously discussed, I’m doing a small series of posts around bringing the AWS infrastructure that I use into the current era, and putting it all into CloudFormation. In this post, I’m going to cover setting up the first stack. This is going to set up a Virtual Private Cloud (or VPC), which is where the rest of the stuff I make later will sit.


What’s a Virtual Private Cloud?

A VPC is a virtual network of virtual servers. It’s your own mini-slice of the AWS cloud, and the machines within the VPC are aware of each other – in fact, they are on their own subnet (or subnets).

Why use a VPC?

You don’t have to set up a VPC to use AWS. You can simply create servers. That’s what I’ve been doing up until now. It’s just that it’s a bit limited.

I want to use a VPC for two big reasons:

  • I want to be able to use more recent/powerful/cheaper machine images, with OpsWorks. They’re only available if I also use a VPC.
  • I want to use an Elastic Load Balancer, in part to manage HTTPS certificates and connections. This requires a VPC and subnets.

Setting up the Stack

Here’s my config, at this particular stage:

I put the VPC into its own file, because I don’t want to delete it when tearing down an environment for the sake of testing. There’s lots of things that get annoying to re-create if the VPC is changed (cough OpsWorks stacks) – so we put the VPC in its own file. (Later, when I bring in nested stacks, this will not be in the nested stack).

Break It Down

First, note that I use YAML for my CloudFormation files. I’m not a huge fan of YAML in general, but the JSON option doesn’t allow you to use comments, and comments are essential. (Sidebar: when parsing JSON, always enable comments. It’s non-standard, but it’s useful). Using YAML also lets me use a more convenient shorthand for accessing some inbuilt functions (the Sub one is used here). I strongly suggest you do the same.

The Parameters block provides me with some configuration options. It allows me, if I want, to create different instances of the stack. In this case, I use an Environment parameter. This particular parameter is common to all my stack files, and I use it to separate test stacks from the prod ones. (I could also do this with AWS sub-accounts)

The Resources block specifies what this file provides. Here I set up the Virtual Private Cloud.

Finally, in the Outputs section, I export the VPC reference out, so that other files can link to it.

AiL – Exploring AWS CloudFormation

I’ve been an active user of AWS for over six years now. The application I work on – a vertical niche app for the mining industry – is hosted there. But as the lead developer – and sole DevOps type – of a small team with an aggressive development schedule, I haven’t ever managed to explore CloudFormation properly, or even organise our servers properly. The most I’d managed to do to keep them organised and under control was to use OpsWorks and Chef to manage configuration and deployment of our servers.

This Christmas period, however, we brought in a number of university students on work experience programs. Most of them went to help the mining engineers on their research programs – but I got one to help me tame our AWS environments. He did a lot of background research on AWS CloudFormation for me, and this week I’ve taken the time to put it to use. I’m going to write a few posts covering some of what I learnt – not just about CloudFormation, but how to configure networks ‘properly’ in AWS – for my own edification and later reference. If that helps some reader as well, I can live with that. 😉


What is CloudFormation?

Put simply, CloudFormation is a way of managing textual descriptions of AWS resources, and a set of tools to take those descriptions and configure AWS resources – such as servers, networks, and load balancers.

Being text-based, you can put the files into source control (very important!). With the tooling, you can do things like:

  • create new ‘stacks’ of resources quickly
  • apply changes to the stacks in a consistent fashion
  • test configuration changes prior to applying them to your production system
  • automatically check for ‘drift’ (changes to your infrastructure that isn’t in the files)
  • automate the above using a build server

You know, all the good sensible adult things you should do with a production system that your livelihood depends on.

What’s the problem I’m solving

First – the app I’m working on isn’t a massive resource hog. It’s an industry-vertical niche web-based engineering app, with less than a thousand active users as a target (we aren’t there yet). It doesn’t need lots of servers, or to cope with typical load patterns. In fact, a lot of the time the app just sits idle, with nobody using it. When they do use it, that use can be fairly sporadic (a few requests per hour) – or they can do large-scale batch jobs that we have to bring up a hundred+ servers to manage. In other words, I’m not looking to solve problems common to consumer-facing sites.

I need to manage approximately 10 servers, each running a slightly different configuration of our app. This includes test environments, a ‘public’ demo site, and then several instances that run the same version but include different run-time plugins based on customer needs. (These plugins model different types of mining equipment, some of which are proprietary to the individual customer). The test environments get updated regularly, while the customer sites get updated roughly quarterly, as feature sets get finished. We don’t need staggered deployments that can be done with no downtime – but we do need to have a controlled process.

We also practice multi-tenancy, at least for customers that don’t have proprietary models. This is done by host-based routing.

What’s my goal here?

Well, besides simply putting as much of my infrastructure into CloudFormation as possible, I also want to improve my network layout to be more secure. So I’m also learning about private & public subnets, load balancers, and the things you need to do to make that all work. This is going to be a series of posts tackling such things as:

I’m not going to get all of my infrastructure into CloudFormation. Some of it you can’t (e.g. SSH keys). Some of it already exists outside of CloudFormation, and I don’t want to destroy it just to recreate it (e.g. my Route53 Hosted Zones). Some of it I just won’t get around to at this time. And some of it I do manage I won’t want to write up. But there’ll be a lot of notes here, and it’ll be in a generally agnostic approach.

I can’t promise that the advice here is “good”. In particular – I’m not a network guy. I know enough to get me in trouble, and some decent rules-of-thumb, but there may be some obvious mistakes here. All I can say is that this is what I’m doing, and it’s hopefully working for me.

I’ll update this page with links to the individual articles as they get written.

Configuring a Multi-Branch Pipeline in Jenkins – Adventures in Learning

In my previous post, I had demonstrated how to configure a Jenkins Server using Docker. The next step is to create a Jenkins job to build some software. Now, we could just do a simple freestyle job, or a basic Maven build – but that will require configuration of Jenkins every time we want to make a new project, and that makes managing the Jenkins Server via Docker more annoying. So, instead, I’m going to use the Cloudbees Bitbucket Branch Source Plugin and create a Bitbucket Team/Project job that will create the rest of my Jenkins jobs automagically for me. A similar plugin exists for GitHub, though I haven’t looked into it.

Continue reading “Configuring a Multi-Branch Pipeline in Jenkins – Adventures in Learning”

Setting up a Jenkins Server with Docker – Adventures in Learning

With the upcoming end-of-life for Bamboo Cloud, I’m in the market for a new build server setup. For this1 experiment, I’m returning to an old favourite – Jenkins – paired with a potential new favourite – Docker. In this post, I describe how I’ve set up a Jenkins server in a Docker container, using the Multibranch Pipeline plugin to automatically configure a simple build2.

Continue reading “Setting up a Jenkins Server with Docker – Adventures in Learning”

Configuring OpsWorks with Chef12 – Don’t forget Security

This is very much in the aide-mémoire category – when setting up an AWS OpsWorks stack with Chef 12, remember to set up the security groups.

Continue reading “Configuring OpsWorks with Chef12 – Don’t forget Security”

Post-Release Testing – what’s that about, anyway?

One of the odder practices of “conventional” software development that I’ve ever come across is the post-release test cycle. It’s something that has baffled me ever since I first saw it; you go through the normal development cycle, including testing, and then you deploy to production – and then you test it again. Sometimes with test cases that weren’t done during development. Why?

Continue reading “Post-Release Testing – what’s that about, anyway?”