AiL – Subnets! Public Subnets! Wow, they are free, and we can au-to-mate!

(to the tune of the Ewoks Victory Song. Now it can be stuck in your head, and not just mine)

Picking up from last time, I needed to start creating things inside my Virtual Private Cloud, or VPC. The first things to create are subnets – public subnets, in particular. Without a public subnet, nothing that I run in the VPC can be accessed from the internet – nor can they access the internet in turn.

Continue reading “AiL – Subnets! Public Subnets! Wow, they are free, and we can au-to-mate!”

AiL – Baby’s First CloudFormation Stack

As previously discussed, I’m doing a small series of posts around bringing the AWS infrastructure that I use into the current era, and putting it all into CloudFormation. In this post, I’m going to cover setting up the first stack. This is going to set up a Virtual Private Cloud (or VPC), which is where the rest of the stuff I make later will sit.


What’s a Virtual Private Cloud?

A VPC is a virtual network of virtual servers. It’s your own mini-slice of the AWS cloud, and the machines within the VPC are aware of each other – in fact, they are on their own subnet (or subnets).

Why use a VPC?

You don’t have to set up a VPC to use AWS. You can simply create servers. That’s what I’ve been doing up until now. It’s just that it’s a bit limited.

I want to use a VPC for two big reasons:

  • I want to be able to use more recent/powerful/cheaper machine images, with OpsWorks. They’re only available if I also use a VPC.
  • I want to use an Elastic Load Balancer, in part to manage HTTPS certificates and connections. This requires a VPC and subnets.

Setting up the Stack

Here’s my config, at this particular stage:

I put the VPC into its own file, because I don’t want to delete it when tearing down an environment for the sake of testing. There’s lots of things that get annoying to re-create if the VPC is changed (cough OpsWorks stacks) – so we put the VPC in its own file. (Later, when I bring in nested stacks, this will not be in the nested stack).

Break It Down

First, note that I use YAML for my CloudFormation files. I’m not a huge fan of YAML in general, but the JSON option doesn’t allow you to use comments, and comments are essential. (Sidebar: when parsing JSON, always enable comments. It’s non-standard, but it’s useful). Using YAML also lets me use a more convenient shorthand for accessing some inbuilt functions (the Sub one is used here). I strongly suggest you do the same.

The Parameters block provides me with some configuration options. It allows me, if I want, to create different instances of the stack. In this case, I use an Environment parameter. This particular parameter is common to all my stack files, and I use it to separate test stacks from the prod ones. (I could also do this with AWS sub-accounts)

The Resources block specifies what this file provides. Here I set up the Virtual Private Cloud.

Finally, in the Outputs section, I export the VPC reference out, so that other files can link to it.

AiL – Exploring AWS CloudFormation

I’ve been an active user of AWS for over six years now. The application I work on – a vertical niche app for the mining industry – is hosted there. But as the lead developer – and sole DevOps type – of a small team with an aggressive development schedule, I haven’t ever managed to explore CloudFormation properly, or even organise our servers properly. The most I’d managed to do to keep them organised and under control was to use OpsWorks and Chef to manage configuration and deployment of our servers.

This Christmas period, however, we brought in a number of university students on work experience programs. Most of them went to help the mining engineers on their research programs – but I got one to help me tame our AWS environments. He did a lot of background research on AWS CloudFormation for me, and this week I’ve taken the time to put it to use. I’m going to write a few posts covering some of what I learnt – not just about CloudFormation, but how to configure networks ‘properly’ in AWS – for my own edification and later reference. If that helps some reader as well, I can live with that. 😉


What is CloudFormation?

Put simply, CloudFormation is a way of managing textual descriptions of AWS resources, and a set of tools to take those descriptions and configure AWS resources – such as servers, networks, and load balancers.

Being text-based, you can put the files into source control (very important!). With the tooling, you can do things like:

  • create new ‘stacks’ of resources quickly
  • apply changes to the stacks in a consistent fashion
  • test configuration changes prior to applying them to your production system
  • automatically check for ‘drift’ (changes to your infrastructure that isn’t in the files)
  • automate the above using a build server

You know, all the good sensible adult things you should do with a production system that your livelihood depends on.

What’s the problem I’m solving

First – the app I’m working on isn’t a massive resource hog. It’s an industry-vertical niche web-based engineering app, with less than a thousand active users as a target (we aren’t there yet). It doesn’t need lots of servers, or to cope with typical load patterns. In fact, a lot of the time the app just sits idle, with nobody using it. When they do use it, that use can be fairly sporadic (a few requests per hour) – or they can do large-scale batch jobs that we have to bring up a hundred+ servers to manage. In other words, I’m not looking to solve problems common to consumer-facing sites.

I need to manage approximately 10 servers, each running a slightly different configuration of our app. This includes test environments, a ‘public’ demo site, and then several instances that run the same version but include different run-time plugins based on customer needs. (These plugins model different types of mining equipment, some of which are proprietary to the individual customer). The test environments get updated regularly, while the customer sites get updated roughly quarterly, as feature sets get finished. We don’t need staggered deployments that can be done with no downtime – but we do need to have a controlled process.

We also practice multi-tenancy, at least for customers that don’t have proprietary models. This is done by host-based routing.

What’s my goal here?

Well, besides simply putting as much of my infrastructure into CloudFormation as possible, I also want to improve my network layout to be more secure. So I’m also learning about private & public subnets, load balancers, and the things you need to do to make that all work. This is going to be a series of posts tackling such things as:

I’m not going to get all of my infrastructure into CloudFormation. Some of it you can’t (e.g. SSH keys). Some of it already exists outside of CloudFormation, and I don’t want to destroy it just to recreate it (e.g. my Route53 Hosted Zones). Some of it I just won’t get around to at this time. And some of it I do manage I won’t want to write up. But there’ll be a lot of notes here, and it’ll be in a generally agnostic approach.

I can’t promise that the advice here is “good”. In particular – I’m not a network guy. I know enough to get me in trouble, and some decent rules-of-thumb, but there may be some obvious mistakes here. All I can say is that this is what I’m doing, and it’s hopefully working for me.

I’ll update this page with links to the individual articles as they get written.

AWS Lambda finally supports Ruby, shared libraries

So here’s my favourite bit of news out of Amazon for today – they’ve finally got Ruby support in AWS Lambda, AND they’ve got support for shared libraries.


I’ve got a metric tonne of scripts for dealing with AWS environments in Ruby that I’ve written over the last six years of working on IES, mostly to organise data and application deployments. I’ve been wanting to put a web interface over them – to make it easier for other people in my team to use them – for some time, and even to make an iOS app for it, but I also didn’t want to run a dedicated backend server for a back-office app that I would use maybe a couple of times a week.

AWS Lambda is the perfect solution for building a backend for backoffice-style apps. But the thought of rebuilding these scripts as Node or Python was too much. Looking forward to trying this out!

Configuring OpsWorks with Chef12 – Don’t forget Security

This is very much in the aide-m̩moire category Рwhen setting up an AWS OpsWorks stack with Chef 12, remember to set up the security groups.

Continue reading “Configuring OpsWorks with Chef12 – Don’t forget Security”

AWS SDK 2.2, iOS 9, Xcode 7 – Adventures in Learning

Well, it’s been three-and-a-half years, but I’ve finally got around to getting to a point of writing an iOS app. I wouldn’t hold your breath waiting to get a copy, though – it’s purely for my private use, to aid in monitoring and administering the IES project.

After doing enough tutorials and similar exercises to be comfortable in building the app and the UI, I got around to trying calls to the AWS infrastructure. This proved a bit more difficult than I anticipated – hence this aide-mémoire. This isn’t going to be useful for non-iOS developers, and I doubt it’s going to have anything new for more seasoned iOS developer; only iOS noobs like me need bother.

Continue reading “AWS SDK 2.2, iOS 9, Xcode 7 – Adventures in Learning”

How-To: Grails, GORM and SimpleDB

I went to build a new Grails-based app today, and I wanted to use SimpleDB as a backend (the app is an internal-use administration app, to configure a suite of AWS-deployed apps). So I went looking on how to use GORM with SimpleDB. This turned out to be a non-trivial task, so I thought I’d share the process with everyone.

Continue reading “How-To: Grails, GORM and SimpleDB”