(to the tune of the Ewoks Victory Song. Now it can be stuck in your head, and not just mine)
Picking up from last time, I needed to start creating things inside my Virtual Private Cloud, or VPC. The first things to create are subnets – public subnets, in particular. Without a public subnet, nothing that I run in the VPC can be accessed from the internet – nor can they access the internet in turn.
Continue reading “AiL – Subnets! Public Subnets! Wow, they are free, and we can au-to-mate!”
As previously discussed, I’m doing a small series of posts around bringing the AWS infrastructure that I use into the current era, and putting it all into CloudFormation. In this post, I’m going to cover setting up the first stack. This is going to set up a Virtual Private Cloud (or VPC), which is where the rest of the stuff I make later will sit.
A VPC is a virtual network of virtual servers. It’s your own mini-slice of the AWS cloud, and the machines within the VPC are aware of each other – in fact, they are on their own subnet (or subnets).
You don’t have to set up a VPC to use AWS. You can simply create servers. That’s what I’ve been doing up until now. It’s just that it’s a bit limited.
I want to use a VPC for two big reasons:
Here’s my config, at this particular stage:
| --- | |
| AWSTemplateFormatVersion: '2010-09-09' | |
| Description: | |
| Global configuration that could be used by multiple related stacks | |
| # Metadata: # no metadata | |
| Parameters: | |
| Environment: | |
| Type: String | |
| Description: | |
| Stack Environment Prefix. | |
| Resources: | |
| # We need at least one resource. The VPC is the logical one to include here. | |
| VPC: | |
| Type: AWS::EC2::VPC | |
| Properties: | |
| # The range of ip addresses used in the VPC. Subnets will be inside this. | |
| # Using the /16 lets me easily have subnets that don't overlap without | |
| # needing to remember bit masks. | |
| CidrBlock: 10.0.0.0/16 # 10.0.0.0 -> 10.0.255.255 | |
| EnableDnsSupport: true # If false, the servers don't seem to get access to DNS at all. | |
| InstanceTenancy: default | |
| Tags: | |
| - Key: Name | |
| Value: !Sub "${Environment} VPC" | |
| Outputs: | |
| VPC: | |
| Description: The ID for the Virtual Private Cloud; needed by more or less everything. | |
| Value: !Ref VPC | |
| Export: | |
| Name: !Sub "${Environment}::VPC" |
I put the VPC into its own file, because I don’t want to delete it when tearing down an environment for the sake of testing. There’s lots of things that get annoying to re-create if the VPC is changed (cough OpsWorks stacks) – so we put the VPC in its own file. (Later, when I bring in nested stacks, this will not be in the nested stack).
First, note that I use YAML for my CloudFormation files. I’m not a huge fan of YAML in general, but the JSON option doesn’t allow you to use comments, and comments are essential. (Sidebar: when parsing JSON, always enable comments. It’s non-standard, but it’s useful). Using YAML also lets me use a more convenient shorthand for accessing some inbuilt functions (the Sub one is used here). I strongly suggest you do the same.
The Parameters block provides me with some configuration options. It allows me, if I want, to create different instances of the stack. In this case, I use an Environment parameter. This particular parameter is common to all my stack files, and I use it to separate test stacks from the prod ones. (I could also do this with AWS sub-accounts)
The Resources block specifies what this file provides. Here I set up the Virtual Private Cloud.
Finally, in the Outputs section, I export the VPC reference out, so that other files can link to it.
I’ve been an active user of AWS for over six years now. The application I work on – a vertical niche app for the mining industry – is hosted there. But as the lead developer – and sole DevOps type – of a small team with an aggressive development schedule, I haven’t ever managed to explore CloudFormation properly, or even organise our servers properly. The most I’d managed to do to keep them organised and under control was to use OpsWorks and Chef to manage configuration and deployment of our servers.
This Christmas period, however, we brought in a number of university students on work experience programs. Most of them went to help the mining engineers on their research programs – but I got one to help me tame our AWS environments. He did a lot of background research on AWS CloudFormation for me, and this week I’ve taken the time to put it to use. I’m going to write a few posts covering some of what I learnt – not just about CloudFormation, but how to configure networks ‘properly’ in AWS – for my own edification and later reference. If that helps some reader as well, I can live with that. 😉
Put simply, CloudFormation is a way of managing textual descriptions of AWS resources, and a set of tools to take those descriptions and configure AWS resources – such as servers, networks, and load balancers.
Being text-based, you can put the files into source control (very important!). With the tooling, you can do things like:
You know, all the good sensible adult things you should do with a production system that your livelihood depends on.
First – the app I’m working on isn’t a massive resource hog. It’s an industry-vertical niche web-based engineering app, with less than a thousand active users as a target (we aren’t there yet). It doesn’t need lots of servers, or to cope with typical load patterns. In fact, a lot of the time the app just sits idle, with nobody using it. When they do use it, that use can be fairly sporadic (a few requests per hour) – or they can do large-scale batch jobs that we have to bring up a hundred+ servers to manage. In other words, I’m not looking to solve problems common to consumer-facing sites.
I need to manage approximately 10 servers, each running a slightly different configuration of our app. This includes test environments, a ‘public’ demo site, and then several instances that run the same version but include different run-time plugins based on customer needs. (These plugins model different types of mining equipment, some of which are proprietary to the individual customer). The test environments get updated regularly, while the customer sites get updated roughly quarterly, as feature sets get finished. We don’t need staggered deployments that can be done with no downtime – but we do need to have a controlled process.
We also practice multi-tenancy, at least for customers that don’t have proprietary models. This is done by host-based routing.
Well, besides simply putting as much of my infrastructure into CloudFormation as possible, I also want to improve my network layout to be more secure. So I’m also learning about private & public subnets, load balancers, and the things you need to do to make that all work. This is going to be a series of posts tackling such things as:
I’m not going to get all of my infrastructure into CloudFormation. Some of it you can’t (e.g. SSH keys). Some of it already exists outside of CloudFormation, and I don’t want to destroy it just to recreate it (e.g. my Route53 Hosted Zones). Some of it I just won’t get around to at this time. And some of it I do manage I won’t want to write up. But there’ll be a lot of notes here, and it’ll be in a generally agnostic approach.
I can’t promise that the advice here is “good”. In particular – I’m not a network guy. I know enough to get me in trouble, and some decent rules-of-thumb, but there may be some obvious mistakes here. All I can say is that this is what I’m doing, and it’s hopefully working for me.
I’ll update this page with links to the individual articles as they get written.
So here’s my favourite bit of news out of Amazon for today – they’ve finally got Ruby support in AWS Lambda, AND they’ve got support for shared libraries.
I’ve got a metric tonne of scripts for dealing with AWS environments in Ruby that I’ve written over the last six years of working on IES, mostly to organise data and application deployments. I’ve been wanting to put a web interface over them – to make it easier for other people in my team to use them – for some time, and even to make an iOS app for it, but I also didn’t want to run a dedicated backend server for a back-office app that I would use maybe a couple of times a week.
AWS Lambda is the perfect solution for building a backend for backoffice-style apps. But the thought of rebuilding these scripts as Node or Python was too much. Looking forward to trying this out!
It was about 19 years ago now that a colleague of mine lent me a copy of a little white book. That book changed the way I looked at programming more than any other book (though The Pragmatic Programmer gave it a run for its money). One of the things, in particular, was Test Driven Development, or TDD.
Continue reading “What TDD means to me…”
So for my first #BlogADayMay, I’m doing a rant I’ve had saved up for a while. Continuous Deployment, which is one of the new shiny hawtness going around, isn’t always the right thing.
Continue reading “Continuous Deployment isn’t always the right thing…”
UUIDs make great identifiers – ones that are, for most practical purposes, unique, easy to generate, and hard to guess. The only problem is that they are long – 256 bits, but with a textual representation that’s 36 characters. So what if they were shorter?
Continue reading “Using shorter UUIDs”
I’ve been meaning to write this for nearly a year, but I held off hoping things would change with the next release. They didn’t, so I’m writing this: the Clipboard plugin for the Ext.grid.Panel class – which provides cut-and-paste support for the enhanced table widget – is borked by design. It does stupid things, and Sencha says it should do the stupid things. In this post I share what these things are, and how I’ve overriden the default behaviour to do something hopefully less stupid. Warning: this is a rant.
Continue reading “ExtJS’s Grid Clipboard plugin is borked by design”
Sencha recently announced the general availability of ExtJS 6.2. People who regularly read my posts (hi, to all three of you. 😉 will be aware that I’ve been working with the ExtJS library for a number of years now, while building a suite of apps of some not inconsiderable size1. It’s probably not the largest ExtJS app in the world, but it’s up there. And when you work on an application for a number of years, you’ll need to upgrade its framework from time to time.
Every time I’ve done this, the process has been a bit different. This time, it’s different in a good way – it was easy.
Continue reading “Upgrading to ExtJS 6.2”
Using Sencha Ext.Config – some nice tips, especially around automatic events for config changes.
Software is too expensive to build cheaply.... 