SRE Meetup 5th December 2018
I gave a talk at the last Bristol SRE meetup earlier this month which this time was held at OVO's Bristol office. The theme I presented on was the importance of simplicity. In particular I wanted to look closely at deployment tooling that is required for Infrastructure as Code (IaC). This blog post is a recap of the talk's main points.
The problem
No matter how good your IaC is, if your tooling is awkward to use people will take shortcuts, particularly when they're in a rush. Shortcuts mean ignoring the automation and applying changes manually, damaging the accuracy of your code. The less accurate your IaC the harder it will be to use it and the associated tooling in future as there will be even more unexpected errors and inconsistencies.
The options
The two responses to this that I discussed are enforce and encourage. While both are valid they are not equal.
-
Enforce - In this context enforcement is the idea of removing permissions so that users cannot take shortcuts and all changes have to go through the proper channels. This is the more common approach that teams tend to take but it's not what I'd recommend. Ultimately, taking shortcuts around an official process is not normally a willful attempt to avoid the rules. It's a sign that the process is tough to use or gets in the way when people are trying to get work done. Forcing your teams to use these flawed tools will make them less productive and will breed apathy.
-
Encourage - Instead it's much better to encourage. By this I mean that you should make sure that the official way is also the easiest way, build automation and tools that are seamless and powerful that your users actually want to use. This will make for happier and more productive teams who feel empowered to do their job.
This is not to say that you should give everybody full admin access. Reducing permissions down to what is actually needed is a sensible and important step but it should be completely invisible to your developers in their day to day role. This means it should be done after you have your automation fit for purpose.
The demos
To demo an example of this focus on simplicity, I first presented a typical non-automated Terraform lifecycle that looks something like:
- Make a change,
- (Optionally) Run a plan,
- Create a pull request
- The change gets approved and merged,
- Checkout master locally & pull down latest changes,
- Run a
terraform apply, - Verify expected changes have happened.
This lifecycle is normally fine when you have one or two users but as you scale up and want your whole team to be able to interact with infrastructure there's an awful lot of assumed knowledge, permissions, and specific configuration every person will need.
As an alternative I then showed off a workflow several teams here at OVO use to run Terraform using CircleCI. Where changes can be applied to live infrastructure entirely through Git. This logic is available as an open source orb for which full credit goes to my colleague Dan Flook who implemented the first example of this method at OVO, formalised it as an orb and has written a post with more in depth information. But as a summary the workflow looks like:
- Make a change,
- Create a pull request,
- CircleCI automatically runs a
terraform planand posts the output to the PR for approval, - The change gets approved and merged,
- CircleCI automatically applies the change.
The code for the demo is available here https://github.com/mikebrooks-ovo/sre-meetup-demo and you can see how easy it is to get it working, in fact it's just 18 lines of yaml.
Summary
Although the more obvious takeaway is the cool CircleCI orb and Terraform workflow, which tool you use is not the most important factor. Certainly some tools will make simplicity easier to achieve than others but all have trade offs and no tool is perfect for every team. The key point really is that when designing deployments for your team make sure they are automated and easy. If something is not working out, look at how you can fix it, don’t just close off alternative avenues.
Meetups are a fantastic way of sharing ideas and seeing how other teams are working, plus there's free beer and pizza. If you're interested in the SRE world everybody is welcome to register and attend the Bristol SRE group. Thanks to Luke Briscoe for organising!