Reusing CI config with CircleCI Orbs

Many teams at OVO are using CircleCI for their continuous integration and deployment. The new orbs feature (https://circleci.com/docs/2.0/orb-intro/) of CircleCI allows for packaging of configuration into reusable elements called ‘orbs’, which can be used from any CircleCI configuration file.

CircleCI uses a simple yaml configuration file at a fixed path inside your git repository (.circleci/config.yml). The first key is version which should be set to 2.1 to use orbs.

A basic configuration consists of jobs and workflows:

version: 2.1
jobs:
  test:
    docker:
      - image: python:3
    steps:
      - checkout
      - run: ./test.sh

  deploy:
    docker:
      - image: ovotech/deploy-image:04-11-2018
    steps:
      - checkout
      - run: ./deploy.sh

workflows:
  commit:
    jobs:
      - test
      - deploy:
          requires:
            - test

This contains two jobs: test and deploy. Each job runs in a container, and we specify the image to use with the docker key. CircleCI will run each of the steps for a job inside the container for that job.

There are a number of step types available including checkout and run. checkout will checkout your source code inside the container; run executes a shell command inside the container.

The workflows key defines how the jobs should be run. In this example the test job is run and only if the tests pass is the deploy job run. CircleCI will trigger this workflow every time someone pushes a change to the repository.

Orbs

A CircleCI Orb is a way to package reusable definitions of jobs, executors, and commands which you can import into a CircleCI config. To import an orb you add a top level orbs amp which contains the orbs you want to use (as the value) and the name you refer to the orb by in the rest of the configuration file (as the key).

orbs:
    terraform: ovotech/terraform@1.4.7

In the rest of the config file we can now reference the contents of this orb using the name terraform. The orb is identified by the namespace 'ovotech', the name 'terraform' and the version '1.4.7'.

An orb is a yaml file which is very similar to a CircleCI config file. An orb can contain jobs, executors, and commands.

Executors

An executor is the environment that runs your steps. In many cases this will be the Docker image, environment variables and other Docker settings that are used to create a container.

In the terraform orb we created an executor named 'default' which is a docker image we built containing terraform and associated tools. To define the executor, we use the top level executor key in the orb yaml file:

executors:
  default:
    description: "Terraform executor"
    docker:
    - image: ovotech/terraform:latest

To use an executor defined in a orb, set the executor key for a job in the CircleCI config file:

jobs:
    setup:
      executor: terraform/default
      steps:
      - checkout
      - run: <command>

This job runs the steps in the executor named default from the terraform Orb.

Commands

You can define commands inside an Orb that consist of multiple individual steps. Once imported into your CircleCI config you can then use the command as a single step. This is useful for packaging long commands (or series of commands) so they can be reused, simplifying your CircleCI config.

To enable reuse of commands, they support parameters. Define the parameters under the parameters key of the command, including a type, description. You can make a parameter optional by including a default value.

Within the steps of a command you can substitute the value of a parameter by using << parameters.parameter_name >>. Here's an example:

commands:
  check:
    description: "Check that terraform state hasn't drifted"
    parameters:
      path:
        type: "string"
        description: "Path to the terraform module"
        default: "tf"
    steps:
      - run:
          name: "terraform check << parameters.path >>"
          command: |
            ./check.sh

We can then use this command from a CircleCI config:

version: 2.1

orbs:
   terraform: ovotech/terraform@1.4.7

jobs:
   check_all:
     executor: terraform/default
     steps:
     - checkout
     - terraform/check:
       path: uat
     - terraform/check:
       path: prod

workflows:
  commit:
    jobs:
      - check_all

Jobs

Just like in our config file, an orb can define jobs. Jobs can have parameters just like commands. This example defines the check job which run the check step we already looked at inside the default executor. Inside the orb you can reference other commands and executors by their name.

jobs:
  check:
    executor: default
    description: "Check that terraform state hasn't drifted"
    parameters:
      path:
        type: "string"
        description: "Path to the terraform module"
    steps:
      - checkout
      - check:
          path: << parameters.path >>

We can use a job that's contained inside an orb by including it as part of a workflow:

version: 2.1

orbs:
    terraform: ovotech/terraform@1.4.7

workflows:
  commit:
    jobs:
      - terraform/plan:
        path: prod

This workflow will run the plan job from the terraform orb.

Let's look at how we used these elements to create the ovotech/terraform orb. This is an orb which packages tasks that were being duplicated many times across our repositories, in some cases implemented quite differently.

Terraform Orb

At OVO we aim to treat our infrastructure as code, with Terraform being a popular tool for achieving that aim. When making a change to an infrastructure git repository we follow this workflow:

1. A short lived branch is created and edits made.
2. The branch is pushed to GitHub and a Pull Request is created.
3. CI generates a Terraform plan and attaches the plan to the PR.
4. A developer reviews the plan and the PR approved or rejected.
5. The PR is merged and the approved plan automatically applied to the infrastructure.

This is similar to the process that teams follow for making changes to their application code.

The CircleCI configuration to enable this workflow has been reimplemented multiple times by individual teams, bloating the size of the CircleCI config in each repository. By creating an orb, we can package this functionality and simply import it. This allows improvements to be made centrally.

Here is an example of how simple it is to implement this workflow using the ovotech/terraform orb:

version: 2.1

orbs:
  terraform: ovotech/terraform@1.4.7

workflows:
  test:
    jobs:
      - terraform/plan:
          path: tf
          filters:
            branches:
              ignore: master

      - terraform/apply:
          path: tf
          filters:
            branches:
              only: master

In this example a plan for the module in tf/ is generated and attached to the open PR. If that PR is then merged, the plan is applied.

Full documentation and source code for the ovotech/terraform orb is in the ovotech/circleci-orbs GitHub repo. All orbs in CircleCI are publicly available - these are in the ovotech namespace.