At OVO we have many cloud projects which need connectivity to services in our datacenter. We have VPNs setup to securely route connections between projects and on-prem services.

One issue with this configuration is name resolution within AWS VPCs. The DHCP options for a VPC has a single set of DNS servers that are provided to all instances in the VPC. By default, these are the AWS Route53 Resolver servers.

This works well, except for private hostnames that need to be looked up using the internal DNS servers. One solution would be to set the DNS servers for the VPC to be the internal DNS servers, but this would cause all DNS requests to go over the VPN. This creates a dependency on the VPN to be up for all name resolution, even for names that can be publicly resolved.

Even services that don't need on-prem connectivity depend on the VPN just for resolving public names, which is not a good situation to be in if the VPN goes down.

Route53 Resolver Rules

In November 2018, AWS announced some new features for Route53 Resolver that allow us to solve this problem - Resolver Endpoints and Rules.

The AWS Route53 Resolver service can be configured with rules for forwarding name lookups to upstream servers. By creating a rule for our internal domain (ovoenergy.local) we can set our VPC to use the Route53 Resolver servers, but still forward to our internal DNS servers to lookup internal names.

To allow Route53 Resolver to forward requests to the internal servers we need to create an outbound endpoint in a subnet that has VPN connectivity to the internal DNS servers.

We use terraform heavily at ovo, but unfortunately the Route53 Resolver is not yet supported (but keep an eye on this PR). But we can create these resource using cloudformation, which can be managed from terraform.

The cloudformation template looks like:

AWSTemplateFormatVersion: "2010-09-09"
Description: Ovo DNS Resolver

Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
  SubnetA:
    Type: AWS::EC2::Subnet::Id
  SubnetB:
    Type: AWS::EC2::Subnet::Id
  SubnetC:
    Type: AWS::EC2::Subnet::Id
  DnsServer1:
    Type: String
  DnsServer2:
    Type: String

Resources:
  OutboundEndpoint:
    Type: "AWS::Route53Resolver::ResolverEndpoint"
    Properties:
      Direction: OUTBOUND
      IpAddresses:
        - SubnetId: !Ref SubnetA
        - SubnetId: !Ref SubnetB
        - SubnetId: !Ref SubnetC
      Name: ovoenergy-local
      SecurityGroupIds:
        - !GetAtt SecurityGroup.GroupId
      Tags:
        - Key: Name
          Value: OutboundEndpoint

  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: InternalDNS
      GroupDescription: Allow access to internal DNS servers
      SecurityGroupEgress:
        - CidrIp: !Join ['', [!Ref DnsServer1, "/32"]]
          FromPort: 53
          IpProtocol: tcp
          ToPort: 53
        - CidrIp: !Join ['', [!Ref DnsServer2, "/32"]]
          FromPort: 53
          IpProtocol: tcp
          ToPort: 53
        - CidrIp: !Join ['', [!Ref DnsServer1, "/32"]]
          FromPort: 53
          IpProtocol: udp
          ToPort: 53
        - CidrIp: !Join ['', [!Ref DnsServer2, "/32"]]
          FromPort: 53
          IpProtocol: udp
          ToPort: 53
      Tags:
        - Key: Name
          Value: InternalDNS
      VpcId: !Ref VpcId

  OvoEnergy:
    Type: "AWS::Route53Resolver::ResolverRule"
    Properties:
      DomainName: ovoenergy.local
      Name: ovoenergy
      ResolverEndpointId: !GetAtt OutboundEndpoint.ResolverEndpointId
      RuleType: FORWARD
      Tags:
        - Key: Name
          Value: ovoenergy
      TargetIps:
        - Ip: !Ref DnsServer1
          Port: "53"
        - Ip: !Ref DnsServer2
          Port: "53"

  OvoEnergyAssociation:
    Type: "AWS::Route53Resolver::ResolverRuleAssociation"
    Properties:
      Name: ovoenergy
      ResolverRuleId: !GetAtt OvoEnergy.ResolverRuleId
      VPCId: !Ref VpcId

And the terraform that drives it:

# Terraform doesn't yet support route53 resolver endpoints or rules
# Use cloudformation for now
resource "aws_cloudformation_stack" "resolver" {
  name = "dns-resolver"

  parameters = {
    SubnetA       = "${module.zone_a.vpn-subnet-id}"
    SubnetB       = "${module.zone_b.vpn-subnet-id}"
    SubnetC       = "${module.zone_c.vpn-subnet-id}"
    VpcId         = "${aws_vpc.vpc.id}"
    DnsServer1    = "10.36.111.10"
    DnsServer2    = "10.60.111.42"
  }

  template_body = "${file("${path.module}/resolver.yaml")}"
}