At OVO Energy, we have a central security engineering team which supports application development and DevOps teams across the business.
We recently open-sourced a new tool to prevent subdomain takeovers: Domain Protect, available at https://github.com/domain-protect/domain-protect . In this blog, I’ll go through the basics of domain takeover, talk about how the requirement arose at OVO, and describe the tool’s features, findings, and planned next steps.
And you’ll see how we saved many thousands of US Dollars in fees to Bug Bounty researchers.
What is subdomain takeover?
Let’s look at a simple example of an Amazon S3 bucket hosting a static web site using HTTP.
This can be set up by creating a S3 bucket and a Canonical Name (CNAME) entry in Route53, Amazon’s DNS service. When a user browses to www.example.com, their device queries DNS and they are then directed to the S3 bucket.
But what happens a few months or even years later? A developer decides the web site is no longer required, so deletes the S3 bucket. However, she forgets to remove the DNS record.
Then, a malicious person sets up an S3 bucket with exactly the same name, but in his own AWS account. He now has complete control over the domain, can create his own content or application which may appear genuine, grab credentials and personal information, and defraud users who visit the web site.
This is just one example of a subdomain takeover – there are many different types. They all occur when a developer or engineer removes a cloud resource, but forgets to delete the corresponding DNS record. All of them can be damaging to an organisation and its customers.
Why did this become important for OVO Energy?
At OVO we have a complex hybrid cloud environment, with multiple autonomous development teams who manage their own cloud accounts. We’re always looking at how we can improve centralised visibility of vulnerabilities in cloud assets to better protect our customers, so a few months ago, we started a private Bug Bounty program. The security researchers found a number of issues, over half of which were subdomain takeovers.
We wanted to get ahead of the malicious actors and Bug Bounty researchers, find any vulnerable subdomains ourselves and fix them – bearing in mind we have a big advantage: access to our DNS records.
We didn’t just want to find the vulnerabilities at a single point in time – we also felt it was important to quickly find new security issues as and when they arose in the future.
We began by searching for open-source tools, however although we found code used by Bug Bounty researchers, we didn’t find any defensive tools which could run on a continuous basis.
We decided to develop our own.
Domain Protect
Domain Protect uses serverless functions in the cloud to detect subdomain takeover vulnerabilities, and alert security and engineering teams. Domain Protect supports Amazon Web Services (AWS) and is installed to a security audit account within an AWS Organization:
A number of Lambda functions are installed, each running at regular intervals triggered by a CloudWatch scheduled event. The Lambda functions look for different types of domain takeover vulnerabilities, and then write their findings to a Simple Notification Service (SNS) topic.
Another Lambda function is triggered by new events arising on the SNS topic and sends an alert to Slack. Optionally, a security administrator can subscribe her email to the SNS topic as well.
The Lambda functions are short pieces of Python code, which scan across the entire AWS Organization:
The Domain Protect lambda functions begin by assuming a role into the AWS Organization primary account, to list all the accounts within the Organization. Then, it assumes a role to a target account, queries Route 53 records and detects domains vulnerable to takeover.
Domain Protect repeats for all accounts in the Organization, and finishes by writing the results to the SNS topic.
Once the scans have completed, Domain Protect alerts the security team by Slack:
Domain Takeover Vulnerabilities we look for
Domain Protect has been written in a modular way, making it straightforward to add code for detection of new vulnerability types. Currently, it scans Amazon Route53 to search for:
- Alias and CNAME records to identify CloudFront distributions with missing S3 origin
- ElasticBeanstalk Alias and CNAME records vulnerable to takeover
- Name Server (NS) records for subdomain delegations vulnerable to takeover
- S3 Alias and CNAME records vulnerable to takeover
- Vulnerable CNAME records for Azure resources
- Vulnerable CNAME records for Google Cloud resources
Security features of Domain Protect
As Domain Protect is a security tool, we felt it was important to lead by example and ensure that it can be implemented in a secure way. Security features include:
- Encryption of Lambda variables and SNS topic using Key Management Service (KMS)
- Least privilege Identity and Access Management (IAM) roles for Domain Protect audit and deployment
- External ID option in role to be assumed across all AWS accounts
- Code in GitHub doesn’t include secrets or sensitive information
We’ve also integrated the repository with Snyk to scan for vulnerabilities and license issues in dependencies:
How we deployed the infrastructure
We took an infrastructure-as-code approach using Terraform, and deployed to our security audit account in AWS with CircleCI:
This allows us to efficiently test and deploy code changes to our development and production environments.
What we found
Domain Protect identified a significant number of vulnerable domains across the various application development and DevOps teams at OVO – all of which are now fixed, preventing hostile takeover. A typical Bug Bounty payment for domain takeover is between $500 and $2,000 depending on severity, so we also saved a substantial amount of money.
Just as important, we now have confidence that if these types of domain takeover vulnerabilities occur in the future, we’ll quickly be alerted and can take prompt action to remediate.
What’s next?
Across OVO, not all of our DNS records are implemented in Amazon Route53. So we’re currently extending the functionality of Domain Protect to DNS records held in Google Cloud Platform (GCP).
And as we become aware of new types of subdomain takeover which may be present across OVO teams, and are feasible to detect, we’ll add these checks as well.
Update
24 January 2022: We've now added GCP support and automated takeover to Domain Protect - see my latest blog post OVO vs Bug Bounty researchers - round 2