Security, yes we're the people that in many organisations are sat in a corner and say no, to whatever it is you want to do. When I joined OVO, I wanted this to be different, I wanted us to be the people that really want to say yes, unless we really have to say no. When I mean really have to say no we’re talking about making an uninformed decision that puts people at risk
Have we got it right?
Hopefully so, but along the way we've definitely made some mistakes but at the same time we've also had a series of successes.
So back at the beginning of the journey we made a decision to base how we do security on the coaching model. In the same way that you have agile coaches, we would have a security coach, this was me. Initially this worked pretty well. I could help engineers understand what improvements could be made to services, infrastructure or challenges in order to improve the their security posture. Some of this was done through briefings, attending stand-ups and conducting security reviews of their products to understand how and why something was implemented the way it had been. All the time I was very conscious that I didn't want to dictate how security should be improved, instead I was keen that we understood why we were making the decisions we were, and whether they were the most suitable when it comes to security. My thinking was that we try to improve the culture, make security more of an embedded part of it.
Moving forward a couple of months it was becoming increasingly hard to provide the coaching service to engineers while at the same time, being approached by the wider business community to help with security challenges, whether this was risk, compliance, general security awareness, data protection, or even business continuity. After a few conversations we decided it was about time we built out more of a security function…. So we did just that.
We brought in an external candidate who understands compliance, was keen to see how we include security compliance requirements in our business. We had another internal candidate transfer in to the team who would run with business continuity, and another internal candidate who was a bit of a hobbyist hacker. We needed one more, so we looked for an external candidate who was a software engineer at heart and with a passion for security.
Now we have a team.....
Ok, so next steps. We agreed we wanted to continue with our coaching approach and so looked at how we would do this. Security is not for the security team to own and do on our own, it's for us to help the engineers make informed decisions. However, it became apparent that we needed to somehow measure success.
Success or performance metrics usually only mean one thing... dashboards. So yep, we built a dashboard. The first dashboard we built combined some of the CIS Benchmarks with our ability to carry out automated checks on the accounts in our cloud environments. Since this went in, we've had additional input from our Production Engineering team on how code could be written better and what additional checks we should include. The alerts it shows, are backed up with links to Wiki articles on what the offending item is i.e. why it shows as a fail. From there an engineer can follow links to understand what they need to do to correct it.
Next we started threat modelling. Our security engineers and our business continuity specialist work with engineering teams using threat modelling sessions to work out where there are potential weaknesses with products and services. Using these sessions, engineering teams can then look at what changes need to be made, and how these could be addressed. In turn we can measure how many teams we’ve worked with, the products and services they and in turn we get to understand the criticality of them. This then helps us work with them, to provide guidance on what they should be considering to improve the security.
We've also spent some time looking at what additional SaaS based tools can be used to complement our dashboard and the threat modelling sessions. We've implemented some tools that look for vulnerabilities of environments, accounts, hosts and services. We've also started looking at how we could check dependencies on code/software to see where there may be security concerns. The results from these tools will help engineers make better decisions about what is an acceptable risk and what work needs to be added to their backlogs. Basically if we can create or find a tool that makes their lives easier, that is what we will do.
Finally, we still believe that talking to people is the best way to improve security. We still provide briefings on important subjects, tech talks about technologies we want engineers to think about implementing, threat modelling sessions, and our monthly DevSecOps meeting.
On that last point, DevSecOps, this is something we feel is important to the culture here at OVO. We use these meetings to provide updates to engineers who are interested in security, and we ask them to contribute in turn for the benefit of everyone else. This could be as simple as providing a review of a recent breach, and how it occurred. In other cases it could be a talk about what they have been working on and why other engineering teams should implement these solutions.
I feel a bit like Chief Brody in Jaws…”We’re gonna need a bigger boat” as I go back to senior management and ask for more resource to enable us to cover more engineering teams and and allow security to work at the speed of DevOps to prevent us from becoming a blocker..