Recently I was facilitating a Value Stream Mapping exercise with one of our clients and they added a process to their enhancements value stream for ‘NFT - Security’ and then added metrics to indicate that this step in the process could take anything from 3 days to 3 months. This was the major factor that caused their cycle time to have a variance of from 19 to 124 days and could add up to $8,000 to the delivery cost of a single enhancement.
And then they said that thing, that thing I’ve heard too many times before:
“Of course, eventually, often they don’t really agree it’s compliant. But they give us something to sign that says we understand the risk and we get to go live.”
CYA. That’s what they are doing. Protecting the individual and not the organisation. At the risk of being a little ranty, that’s simply not doing a job properly. And it’s not very DevOps.
Since the dawn of DevOps we’ve been trying to have the security conversation. Those DevOps die-hards among you will be familiar with John, the CISO character from The Phoenix Project, he who disappears halfway through the story, so frustrated he is with people not listening to him he goes a bit bonkers, hits the bottle and sprouts a lush beard. It’s not just him that’s frustrated though, just before his disappearance, he and Bill (the VP IT Ops) come into conflict during a PCI audit. Bill says:
“Look, I know that sometimes people think you’re not on our side, but I really need your help.”
When John agrees to help, and keep the auditors out of Bill’s way (he just can’t cope with this on top of all the fire-fighting he’s dealing with—familiar?) Bill makes a suggestion that they ship the cardholder data they have offshore in order to reduce access to it. John responds:
“No, no, no. That’s even worse! Remember, we’re not even allowed to transmit it, let alone send it to a third party. Understand? Look, just so we claim plausible deniability, I’m going to pretend I didn’t hear that just now. You’ve got to figure out how to destroy all this prohibited data!”
At the words, “plausible deniability”, Bill’s red mist descends.
It mirrors the CYA behaviour I described that we regularly observe in the organisations we work with. I’d also like to draw your attention to Bill’s assertion that not everyone believes that they are on the same team, with the same goals. One of the first things I ask an organisation when we start having The Security Conversation about DevOps, is where, organisationally, the security people sit. Is there one team? Or an InfoSec team and a Security team? Do they report into the CIO or the CEO? Systems mirror systems (the basis Conway’s Law) and we know that if we have separate teams, processes will be disconnected, handoffs will occur and the waste and blame we’ve already seen in this article exist.
What we often see is what I describe as an ‘ivory tower’ of security: a handful of people, in their own bubble, disconnected from the rest of the technology organisation that either do have, or are perceived to have, some sort of superiority complex. The global cybersecurity skills marketplace is massively under-resourced, which means the cybersecurity people you have are likely to be massively overworked –no wonder they are hard to get hold of and seen as a constraint.
Security often feels like the last bastion of DevOps—I frequently note that security is tackled in the last chapters of the DevOps Handbook, although Chapter 22 is titled: “Information Security as Everyone’s Job, Every Day” which is, of course, what we’ve been saying about quality all along and what is security, if not just another angle of quality?
When I spoke at AppSec EU, Belfast, in 2017 about DevSecOps Engineering, one of my fellow speakers in the DevSecOps track mentioned that he did not like the term ‘DevSecOps’. I was a little surprised, we were speaking on the DevSecOps track after all, and asked him what he saw to be the issues with the term. He said that he thought it was hard enough for people to understand the term ‘DevOps’ without the industry creating another for them to deal with too.
I saw his point. But I sat with it for a while and found my position on this which is, bear with me, it’s a little like feminism. Security is a big problem in all of the organisations we work with. When I do a quick DevOps toolchain gap analysis with clients, it’s more often and not the pre-blessed security libraries and automated security checks that are missing in the pipeline. Nobody I’ve even met has said they have too many security people (but I do frequently find surfeits of project managers, BAs, architects, developers etc). In the world of DevOps, security has been undeniably neglected, and until we treat it as an equal, it needs to have special attention for us to do the work to ensure its proper consideration.
So, now we know all about the problem, let’s have a think about the solution. We have a constraint that is causing waste. How can we remove this constraint?
This constraint is mainly about not enough people having the right knowledge. We assert the 80:20 rule is at play here; that is, 80% of security considerations related to 20% of the knowledge so, recognizing that knowledge is in people’s heads, we need to find a way to get it out of there, and share it. These are the most effective ways we have observed to do this:
Embed security resource in feature teams/squads for short (weeks) periods of time I know! You don’t have hardly any resources already, and what you do have is overworked resources. But that knowledge rubs off fast when people sit and work together day to day. Get that 20% of knowledge shared with more people and it will proliferate. Many improvements require an initial investment or time to reap the long-term benefits - this is no different.
Extract and share that knowledge Write it down—write non-functional requirements into user stories in Jira, tell stories and document best practice into Confluence, have security people talk at your Communities of Practice, have a cybersecurity Community of practice, use ChatOps for problem and incident management and to track progress of your continuous delivery pipeline and keep the records in your collaboration systems too. Have a cybersecurity hackathon or a dojo with a dedicated security expert, use a cybersecurity simulation or invest in formal DevSecOps training.
Automate it Automate the knowledge and automate the tasks. Use tools that tell your developers what vulnerable code and artifacts they are using or proposing to use and what to use instead so they can get on with building value. Automate the processes of securing your systems and machines so your operations people can get on with building reliability and resilience into your critical infrastructure.
One day, I hope, DevSecOps will die, but only because it’s alive, everywhere and it’s part of what we all do, every day.