Well it has been more than a year, well past time for a new post!
This is a story about two approaches to creating a Software Delivery Lifecycle, or SDLC. The goal of both SDLCs is obviously to get code into production, and to ensure that certain checks are passed before the code can go into production. The difference in the two SDLCs is in how each of those tasks are approached.
Audit-Driven
In the first SDLC, the approach is to look at all the audit and compliance checks first, and see how each one can be checked off. We’ll call this approach the “audit-driven” SDLC. These checks are all quite reasonable; we need to make sure that new code passes unit tests, is checked for common security holes (like the OWASP top 10), that QA tests the code, the regression tests pass, etc.
Taking one check as an example, let’s investigate what it looks like to ensure that all new code is reviewed (and approved) before it goes into production. We obviously want to make sure not just any junior dev can mark approval, so a list of accepted approvers is generated for each repository. Pull Request checks are added to require approval before merging. The audit task is done.
Prod-Driven
In the second SDLC, the approach is to first look at how to quickly get a change into production. We’ll call this approach the “prod-driven” SDLC. Yes, we know we need to check off the same boxes as with the other approach, but we know that we only get value from deployed and running code, so we place as top priority getting code into production. Pipelines are built and tested, test environments are created, infrastructure is codified etc.
Suddenly, the developers, and everybody else involved, are truly more involved in this process. They know that the code they write is going to production today, or maybe tomorrow if things go slow. They become more driven to make sure the code is actually functioning, because it now becomes super easy to point out who broke something. So many of the checks that we need to check off get built-in without even realizing it. The developers know they can make mistakes, so they want somebody else to check over their code. Because they just worked on this, they know who else would be best to also take a look at that. The checkbox for peer-reviewed code is done.
Reviews
Now, let’s keep running with the peer-review example.
In the audit-driven approach, every change now goes out to a group of folks to be reviewed. Because of group mentality, everybody in the group assumes somebody else will look at this, and ignore it. They will even codify this by adding email filters to send all these emails into a folder they will never open. Well, maybe if they really care about the code they will go into that folder once a day and look at some of the changes. A PR will sit for a couple of days, on average, before even being looked at. Some higher priority items perhaps get faster review, because managers are bugging people to add approvals (or even worse, the managers themselves are adding approvals).
When issues are found, they go back to the dev. But because approvals take so long, the dev has already started working on something else. So that kick-back sits back in their queue for a while, until they pick it up again, and the cycle continues.
In the prod-driven approach, the developer that made the change is hand-picking folks to send the review to. They will want to find somebody else that is familiar with this code, or this area of business functionality, and get their eyes on the changes. The communication is much more direct, and more synchronous. The changes do not sit for a long time with no action. The developer might even want to get immediate feedback from another team member, and pair-program with them, getting essentially an instantaneous review.
Because reviews are fast, the dev doesn’t need to start work on something else, they probably only have time to make sure the tickets are in the right status and any other paperwork is actually done (because we know there’s a good chance the JIRA ticket was actually still marked as “Ready for Dev”).
Expanding
That is just one piece of the pipeline. Now what if we expand that out to a whole SDLC process? Keeping the focus on getting the code to production will naturally lead us towards automating as much of the process as possible. Even parallelizing as much as possible. Why not run the unit tests and the static security scans at the same time? Of course I want to enable SonarLint in the IDE so I can see an issue as soon as I write it, rather than waiting five minutes after I push code.
The developer, and everybody else who is involved with a piece of code, will want to actually follow each change all the way into production, and even watch it in production for a while, just to make sure it gets there safely and is causing no issues.
Roadblock
I feel like I’m constantly fighting against the SDLC process we currently have, and that shouldn’t be how it is. The SDLC process should offer reassuring guardrails, not roadblocks. I should feel invigorated that a piece of code made it into production, not tired and worn down. But of course, whenever I ask why this process sucks, you know what the reply is: “we have to do it this way for the auditors”.
No, we really don’t. We have to do it that way because somebody who doesn’t have any actual interest in getting code from a developer’s laptop into a production system decided this is the way it has to be done. How about we start with the people who actually build the systems to design their runway into production, and you come in to verify we did not miss any checkboxes?