Today, there are a plethora of quality, free tools designed to help teams build a pipeline for taking code from the developer’s head and placing it in front of paying customers. Some of these tools only come into play for a small portion of that pipeline, and some are really the backbone for the pipeline itself. What should these pipelines really do? Where should they come into play, and where should they fade away into the darkness? How do they work to bring a Software Delivery Life Cycle to life?
In The Beginning
A software’s life cycle really starts as an idea in somebody’s head. This quite often isn’t even a software developer’s head. Now this idea needs to be communicated to somebody that knows how to take that idea, and turn it into something a computer can understand and do something with. Many times the idea needs to be refined, vetted, pared down, and reworked before anything useful can actually be codified. These ideas need to be captured, and they need to be fed into this huge machine that we will be building. They will act as the fuel that keeps the machine churning.
Tools that help in this realm are ones that developers tend to learn to hate. They include things like bug trackers, iteration planning tools, Kanban boards, etc. even all the way down to low-tech index cards. Developers tend to start hating these tools because of what all these tools grow to become. The managers and people who create the ideas start wanting metrics on how their ideas are progressing, and those metrics tend to be gathered in these tools, since that is a very natural place for these folks to go to try and find that information. However, the developers want to get in and out of these tools as quickly as possible, because for them it is really rote and uninteresting work. Then you ask them to spend even a few moments of extra time here to add in how long they spent working on something, and the developers begin to loath these tools.
These tools all should be around driving conversation and collaboration between product folks (the ones generating and curating ideas), developers (the ones implementing those generated ideas), and the customers (the ones supposedly benefiting from those ideas). Some considerations here are how to track an idea from conception, to validation, through iterative refinement, into implementation, and finally into the customer’s hands. Technically, this may not have much to do with the build pipeline itself, but the tools that come into play here generally will be encoding information that will be required in the actual pipeline, and feedback from the pipeline will need to be directed into these tools.
From Dev Head to Keyboard
Once the developer grabs hold of the idea, he or she will need a few tools at their disposal to translate the idea into a language even a computer can understand. The tools here are the fairly obvious things like code editors, version control systems, database tools, and even programming languages themselves. Discussions around these types of tools can turn “religious” very quickly, so tread lightly.
Developer preference landmines aside, what conversations are these tools designed to facilitate? Likely, one of the first that comes to mind is a conversation between the developer and the idea generator. Even when the idea was discussed before-hand, the rigorous precision that the computer will ultimately demand will likely force some questions that were not yet thought of. Or change answers that were given previously. But another very important conversation (or at least collaboration) here is between the developer and the computer, or software, itself. Sometimes this conversation can be very abusive and one-sided, and sometimes it can be as easy and natural as talking to an old friend. Where this ends up landing on that spectrum quite often is up to the developer. Sometimes it is also predicated on “all the jockeys that were there before me” too.
The software can communicate back with us too; it isn’t just a one-way street. This collaboration also isn’t like reading a book either, where the developer is taking in predefined information. It is more interactive than that, where the code should be helping out by telling us when it is sick and not performing well, or when too much work suddenly showed up for it to process, or when other collaborators it relies on are not working as expected.
Quality
As the idea is being translated from human language to computer language, bits and pieces can be easily lost in that translation. We tend to double-check ourselves at this point, to make sure nothing too big was missed, and go back and fix it if something was missed. This can be a range from simply making sure two or more different people expect the software to behave in the same way (codified in a QA test and the Developer’s code) to automated checks for common bugs deep in the code.
The tools that can come into play here can be unit testing frameworks, code linters, static code analysis, integration test frameworks, etc. The conversations that should be driven from these stages are around the translation, and again double-checking any assumptions that were made by either the idea generator or the developer. “Is it really supposed to do this?” “This button looks different than it does on that other page.” “What happens if we put a negative number here?”
Delivery
If the translation has been vetted, and the best attempt has been made to thwart any lingering bugs, the software should be ready to be delivered to the customers. The types of tools that come into play here vary greatly depending on how exactly the software is delivered. Cloud hosting services, physical media tooling, over-the-wire transfers, etc.
What conversations are driven here? Mostly these are between the customers and the “idea” folks, as a final check to make sure this new idea really is useful. Or to say “hey this is great, but why does it take so long when I try to run 16 years worth of data through it?” The conversations can also be more narrow; as quick tests to validate an idea that we could not before. “If we turn this on, how does it affect user experience?” “Does the upgrade still work as expected?”
SDLC
The above is a very broad-brush view of what is probably a typical Software Delivery Life Cycle. But did we ever actually get around to answering the question in the title, what lives in the build pipeline(s)?
While you may have been hoping for some direct technical answers like “make sure you have a SonarQube check in your pull request build, and fail the build when the Quality Gate turns red”, that type of guidance is really very specific to individual teams and organizations. What pieces go where depend greatly on organizational and team maturity, how rigorous your SDLC needs to be, legacy code versus now code, etc. Do you need to track who is touching each piece of code for audit purposes? Do you need to lock down the third-party tools and projects available to the software? I cannot address all of those possible permutations here, but I can hopefully give some pointers towards who needs to be involved at each step in the flow, and how to strike up the appropriate conversations at those steps. In short, what lives in the build pipelines are the things that can either facilitate conversations, such as “Is this ready to go to prod?” “Yes, the testing stage passed without issue, I’m confident we can go to prod”, or strike up conversations like “Hey, it looks like you added a hard-coded password here, why didn’t you extract that into a secure file?”