How do you keep microservices up to date? I don’t mean the code that you write, but all the underlying stuff that goes into your services. Things like the OS, programming language, software packages, etc.
Monolithic
In a monolith, this was relatively easy to take care of. There was a very small number of servers and code bases that needed to be kept up to date. The hard part was that the upgrades were usually long overdue, and required many weeks of effort to actually pull off.
There were also multiple teams involved. Developers did not have control over the boxes that the code ran on. Other teams controlled those boxes, and handled the updates. They were responsible for keeping the OS and other low-level packages up to date. The developers are responsible for keeping the software packages up to date.
There is a bit of a gray area around the programming language, or any other OS packages specifically required by the application. These need coordination between teams or departments. The developers cannot just update the programming language without telling the Operations people; otherwise the application will fail on any environments outside of Dev control. The Ops people cannot just blindly update Java, or it might also cause the application to crash.
Docker
Docker helps greatly with that gray area. Now the developers can control the entire stack that they care about. All the way from the OS up to the application code. There is a clear distinction between the maintenance of the boxes that the software runs on and the containers that run there.
Docker itself of course needs updated periodically, but this seems to be a much easier problem than others in the past. The compatibility of Docker images with versions of the engine allows for pretty easy upgrades.
However, this now introduces a new problem. Now we have a bunch of containers out there in the wild, all of which are running different versions of software. A new CVE just came out for software x, which containers are effected? How do we update them?
No Latest
It is best practice to not use the latest tag for images that are running production loads. It is even questionable to use tags like stable or “latest-version-x”. There is risk with these, that whenever a service is updated for some minor change, that it can start seeing failures around some random OS package that happened to be updated in the latest version of the base image. This can of course be mitigated with proper testing of the service, but developers can still get mad when their “simple change” turns into a week’s worth of debugging and fixing.
It also looks odd, especially from an audit perspective, to have new things deployed without any code changes. Pushing tickets that say things like “rebuilding to get latest base image” is just a bit odd. Having your production environment always pull the “latest” version of a service on restart (Kubernetes supports this) faces a similar issue. While the containers may invisibly update, there can also be that one container out there that has been alive for months and is now very lacking in updates.
But on the other hand, enforcing code changes and new pushes for any code updates becomes very hard to maintain. With any real number of microservices, you may feel pressure to hire a team dedicated to just “curating” microservices. To just go through all the microservices and update them to the current versions. That obviously is not a good approach though, as now the services are not owned by a team.
Something In The Middle
There must be something in the middle there. Something between requiring a dev to go in and manually make changes, and blindly accepting any upstream updates. Perhaps an automated system that sees changes to a base image, and magically goes and updates all the child images?