When deploying the logging aggregation tool Fluentd to our Kubernetes cluster, Fluentd was failing to start up. It was failing with a permission denied error when trying to create the directory /var/log/fluent. I had Fluentd configured to write its position files in the /var/log/fluent directory. /var/log was mounted as a hostDir, from (predictably) /var/log on the host node, in order to … Read more
If you need to get log events from fluentd up to a Splunk HTTPEventCollector (HEC), I’ve found that you have two options, splunk-http-eventcollector and splunkhec. Both of these plugins have fallen out of use in the last couple of years. When using either of these plugins, you lose the milliseconds in log event times. So what? We … Read more
I’ve spent a lot of time trying to get our logging to behave sensibly with all the microservices we have running in Kubernetes. Some of our apps log in JSON, some log with positional parameters, some log with a key=value, some use a mixed format, where part of the message is positional, and the rest … Read more
I think it is a bit of a shame that software developers, and technical folks in general, do not study more philosophy. It turns out that many of these old stuffy guys have thought about many of the things that we actually run into on an ongoing basis. Even the vocabulary that the ancient philosophers … Read more
We all know that architects need to make decisions. These decisions have lasting impacts, and can be in various states, such as new, in force, obsolete, etc. How do you record these decisions, and keep those records up to date? I recently read through this blog post: http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions which describes what looks to me to … Read more
If you work in any kind of traditional office space, you are likely plagued by drive-by interruptions. People will constantly stop by and talk to you. Some of these conversations are fruitful, some are useless, many could be a chat or email, and all of them cause you to lose focus on what you were … Read more
Do you spend your time researching and debugging bugs that aren’t really bugs? They turn out to be misconfigurations, or the system functions as designed, but perhaps not as this customer expects? How do you improve the quality of the bugs you get, so that you don’t spend time trying to track down false positives? … Read more
When an organization moves away from a centralized monolith to more decentralized and distributed architectures, such as microservices, the development culture, and therefore, the culture around the system’s architecture also changes. It too must become more distributed and less centralized. Development teams, because they are ideally autonomous in a microservices environment, require more freedom than … Read more
Do you lead your customers, or follow them? Most companies really start by following the customers. That is how you get capital, you prove there is a need, and that you can fill that need. It may be a need that nobody has seen before, or tried to do anything about before, but you are … Read more
Do you know what you want, and who you want to be? It seems like developers quite often do not actually know this, or if they do, they do not take any steps to lead them down the road they want to walk. Where do you want to be in your career in 5 years? … Read more