When deploying the logging aggregation tool Fluentd to our Kubernetes cluster, Fluentd was failing to start up. It was failing with a permission denied error when trying to create the directory /var/log/fluent.
I had Fluentd configured to write its position files in the /var/log/fluent directory. /var/log was mounted as a hostDir, from (predictably) /var/log on the host node, in order to read logs from Journal. When Kubernetes mounts directories into a pod, it mounts them with the root user and group, I believe with 755 permissions. While this at first glance seems like the root issue to my problem, it gets a bit more intriguing. The user inside my Docker container is the root user, so it should be able to create directories in /var/log and write files there. And it turns out that user can create directories there. If I change the entrypoint for my container to mkdir /var/log/fluent, or change it to basically an interactive container, I can create the directory just fine. Something else was at play.
It turns out that the default entrypoint for the Fluentd image I was using (the ‘debian’ flavor) runs fluentd with the --under-supervisor flag. Fluentd runs one or more workers, and a supervisor process to handle signals from the OS. But it seems that when running in this way, the user that actually runs (probably the worker) processes is not root. So when the worker goes to create the directory, it is denied.
The solution I found was to change the entrypoint for my container to
ENTRYPOINT [ "fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins" ]
This is nearly identical to the original, but works around the user permissions error. It looks like fluentd still runs with a worker and a supervisor, but for whatever reason this method runs the processes with the correct user, whereas the default does not.