If you need to get log events from fluentd up to a Splunk HTTPEventCollector (HEC), I’ve found that you have two options, splunk-http-eventcollector and splunkhec. Both of these plugins have fallen out of use in the last couple of years. When using either of these plugins, you lose the milliseconds in log event times. So what? We have the original timestamp on the log message itself. Not a big deal. However in this scenario, Splunk has no way of ordering multiple messages received with the same second. It is very difficult to get the correct sequence of events in this scenario. This renders log aggregation useless.
These Fluentd plugins are not the only things that exhibit this problem. It seems like a lot of the Splunk tooling drops milliseconds from log events. I do not know if this is consistently a poor configuration on our part, or if Splunk prefers by default to just upload events with a “generic” time on them. You can always set up a record parser to pull out the true time when indexing. This is an unnecessary process, but is this how it is supposed to work? Are there some benefits to this method that I do not see?
To preserve the millisecond precision on my log events, I had to make a small modification to the splunk-http-eventcollector plugin (see the change). Unfortunately, the plugin hasn’t had a new release since that change (2 years ago). To make use of this you will need to build the plugin locally.
Important Note
Since I originally looked at these plugins, it looks like Splunk itself has created a fluentd plugin for the HttpEventCollector. https://github.com/splunk/fluent-plugin-splunk-hec. I have not evaluated this plugin to see if it has addressed this issue, but it is on my list of things to look into.