Fluentd record_transformer Example

record_transformer is another filter in fluentd. It allows you to modify a matching record. Let’s take a look at an example using the fluentd record_transformer.

Full Example

Let’s start off with the full example, and then break it down.

 <filter kubernetes.journal.container.**>
   @type record_transformer
   enable_ruby true
   remove_keys hostname,name,msg
   <record>
     message ${record["msg"] ? record["msg"] : record["message"] ? record["message"] : record["MESSAGE"]}
  </record>
</filter>

Break Down

First of all, it will work on matching records:

<filter kubernetes.journal.container.**>
  @type record_transformer
  ...
</filter>

This will run the record transformer over any records with a tag matching the kubernetes.journal.container pattern. Notice the glob-style double astericks, which matches any number of “descendant” patterns.

Enable Ruby

record_transformer supports an enable_ruby boolean parameter. This enable’s Ruby syntax inside the ${} terms in the record parameter. I don’t really know what that means, as the examples in the docs don’t seem to have any difference when this is true vs false. We have it set to true for the examples provided here.

Remove Keys

We also remove some keys from our records, using the remove_keys parameter. We use this to eliminate duplicate data in our records.

remove_keys hostname,name,msg

Removes the hostname, name, and msg keys. From the next section you will see we are normalizing the records’ messages to the message key, so we have no reason to keep msg. Why isn’t MESSAGE also included there? Probably some latent bug we haven’t fixed yet! This effectively runs after the record transform, so we can safely remove keys that we use in the next section.

Record

The most obvious parameter is record. This contains the key/values to add, or update, on the record.

<record>
  message updated ${record["message"]}
</record>

This will simply prepend the original record’s message with the string “updated “. So given an original message like {"message": "hello"} this would output {"message": "updated hello"}.

Our use case for this is around message normalization. We take records that originate from multiple sources and coerce them into a standard format. For example, taking records that may have “message” fields called either msg, message, or MESSAGE and normalizing them to message:

<record>
  message ${record["msg"] ? record["msg"] : record["message"] ? record["message"] : record["MESSAGE"]}
</record>

That is a complex ternary term. It breaks down into

We will be setting the message key
If the record has the msg key, use that
Else, if the record has the message key, use that
Else, fallback to the MESSAGE key