Cri-o: How to set containers logs format?

[luizbafilho]

Hi,

There is a way to setup containers log format to json?

Today, the logs are in plain text

2017-09-12T22:32:21.212861448+00:00 stdout 2017-09-12 22:32:21.212 [INFO][88] table.go 710: Invalidating dataplane cache ipVersion=0x4 reason="refresh timer" table="nat"

But I would like them to be Json, like docker does:

{"log":"2017-09-12 22:19:32.904 [INFO][92] table.go 431: Loading current iptables state and checking it is correct. ipVersion=0x4 table=\"filter\"\n","stream":"stdout","time":"2017-09-12T22:19:32.906219687Z"}

That very important, because not being Json, will break every single fluentd deployment.

I guess most of people who uses fluentd, has the following config

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  format json
  read_from_head true
</source>

which make fluentd flood the logs with the pattern not match: error.

That would not be a problem if your entire cluster is using CRI-O,you could just change the fluentd format, but I guess that most people will follow the same path I'm, changing 1 node at time.

[TomSweeneyRedHat]

@baude @umohnani8 PTAL

[rhatdan]

@TomSweeneyRedHat This is a crio issue not a kpod logs. But it is probably a good idea if we allow someone other then @mrunalp or @runcom to look at this.

[runcom]

So, we follow what kubernetes needs for logging and afaict it's just plain text. Kubernetes reads from plain text under /var/log/pods/<id> and expect a given format. We can't just switch logging format and CRI-O isn't logging anywhere else but just where Kube reads those logs.

That very important, because not being Json, will break every single fluentd deployment.

isn't Fluentd able to read Kubernetes format? that would highly surprise me. You're probably talking about "docker logs", Fluentd understands "docker json logs". That is another stuff...

That said, we just can't switch log format, we do what Kube asks for. One option would be to write to json in another location but that's IO overhead I don't want to incur into...I believe the best is to create a Fluentd parser fo Kubernetes' /var/log/pod/<>

[runcom]

For reference, you're using the fluentd-docker plugin https://github.com/fabric8io/docker-fluentd-kubernetes/blob/master/fluent.conf ... so it's related to docker, we need a Kube-plugin maybe :)

[mrunalp]

We will add support for fluentd to understand the CRI log format.

On Sep 13, 2017, at 6:14 AM, Antonio Murdaca [e-mail] wrote:

For reference, you're using the fluentd-docker plugin https://github.com/fabric8io/docker-fluentd-kubernetes/blob/master/fluent.conf ... so it's related to docker, we need a Kube-plugin maybe :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[luizbafilho]

@runcom Fluentd can definitely read Kubernetes format, but as a said, I think that you should provide the smoothest path for users to move to CRI-O. Not providing a JSON output, you force people to migrate all their nodes at once, given that fluentd is deployed as a daemonset, unless you provide a way to let Fluentd detect what is the node container runtime and parse the logs accordingly.

What I'm asking is just to provide a option to output json format just like you already have for cri-o's logs.

I know, that is not required by kubernetes but whether you like it or not, everyone runs docker in their cluster, and those people might want to migrate to CRI-O, and no in his right mind will do it in all nodes at once.

We can't just switch logging format

There is any technical reason for this?

[runcom]

There is any technical reason for this?

Well, the technical reason is CRI needs a plain text format. If you switch the whole cri-o to use json you'll break CRI logging.

BTW, I haven't said we won't support this ever, indeed I also provided some paths forward but it still needs thinking. For instance, as @mrunalp said, we'll probably create a fluentd adapter for the CRI logging.

[runcom]

And thanks for pointing this out! Of course we do care about a smooth transaction to use cri-o

[rhatdan]

@luizbafilho Are you suggesting we support writing both? Since kubernetes will be reading its own format, while fluentd will be attempting to read the docker json format.

[runcom]

Dan, because the fluentd docker plug-in looks for that. Best would be to have fluentd understands the kube format directly

[luizbafilho]

@rhatdan no, writing just one. Kubelet would have to read json. But I guess that is not possible if we want to comply with CRI.

as @runcom mentioned a fluentd plugin might be the best way. But I would like to suggest something.

I don't know if it's even possible, but, the fluend cri-o plugin could detect what format is before starts collecting and parse accordingly. With that in place, the transition would be easier.

[luizbafilho]

Good news, looks like I found the solution: https://github.com/repeatedly/fluent-plugin-multi-format-parser

I'm going to test it, to see if works out.

[luizbafilho]

Tested the multi_format parser and works well.

For those who wants to try, here is how the config should look like when using fluentd v0.14, the syntax is a bit different when using fluentd v0.12, please check out the plugin readme for more details.

<source>
      @type tail
      tag kubernetes.*
      path /var/log/containers/*.log
      pos_file /var/log/kube-containers.log.pos
      read_from_head true

      <parse>
        @type multi_format
        <pattern>
          format json
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format regexp
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
          expression /^(?<time>.+)\b(?<stream>stdout|stderr)\b(?<log>.*)$/
        </pattern>
      </parse>
</source>

[StephanX]

@luizbafilho It only took me two days of banging my head to find your comment, after finding the same solution.

For other wandering souls, if you get flooded with fluentd logs with seemingly infinite amounts of '\\\\\\\\\\\\\\\\\\\', it's probably because you're using fluentd 0.14, and used

format /your-regex-here/

instead of

format regexp
expression /your-regex-here