Datadog-agent: Logs in JSON format are prefixed with non-JSON

[jonmoter]

Follow-on from #971.

I've deployed the datadog agent (as a docker container in Kubernetes), setting the DD_LOG_FORMAT_JSON ENV var, and now I see logs appear in JSON format. Kindof. My logs have lines that look like this:

[ AGENT ] {"time":"2018-04-17 00:20:21 UTC","level":"INFO","file":"start.go","line":"174","func":"StartAgent","msg":"Hostname is: i-0d6aeb84eb49fe72b"}

That string is not parseable JSON. It has a sub-string that is parseable JSON. But I'd need to pre-process each log line to extract the part that is JSON, and feed that into my JSON parser.

I'm using fluentd for log collection. So it is possible to do my own regex parsing of logs. But I was hoping that when I enabled logging as JSON, each line would be a valid JSON hash. Ideally, something like:

{"component": "AGENT", "time":"2018-04-17 00:20:21 UTC","level":"INFO","file":"start.go","line":"174","func":"StartAgent","msg":"Hostname is: i-0d6aeb84eb49fe72b"}

Then my log processor is really simple, which is why logging in JSON is appealing.

I'm currently running the agent as a docker image, using the datadog/agent:6.1.0-jmx image.

[jonmoter]

Also, I notice that only enable JSON logging in for AGENT. Logging for the trace agent still appears in non-JSON format.

Is there a way to set JSON logging globally for all components?

[truthbk]

Hi @jonmoter, you definitely have a point, and it also looks like your proposed format would work nicely. Let me discuss internally if there's any strong reason not to make this change (I can't really see any other than perhaps backward compatibility, you know in case other people like you might be ingesting the logs through custom parser). You'll hear back from us soon! Thanks for reporting this.

[mattcamp]

Any update on this one?

I too would like my datadog-agent logs to be parsed correctly.

I'm running in kubernetes using the standard daemon-set and pod logging so don't have fluentd to do any custom regex parsing for me, and I'd like to avoid using a complicated pipeline processor.

[BrianChristie]

Ping @truthbk

[jcardoso-bv]

A further update on this would be appreciated given that your new Cluster Agent is spitting out a proper JSON log without the annoying prefix.