Telegraf: docker log input plugin

[alexbardas]

Feature Request

Proposal:

Allow parsing the docker logs output for a container. Keep in mind that it's useful to differentiate between the 2 types of output: stdout & stderr.

e.g:
docker logs 2>/dev/null shows stdout logs
docker logs >/dev/null shows stderr logs

Current behavior:

Logparser supports only reading logs from files.

Desired behavior:

Logparser should be able to parse stdout / stderr of docker logs <container> cmd.

Use case: [Why is this important (helps with prioritizing requests)]

It's considered best practice to write logs to stdout / stderr.
e.g. nginx's Dockerimage:

&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log

Given that many services can run in different docker containers, it becomes hard to manage all the volumes which need to be shared with the telegraf container. It would be very handy to declare & manage this in telegraf.conf.

[sparrc]

the logparser plugin is only going to work for logfiles, not arbitrary commands. I would recommend using the exec plugin with a custom script, or redirecting your docker log commands to a file to get around this.

There is also a docker input plugin which may suit your needs better.

[alexbardas]

Thanks for the suggestion. It makes sense to use the exec plugin, just unfortunate that it cannot be extended by the LogParser plugin.

Maybe a better idea would be a new docker log input plugin similar in functionality to the Heka's DockerLogInput (https://hekad.readthedocs.io/en/v0.10.0/config/inputs/docker_log.html) which is very handy to have.

[sparrc]

that sounds reasonable to have a plugin specifically for that, I'll reopen the case

[xmackex]

+

[russorat]

anyone tried using the new syslog input with docker?

[prashanthjbabu]

I've created a PR with a new docker log input plugin https://github.com/influxdata/telegraf/pull/4360

@russorat
I have tried the the syslog plugin with docker and it works ! The only challenge being, the moment you redirect to syslog , you lose log access via "docker logs" or the docker engine APIs for that matter.
It also requires the docker container to have the syslog option as part of its run parameters(or docker-compose file) OR for the docker daemon config to be edited to transfer all container logs via syslog which maybe a bit intrusive.

[srgl]

Nice to see that the plugin is scheduled for the next version, hope to see it soon! While that there is the logspout router, which routes docker logs to any syslog server. And it seems it works well with the telegraf's syslog input plugin. I've put together a docker image and docker-compose example https://github.com/srgl/telespout

[IosifZ]

Nice to see that the plugin is scheduled for the next version, hope to see it soon! While that there is the logspout router, which routes docker logs to any syslog server. And it seems it works well with the telegraf's syslog input plugin. I've put together a docker image and docker-compose example https://github.com/srgl/telespout

Many thanks for your briliant setup. For the moment is exactly what I was searching for.