Loki: Docker logging driver?

[prologic]

Are there plans to build a Docker logging driver plugin based on Loki?

[eeddaann]

Instead of implementing it from zero, this driver can be used: https://github.com/deep-compute/docker-file-log-driver
and promtail can be configured to collect the logs from that file

[lfdominguez]

I'm coding one for Docker... (and with support for Docker Swarm tags as service name, etc) https://github.com/lfdominguez/docker_log_driver_loki is in early stage yet.

[lfdominguez]

In futher version i will change to use the protobuff protocol, i need get working with json first gg.

[SvenDowideit]

Noice!

for now, i'm using the journald docker daemon log setting, and then grabing those using fluentd which can push to loki

[nunofernandes]

It would be awesome if we could send the docker logs to loki directly (without having the logs sent to file and then to loki).

[lfdominguez]

It would be awesome if we could send the docker logs to loki directly (without having the logs sent to file and then to loki).

check
https://github.com/lfdominguez/docker_log_driver_loki

[y0zg]

It would be awesome if we could send the docker logs to loki directly (without having the logs sent to file and then to loki).

check
https://github.com/lfdominguez/docker_log_driver_loki

@lfdominguez

Thank you, any guide how to start using it? ;)

[lfdominguez]

It would be awesome if we could send the docker logs to loki directly (without having the logs sent to file and then to loki).

check
https://github.com/lfdominguez/docker_log_driver_loki

@lfdominguez

Thank you, any guide how to start using it? ;)

only use de Makefile and put the options loki-host and loki-port. Soon I will put the configuration on README... sorry

[TigerC10]

+1

Need an official Docker Logging Driver to get buy in with the enterprise.

[cyriltovena]

This is totally a legitimate feature, I’m not sure if the docker plugin is solution. May be for swarm users.

For sure, in the meantime I would recommend to use a service discovery compatible with Prometheus service discovery such as consul, very simple to use.

[TigerC10]

This is totally a legitimate feature, I’m not sure if the docker plugin is solution. May be for swarm users.

I disagree, we use Amazon ECS and Fargate and currently configure Docker to send logs through the Splunk logging driver. We also have some teams using the CloudWatch logging driver. Because of how easy it is to configure logging drivers, pretty much the only way I will get buy-in to switch to Loki from the enterprise teams logging in this way is if Loki also has a logging driver.

[nunofernandes]

Completely agree with @TigerC10 as I do have the same scenario (using cloudwatch logging driver).

[cyriltovena]

Sounds fair, I'll dig into it.

[lfdominguez]

You can help or fork or whatever you want ;) with my plugin

[cyriltovena]

I'll read it for inspiration but I think it's better if this code live in grafana/loki as there is a lot of pkg I could use.

[cyriltovena]

@TigerC10 if we do release a docker driver it won't be added by default to ecs image. Would you install it yourself or will you wait for the support in the ecs ?

[nunofernandes]

@Kuqd yes.. I do that for other drivers (rexray/ebs and efs). So I would just add an aditional driver there.

[TigerC10]

@Kuqd Yes, we maintain our own ECS images for other monitoring and security software, can easily add Loki driver to the images.

[cyriltovena]

Alright so that also seems fair to add the possibility to parse lines via config file optionally so you can index log level.

host and container_name labels will be there by default and you would be able to include anything from docker labels and docker env container runtime via plugin option.

[cyriltovena]

I'm aiming for this config:

{
  "log-driver": "loki",
  "log-opts": {
    "url": "http://loki/api/prom/push",
// many other tls option.
    "external-labels": "container_name={{.Name}},env=prod",
    "labels": "production_status",
    "env": "os,customer"
  }
}

[lfdominguez]

I'm aiming for this config:

{
  "log-driver": "loki",
  "log-opts": {
    "url": "http://loki/api/prom/push",
// many other tls option.
    "external-labels": "container_name={{.Name}},env=prod",
    "labels": "production_status",
    "env": "os,customer"
  }
}

umm.. i will change the config options of my plugin... now are
loki-host & loki-port

[cyriltovena]

how do you usually deal with config file when using docker drivers ? I don't seem to understand how to mount a hostpath in my plugin to add a config.

[lfdominguez]

I don't have config files... the config is on docker daemon or used directly on docker command

[TigerC10]

In AWS, we use the USERDATA feature of EC2 and/or the Init Scripts in CloudFormation Templates. See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html

So, on the consumer side of this plug-in, we would install the Loki driver plugin with USERDATA on the instance profile, then we would use the Init Scripts to define the config file(s).

[lfdominguez]

With it you can populate the /etc/docker/daemon.json (Debian) with the configuration, and every container use them.

[cyriltovena]

Thanks everyone, I should release the driver today or tomorrow.

@lfdominguez I've added swarm support like you did but using the ContainerLabels property as the extracted data doesn't contains those swarm labels.