Loki: Display docker logs with human/logical name

[Andysimcoe]

I have promtail set up to scrape Docker container logs and can view them in loki/grafana.

I'm wondering if there's a solution to displaying them with meaningful names, like the names giving to the container? My promtail config looks like:

- job_name: docker
  entry_parser: raw
  static_configs:
  - targets:
      - localhost
    labels:
      job: dockerlogs
      __path__: /var/lib/docker/containers/*/*log

So currently if I go to Explore in Grafana and select Loki, I can choose the entire dockerlogs job or individual files - the JSON logs are obviously their file name which doesn't mean much to people without host access or looking through each.

I'm not using Kubernetes. Do I need to create some kind of custom label which will survive a container being redeployed?

If this is a stupid question I'm happy to be pointed to some code or doc that will help me.

Really liking this already though.

[leominov]

Hello, @Andysimcoe. Do you try entry_parser: docker?

[Andysimcoe]

@leominov yeah that's fine for removing a bit of noise but I mean the ability to select a log for one container by its name. The attached image might help:

[leominov]

Yeah, those information does not exists in docker json logs and promtail does not collect it from daemon – https://github.com/grafana/loki/blob/master/pkg/promtail/api/entry_parser.go#L62 :(

[retzkek]

My current solution (not in production yet) for docker is to use a cron job that runs docker ps to write a service discovery file:

docker ps --format '- targets: ["{{.ID}}"]\n  labels:\n    container_name: "{{.Names}}"' > /etc/promtail/promtail-targets.yaml

Then in the promtail config use relabeling to get the log files:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: http://loki:3100/api/prom/push

scrape_configs:
- job_name: containers
  entry_parser: docker
  file_sd_configs:
  - files:
    - /etc/promtail/promtail-targets.yaml
  relabel_configs:
  - source_labels: [__address__]
    target_label: container_id
  - source_labels: [container_id]
    target_label: __path__
    replacement: /var/lib/docker/containers/$1*/*.log

A Docker logging driver ala #195 or logspout ala #82 would be better of course, but in the meantime this seems to work, mostly: I've been having problems removing containers while promtail is running, since it's holding open the log file.

[Andysimcoe]

Thanks for that @retzkek , that's good enough for me right now. For whatever reason something wasn't seeing the change but I happened to update to the latest Grafana beta (3) along with the latest promtail and loki and it's showing as expected.

Cheers,

[daixiang0]

@tomwilkie do you think it need to support in the future?

[ratio91]

"WARNING!!! entry_parser config is deprecated, please change to pipeline_stages

@retzkek is there a way to do this with pipeline stages as well? thanks!

[retzkek]

@ratio91 I haven't had a chance to do any more with loki, sorry. Since the docker container logs are just json by default it should be straightforward to do. https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md

[cyriltovena]

You can use the docker driver now and will automatically named correctly container/stack/service.

see https://github.com/grafana/loki/blob/master/cmd/docker-driver/README.md

[megamorf]

FYI: Scraping the docker container logs only works as long as they are stored in json format. Recently, the local driver was added which essentially does the same as the json driver but uses protobuf instead of json (https://github.com/moby/moby/blob/master/daemon/logger/local/local.go). in that case, one is forced to use the docker API to query container events.

[bukowa]

You can use this config (json-driver supports this, i don't know about the other drivers).
shell script docker run --log-opt tag="{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}" ...

````yaml
server:
http_listen_address: 0.0.0.0
http_listen_port: 9080

positions:
filename: /tmp/positions.yaml

clients:

• url: http://loki:3100/loki/api/v1/push

scrape_configs:

• job_name: system
  static_configs:

  • targets:
    
    
    
    • localhost
      
      labels:
      
      job: varlogs
      
      __path__: /var/log/*log
    
    

• job_name: containers
  entry_parser: raw

  static_configs:

  • targets:
    
    
    
    • localhost
      
      labels:
      
      job: containerlogs
      
      __path__: /var/lib/docker/containers//log
    
    

  --log-opt tag="{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"

  pipeline_stages:

  • json:
    expressions:
    stream: stream
    attrs: attrs
    tag: attrs.tag

  • regex:
    expression: (?P(?:[^|][^|])).(?P(?:[^|][^|])).(?P(?:[^|][^|])).(?P(?:[^|][^|]))
    source: "tag"

  • labels:
    tag:
    stream:
    image_name:
    container_name:
    image_id:
    container_id:

````

[venkat424]

I have promtail set up to scrape Docker container logs and can view them in loki/grafana.

I'm wondering if there's a solution to displaying them with meaningful names, like the names giving to the container? My promtail config looks like:

- job_name: docker
  entry_parser: raw
  static_configs:
  - targets:
      - localhost
    labels:
      job: dockerlogs
      __path__: /var/lib/docker/containers/*/*log

So currently if I go to Explore in Grafana and select Loki, I can choose the entire dockerlogs job or individual files - the JSON logs are obviously their file name which doesn't mean much to people without host access or looking through each.

I'm not using Kubernetes. Do I need to create some kind of custom label which will survive a container being redeployed?

If this is a stupid question I'm happy to be pointed to some code or doc that will help me.

Really liking this already though.

Hi , how do you scrape docker logs , is it required to use sudo somehow to fetch the logs?

[cyriltovena]

Depends on your system, although you don't need to care about any files if you use the docker driver. https://github.com/grafana/loki/tree/master/cmd/docker-driver

[advissor]

My current solution (not in production yet) for docker is to use a cron job that runs docker ps to write a service discovery file:

docker ps --format '- targets: ["{{.ID}}"]\n  labels:\n    container_name: "{{.Names}}"' > /etc/promtail/promtail-targets.yaml

Then in the promtail config use relabeling to get the log files:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: http://loki:3100/api/prom/push

scrape_configs:
- job_name: containers
  entry_parser: docker
  file_sd_configs:
  - files:
    - /etc/promtail/promtail-targets.yaml
  relabel_configs:
  - source_labels: [__address__]
    target_label: container_id
  - source_labels: [container_id]
    target_label: __path__
    replacement: /var/lib/docker/containers/$1*/*.log

A Docker logging driver ala #195 or logspout ala #82 would be better of course, but in the meantime this seems to work, mostly: I've been having problems removing containers while promtail is running, since it's holding open the log file.

Did you find a wat not to lock the logs files ? Something like Read Only mode ?