Loki: Docker driver doesn't work with warning "no logs are available with the 'loki' log driver"

[rndmit]

Describe the bug
Docker driver configured as written in docs but after running docker-compose up docker shows warning. And there's also no logs in Loki.

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (master)
2. Installed and enabled Docker-driver via docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions command
3. Added logging section to docker-compose file as said in docs
4. Run docker-compose up
5. See the warning: app_1 | WARNING: no logs are available with the 'loki' log driver

Expected behavior
Logs should fall into Loki but they don't.

Environment:

• Infrastructure: Docker Desktop 2.1.0.5 (Docker Engine 19.03.5 and Compose 1.24.1) on MacOS Catalina
• Deployment tool: docker-compose

Screenshots, Promtail config, or terminal output
docker-compose.yml

version: '3.7'

services:
  loki:
    image: grafana/loki:master
    container_name: loki
    command: -config.file=/etc/loki/local-config.yaml
  app:
    build:
      context: .
      dockerfile: Dockerfile
    logging:
      driver: loki
      options:
        loki-url: "http://loki:3100/loki/api/v1/push"

[cyriltovena]

This is because the driver runs with host network, I need to try bridge and make it defaults.

Edit: not sure actually

[inliquid]

Same weird behavior in my case.

[dmitriyminer]

@inliquid, @scobcov
Try this one:

version: '3.7'

services:
  grafana:
    image: grafana/grafana:master
    ports:
      - "3000:3000"
    environment:
      GF_EXPLORE_ENABLED: "true"

  loki:
    image: grafana/loki:master
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  app:
    image: alpine:latest
    logging:
      driver: loki
      options:
        loki-url: "http://host.docker.internal:3100/loki/api/v1/push"
    entrypoint: ping host.docker.internal

[rndmit]

@dmitriyminer well, it works but looks like a kludge. I think it's a really bad idea to expose Loki outside. However, thanks. I'll use it as a temporary solution and restrict access with a firewall.
Btw, I'm still waiting for a solution from devs.

[cyriltovena]

We don’t have a solution because we are restrained by how docker plugins work which is either bridge or host network, currently it configure as host. I’m really busy to try out bridge which should solve most use cases except custom network. Wanna try bridge here is an image:

grafana/loki-docker-driver:latest-bridge

[tomdsmartdata]

@cyriltovena, I'm encountering a similar as @scobcov and am interested in trying your bridge image. Where can I pull it from?

[cyriltovena]

grafana/loki-docker-driver:latest-bridge

[rhbroberg]

As of 2.2.0.3 (42716) DockerDesktop on the mac, or 19.03.5 on linux, neither of these approaches works for me: both of them result in the same error message from 'docker-compose logs':

Attaching to docker-loki_app_1, docker-loki_grafana_1, docker-loki_loki_1
app_1 | WARNING: no logs are available with the 'loki' log driver

I'm using the exact docker-compose file as posted above, and I've tried it with the grafana/loki-docker-driver:latest-bridge plugin installed. Any suggestions?

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[moeenz]

I'm experiencing the same thing on Mac and Ubuntu. Tried with both :latest and :latest-bridge. Exposing Loki port or not, the problem persists.

[cyriltovena]

Hey Everyone,

For those who have the issue related to no logs are available this is in fact on purpose from the docker-compose tooling.

see https://github.com/docker/compose/blob/master/compose/container.py#L196

Basically no log from docker-compose if you use a plugin, not sure why. But docker logs should work still, if you want to use that.

[andoks]

I too have experienced this issue. I have worked around it by binding port 3100 of the Loki container to the hosts' port (127.0.0.1:3100), but I expect that when I'm going to run more than one compose-project on the same host, each with their own Loki container inside, this will break down and I will have to change the compose-file.

I would love for this issue to be reopened. A solution to the issue I would really like, is if the loki logging driver got an option for specifying that the push-url is inside the source docker bridge network (or something more powerful and flexible if there are more use-cases to cover)

[cyriltovena]

@andoks what you asking is not really related to this issue. But if you want to try the bridge network be my guest here is the image grafana/loki-docker-driver:latest-bridge, so far no one has reported that it fixed the network issue.

From my understanding this is a docker limitation.

[andoks]

what you asking is not really related to this issue

@cyriltovena in that case I am sorry.

I thought the original issue as described was "I am not able to send logs to the container running inside the compose setup using the docker loki logging driver".

And that dmitriyminer in his post provided a work-around for this by suggesting binding lokis' API port to the host to make it available to the logging driver.

But if you want to try the bridge network be my guest here is the image grafana/loki-docker-driver:latest-bridge, so far no one has reported that it fixed the network issue.

From my understanding this is a docker limitation.

I tested it in place of grafana/loki-docker-driver:latest using loki 1.5.0 in a container. It did not solve my issue with wanting to log to the loki container running inside the compose setup without binding the loki API port to a host port. Actually, the work-around (binding the loki-container api port to the host) did not work with the latest-bridge. Should it have worked?

[cyriltovena]

Not if you're using the default bridge.

see https://docs.docker.com/network/bridge/#differences-between-user-defined-bridges-and-the-default-bridge

[inliquid]

not really related to this issue

It's absolutely related. Many ppl asked about this in Slack as well. The problem is that Loki Docker driver should recognize docker host names instead of just one single 127.0.0.1.

[cyriltovena]

not really related to this issue

It's absolutely related. Many ppl asked about this in Slack as well. The problem is that Loki Docker driver should recognize docker host names instead of just one single 127.0.0.1.

It has nothing to do with Loki, I've answered it many times, this is a docker plugin restrictions, we can only set the network to either host or bridge.

Please if I'm wrong show me in the doc where I can allow the driver to access docker network.

[cyriltovena]

see

[andoks]

@cyriltovena: I gave the latest-bridge another go, but I was still unable to solve my problem with it (and unable to actually get data into loki using the plugin, but that might just be something I did wrong).

After reading up a bit, I think I understood the bridge feature better: that instead of binding to the host network, it binds to the built-in docker default network called "bridge" (the description is kinda terse. This was my interpretation of it at least, please correct me if I am wrong).

This means that the loki instance I have currently running in my docker-compose setup still is unavailable for the loki driver plugin in "bridge" mode. I tried experimenting with putting the loki container inside the "bridge" network, and learned some of the pains of using this network: no built-in service dns, not possible to be connected to a user-defined docker network at the same time as you are connected to the built-in "bridge" network etc.

After discovering the two issues above, this was enough for me to conclude that binding the loki-containers' port to the docker host, and pushing logs using the loki driver plugin as the work-around described above, will be far less of a hassle. The issue with this of course, is that it will not be possible to run multiple systems from the same compose-file without changing that port. It is not perfect, but it is something I got working, and it is relatively obvious (to describe, and to reason about).

Thanks for your help trying to find a solution to this.

PS: is this a TBD repo?

[cyriltovena]

Thanks for the update, I think docker never really intended to allow logging driver to connect to interval network and that’s why it’s like this.

If someone wants to push this further we could open an issue on moby/docker side.

And no that repo is not used, we will keep the code of the driver in the main repository.

[tam481]

Hello all,
I'm confused. Are we saying that the docker loki driver does not work if the containers are started using docker-compose because of some limitation?!

I have loki running behind traefik and the driver configured to push docker logs to it via docker-compose.yml for the services I am running. None of them seem to be logging anything to loki.

Loki returns ready when I visit (https://loki.domain.tld/ready) but nothing is actually being logged.

I tried the "grafana/loki-docker-driver:latest-bridge" to no avail

[denisgolius]

what you asking is not really related to this issue

@cyriltovena in that case I am sorry.

I thought the original issue as described was "I am not able to send logs to the container running inside the compose setup using the docker loki logging driver".

And that dmitriyminer in his post provided a work-around for this by suggesting binding lokis' API port to the host to make it available to the logging driver.

But if you want to try the bridge network be my guest here is the image grafana/loki-docker-driver:latest-bridge, so far no one has reported that it fixed the network issue.
From my understanding this is a docker limitation.

I tested it in place of grafana/loki-docker-driver:latest using loki 1.5.0 in a container. It did not solve my issue with wanting to log to the loki container running inside the compose setup without binding the loki API port to a host port. Actually, the work-around (binding the loki-container api port to the host) did not work with the latest-bridge. Should it have worked?

docker pull grafana/loki-docker-driver:latest

Using default tag: latest
Error response from daemon: pull access denied for grafana/loki-docker-driver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Or with latest-bridge

docker pull grafana/loki-docker-driver:latest-bridge
Error response from daemon: pull access denied for grafana/loki-docker-driver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

[andoks]

@tam481: no, the docker log loki driver works fine, it is just that since it is sitting on the host network, it does not have automatic access to the loki container running within a docker-compose project. The way I made this work, was to bind the loki containers log input port to host, and then the docker log loki driver plugin has access to write logs to loki over the host port

@denisgolius: I am pretty sure you see those errors due to pulling (docker pull) the images instead of installing them using the plugin command.

[tam481]

hi @andoks
I'm running Loki behind Traefik v2 reverse-proxy. I can log to it just fine from other devices using promtail. However I cannot get docker to ship its logs using the driver. Port 443 is bound to the IP address of the device so when you run docker ps you can clearly see that port 443 is bound to it correctly but I still can't get docker logs shipped to Loki from the local docker! I don't have docker running elsewhere so I can't test it from a remote device.

[tam481]

Hello all
I hope this helps someone.

I resorted to assigning a static address to the Loki container in Docker e.g. 172.20.0.10 and configured the docker-compose projects to log to Loki internally via IP address

&loki logging: driver: loki options: loki-url: "http://172.20.0.10:3100/loki/api/v1/push"
Logging is working perfectly now. Externally, logs are going via Traefik on port 443 reverse-proxied to 3100

[cyriltovena]

Sounds like a good work around well done.