Loki: Loki Docker Driver "loki-pipeline-stage-file" no such file or directory

[Strum355]

Describe the bug
Loki docker driver gives "no such file or directory" for the given file to loki-pipeline-stage-file set in a docker-compose.override.yml section, shown below

To Reproduce
Steps to reproduce the behavior:

1. Loki version: grafana/loki:0e04ca2133fb
2. Promtail version: grafana/promtail:518791014789
3. Docker driver:

docker plugin ls
ID                  NAME                DESCRIPTION           ENABLED
e0eb93aa1f77        loki:latest         Loki Logging Driver   true

Expected behavior
I guess it should find the file

Environment:

• Infrastructure: local PC
• Deployment tool: using sample docker-compose file from either loki or grafana repo

Screenshots, promtail config, or terminal output

    logging:
      driver: loki
      options:
       loki-url: "http://localhost:3100/api/prom/push"
       loki-pipeline-stage-file: '/home/noah/Netsoc/dev-env/nsa/logging.yml'

[cyriltovena]

Hello @Strum355,

The file /home/noah/Netsoc/dev-env/nsa/logging.yml must be available inside the docker driver.

I think you can do that by installing the driver as deamon or simply rebuild the driver with your own Dockerfile that adds this file to its filesystem.

[Strum355]

Are there instructions for installing as a daemon?

[Strum355]

> sudo cat /lib/systemd/system/loki.service
[Unit]
Description=Loki Logging Driver
Before=docker.service
After=network.target loki.socket
Requires=loki.socket docker.service

[Service]
ExecStart=/home/noah/Go/bin/loki-docker-driver

[Install]
WantedBy=multi-user.target

> sudo cat /lib/systemd/system/loki.socket
[Unit]
Description=Loki Logging Driver

[Socket]
ListenStream=/run/docker/plugins/loki.sock

[Install]
WantedBy=sockets.target

> journalctl -u loki
-- Logs begin at Sat 2018-10-06 01:00:19 IST, end at Mon 2019-06-24 23:08:45 IST. --
Jun 24 22:41:49 noah-Banana systemd[1]: Started Loki Logging Driver.

No plugins are listed with docker plugin ls

[Strum355]

Went down the 'rebuild the driver' route and that got it sorted :)

[raneq]

Hi @Strum355 , I am stuck with the same issue. Can you help me to "rebuild the driver"?

PS: IMO this should be documented at loki's docker-driver wiki page, as this option is listed with no further explanation, and the recommended way to install the plugin is via docker plugin install, not "installing the driver as a daemon" or "rebuilding the driver"

[cyriltovena]

There’s an open issue for that.

[raneq]

@Kuqd You mean #700 , right? I find references between issues very useful, I missed that you linked it here. It's taken me a while for me to find it.

[cyriltovena]

Sorry you are right.

[cyriltovena]

Poke me on slack if you can't wait that I take on #700.

[BouchaaraAdil]

i managed to create my own plugin locally, using make build-plugin:
1- i cloned loki repository inside $HOME/go/src/github.com/
2- i have created a pipeline.yml inside cmd/docker-driver/ folder :

cat <<EOF > cmd/docker-driver/pipeline.yml
pipeline_stages:
- regex:
     expression: '(level|lvl|severity)=(?P<level>\\w+)'
- json:
    expressions:
      level: level
- labels:
    level:
EOF

3- i added pipeline.yml in Dockerfile COPY ./pipeline.yaml /etc/pipeline.yml
4- finally make build-plugin
then

  hello-world:
    image: hello-world
    container_name: hello
    logging:
      driver: "grafana/loki-docker-driver:master-cb7c8f6-WIP"
      options:
        loki-url: "http://localhost:3100/api/prom/push"
        loki-retries: "5"
        loki-external-labels: "app={{.Name}},env=dev"
        loki-pipeline-stage-file: /etc/pipeline.yml

[Strum355]

Problem with this approach is that you have to rebuild the image every time you want to add something. If you build the driver with a mount point specified you can add more pipelines by just putting them in the mounted folder

[BouchaaraAdil]

yes, it takes a lot of time, what i suggest is that you don't need to rebuild it, just go to /var/lib/docker/plugins/, and under your plugin id, add or update your pipelines, then add the path to docker containers via loki-pipeline-stage-file, i test it and it working, but it's not ideal

[wapmorgan]

What about inlining loki-pipeline-stage in docker-compose?