Loki: Promtail will not start when running in docker on a Raspberry Pi 4

[jongyllen]

Describe the bug
Promtail will not start when running in docker on a Raspberry Pi 4

To Reproduce
Run the Loki docker image on armv7 (in this case a Raspberry pi 4)
```bash
$ wget https://raw.githubusercontent.com/grafana/loki/v1.3.0/production/docker-compose.yaml -o docker-compose.yaml
$ docker-compose -f docker-compose.yaml up
````
Expected behavior
Loki, Grafana and Promtail should be up and running

Environment:
Raspberry Pi 4, running Raspbian Buster (armv7)
Docker

Screenshots, Promtail config, or terminal output
Got error message from Promtail saying it could not find file or directory.
After that Promtail exits

Additional information:
Running Promtail manually inside the container yields the same error,
it seems that /lib/ld-linux-armhf.so.3 it is missing.
Doing ln -s /lib/ld-linux.so.3 /lib/ld-linux-armhf.so.3 inside the container fixes the issue temporarily and Promtail can be run.

[HaveFun83]

Same issue on raspberry 4 k8s node:
promtail pod crashloop with the following error

standard_init_linux.go:211: exec user process caused "no such file or directory"

[REBELinBLUE]

Yep same, been working fine previously

[billimek]

For what it's worth, this is not an issue when running on an arm64-based pi4 with containerd (via k3s v1.17.0+k3s.1):

➜ k get nodes -o=wide
NAME    STATUS   ROLES    AGE     VERSION         INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION      CONTAINER-RUNTIME
k3s-2   Ready    worker   106d    v1.17.0+k3s.1   10.2.0.32     <none>        Ubuntu 19.10   5.3.0-26-generic    containerd://1.3.0-k3s.5
k8s-4   Ready    worker   106d    v1.17.0+k3s.1   10.2.0.16     <none>        Ubuntu 19.10   5.3.0-26-generic    containerd://1.3.0-k3s.5
k3s-1   Ready    worker   106d    v1.17.0+k3s.1   10.2.0.31     <none>        Ubuntu 19.10   5.3.0-26-generic    containerd://1.3.0-k3s.5
pi4-b   Ready    <none>   7d20h   v1.17.0+k3s.1   10.2.0.73     <none>        Ubuntu 19.10   5.3.0-1015-raspi2   containerd://1.3.0-k3s.5
pi4-c   Ready    <none>   38d     v1.17.0+k3s.1   10.2.0.69     <none>        Ubuntu 19.10   5.3.0-1015-raspi2   containerd://1.3.0-k3s.5
k3s-0   Ready    master   106d    v1.17.0+k3s.1   10.2.0.30     <none>        Ubuntu 19.10   5.3.0-26-generic    containerd://1.3.0-k3s.5
pi4-a   Ready    <none>   36d     v1.17.0+k3s.1   10.2.0.17     <none>        Ubuntu 19.10   5.3.0-1015-raspi2   containerd://1.3.0-k3s.5

➜ k -n logs get pods -o=wide
NAME                  READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
loki-promtail-rnz7j   1/1     Running   0          9m42s   10.42.1.212   k3s-1   <none>           <none>
loki-promtail-9hxf9   1/1     Running   0          8m46s   10.42.3.108   k8s-4   <none>           <none>
loki-promtail-82hws   1/1     Running   0          7m39s   10.42.6.135   pi4-c   <none>           <none>
loki-promtail-hc28l   1/1     Running   0          6m26s   10.42.9.77    pi4-a   <none>           <none>
loki-promtail-xvtwm   1/1     Running   0          5m14s   10.42.2.236   k3s-2   <none>           <none>
loki-promtail-9f5kz   1/1     Running   0          4m21s   10.42.4.51    pi4-b   <none>           <none>
loki-promtail-t8dqv   1/1     Running   0          3m10s   10.42.0.45    k3s-0   <none>           <none>
loki-0                1/1     Running   0          107s    10.42.2.237   k3s-2   <none>           <none>

➜ k get pods -n logs -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'

loki-promtail-rnz7j:    grafana/promtail:v1.3.0,
loki-promtail-9hxf9:    grafana/promtail:v1.3.0,
loki-promtail-82hws:    grafana/promtail:v1.3.0,
loki-promtail-hc28l:    grafana/promtail:v1.3.0,
loki-promtail-xvtwm:    grafana/promtail:v1.3.0,
loki-promtail-9f5kz:    grafana/promtail:v1.3.0,
loki-promtail-t8dqv:    grafana/promtail:v1.3.0,
loki-0: grafana/loki:v1.3.0, 

[varac]

Same problem here with a Raspberry Pi 2 Model B Rev 1.1:

root@srv-dashboard:/tmp# docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:v1.3.0 -config.file=/mnt/config/promtail-config.yaml
standard_init_linux.go:207: exec user process caused "no such file or directory"

[HaveFun83]

@slim-bean any news with the issue here?

[khaines]

This is also occurring on Pi 3 Model B+ k8s nodes (using docker)

[cyriltovena]

@rfratto not sure if you have time to also look at this, I think @slim-bean has been very busy lately.

[rfratto]

I'm planning on taking a look at this sometime this week. I bought a Raspberry Pi 4 to investigate this a few weeks ago and installed a 64-bit OS to reproduce. I hadn't read this issue and all the comments, so I didn't realize this was only happening on 32-bit installs. I need to install a different image and try again.

[slim-bean]

@rfratto and I spent most of the day yesterday looking at this, and it's still not entirely clear why this is happening. From what best we can tell the issue lies in the arm image being used by drone to build the image. Looking at the drone logs, it looks like they are using an arm v8 architecture image for running what they describe as 32bit arm builds. It's hard to get much detail on this from their docs.

Ultimately you can run the build on a raspberry pi 4 directly and it creates a working image with the necessary /lib/ld-linux-armhf.so.3 symlink in place. In fact this link exists in the debian stretch-slim base image when pulled from docker. So something in the docker build results in this link being removed, but this only happens on the drone build server and not a ras pi 4.

Very confusing, our best guess here is that maybe one of the triggers of an installed package is running ldconfig and resulting in the symlink being removed.

At any rate, #1740 is a workaround for this which at least makes a working 32bit arm build.

I think the real solution is probably going to be to switch back to using dockers buildkit to cross compile the images, this is slower, but given we lack control over the very specific machines drone is providing for arm builds, I don't really see an alternative (unless we setup our own runners and figure out where to run some armv7 machines).

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[slim-bean]

As a quick update, Loki 1.4.1 has this fix, and promtail should hopefully start properly on 32bit arm builds