Loki: promtail logs no such file or directory

[matti]

installed with:

curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 1234 asdfasdfasdfasdfasdf= logs-us-west1.grafana.net default | kubectl apply --namespace=default -f  -

promtail logs are getting filled with:

3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log
level=error ts=2019-03-27T09:05:14.8651359Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log
level=error ts=2019-03-27T09:05:24.8647833Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log=error
level=error ts=2019-03-27T09:05:24.864907Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log=error
level=error ts=2019-03-27T09:05:24.8649644Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log
level=error ts=2019-03-27T09:05:24.8650161Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log
level=error ts=2019-03-27T09:05:34.8644351Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log=error
level=error ts=2019-03-27T09:05:34.8645299Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log=error
level=error ts=2019-03-27T09:05:34.8645833Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log
level=error ts=2019-03-27T09:05:34.8646217Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log
level=error ts=2019-03-27T09:05:44.8303715Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log=error
level=error ts=2019-03-27T09:05:44.8304966Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log=error
level=error ts=2019-03-27T09:05:44.8305617Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log
level=error ts=2019-03-27T09:05:44.8306023Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log
level=error ts=2019-03-27T09:05:54.8299935Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log=error
level=error ts=2019-03-27T09:05:54.8300764Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log=error
level=error ts=2019-03-27T09:05:54.8301243Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log
level=error ts=2019-03-27T09:05:54.8301608Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log
level=error ts=2019-03-27T09:06:04.8302938Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log=error
level=error ts=2019-03-27T09:06:04.8304231Z caller=filetarget.go:300 msg="failed to stat matched file, cannot report size" /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log=error
level=error ts=2019-03-27T09:06:04.8304807Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/1.log
level=error ts=2019-03-27T09:06:04.830531Z caller=filetarget.go:247 msg="failed to tail file, stat failed" error="stat /var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log: no such file or directory" filename=/var/log/pods/3e18ac30-4998-11e9-bfbf-025000000001/kubedns/2.log

[daixiang0]

have you customized docker data root? If so, you need add option for it.

usage: ./tools/promtail.sh <instanceId> <apiKey> <url> [<namespace>[<container_root_path>[<parser>]]]

[Serrvosky]

If you are trying that on a Kubernetes Cluster, please check if you don't forget to mount the host logs directory.

      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: promtail-conf
        configMap:
          name: promtail-conf

and

        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: promtail-conf
          mountPath: /etc/promtail

I also had that problem because I forgot varlibdockercontainers mount.

[slim-bean]

I've also seen this locally with docker for mac, it feels like a bug where the symlink in /var/log/pods still exists but points to a file that doesn't exist anymore. I usually reset the kubernetes environment in docker for mac to clean things up.

@matti are you still having this issue? Did either @Serrvosky or my suggestions above help?

[javefang]

I'm also having this problem, it does look like the symlinks in /var/log/pods are not cleaned up after the actual log files under /var/lib/docker/containers have been removed by the docker daemon (our setting keeps 3 log files for each container). This behaviour seems pretty normal and fluent-bit doesn't seem to complain. Maybe it is safe to make promtail ignore this error? (or print it in level=debug)

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[chenpfeisoo]

@Serrvosky ,I had this configation , but the issue still

[chenpfeisoo]

[Serrvosky]

Hello @chenpfeisoo. Actualy, I don't work with promtail for a long time ago, but this was my implementation at the time:

volumes:
      - hostPath:
          path: /var/log
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
        name: varlibdockercontainers
...
volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true

I think, I used this configuration with kubernetes version 1.13. Which version are you using?

[chenpfeisoo]

@Serrvosky Thanks for your reply .The version of my cluster is 1.15.0
I have an other issuse #1329 ,a complete record of my problems

[per-lind]

We have also run into this, it would be nice for a way to supress the error. We are hitting around 1000 errors / s in unnecessary log.

[jumping]

We have similar issues, those logs under /var/log/pods/*/*/*.log are symbol link which's destination are /var/lib/docker/containers/*/*.log.