Thanos: sidecar: reloader fails on ConfigMap mounts for rules

[clmssz]

Kubernetes ConfigMap volumes contain a .data directory, eg:

/ # ls -la /etc/prometheus
total 12
drwxrwxrwx    3 root     root          4096 May 24 08:03 .
drwxr-xr-x    6 root     root          4096 May 24 08:03 ..
drwxr-xr-x    2 root     root          4096 May 24 08:03 ..2018_05_24_08_03_36.264564876
lrwxrwxrwx    1 root     root            31 May 24 08:03 ..data -> ..2018_05_24_08_03_36.264564876
lrwxrwxrwx    1 root     root            26 May 24 08:03 prometheus.tmpl.yml -> ..data/prometheus.tmpl.yml

When using a ConfigMap volume mount as reloader.rule-dir , reloader fails and sidecar crashes :

level=error ts=2018-05-24T11:47:54.472408145Z caller=main.go:147 msg="running command failed" err="build hash: read /etc/prometheus-rules/..data: is a directory"

[bwplotka]

Thanks! More testing needed. PRs welcome (:

[szalai1]

Hi @bwplotka @clmssz,
Has anybody started working on this? If no I would like to.

[FUSAKLA]

Hi @szalai1, I think noone is working on it and I can confirm I see the log all the time.

If you want it would be great to solve this issue! So feel free to create PR :)

[brancz]

we recently wrote some code to handle this, in case someone is interested in transferring the approach: https://github.com/brancz/kube-rbac-proxy/pull/45

[FUSAKLA]

Great thanks for the info! I'll take a look at this if noone will pick it.

[szalai1]

@FUSAKLA I didn't finish the work on this issue. So you can work on it.

[povilasv]

FYI we had the same issue and the workaround is to set - --reloader.rule-dir=/etc/prometheus-rules/..data .

We've been running like that for a while now and it works flawlessly.

[FUSAKLA]

Great, will try it out. Thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[brancz]

I believe this still exists as originally reported.

[daixiang0]

@brancz @FUSAKLA Test latest reloader pkg with following dir and can not reproduce:

[test]# ls -al
total 4
drwxr-xr-x  3 root root   45 1月  15 17:40 .
drwxr-xr-x 16 1000 1000 4096 1月  15 17:18 ..
drwxr-xr-x  2 root root    6 1月  15 17:19 ..aaa
lrwxrwxrwx  1 root root    5 1月  15 17:19 ..data -> ..aaa

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.