External-dns: External DNS requires writeable container file system

[hjacobs]

Trying to set securityContext.readOnlyRootFilesystem: true (https://github.com/zalando-incubator/kubernetes-on-aws/pull/1161) fails for External DNS as it sadly requires a writeable file system for logs (?):

zkubectl logs external-dns-669fccc7bd-tjzlc -n kube-system
time="2018-06-14T21:00:06Z" level=info msg="config: {.....}"
W0614 21:00:06.725897       1 client_config.go:517] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
log: exiting because of error: log: cannot create log: open /tmp/external-dns.external-dns-669fccc7bd-tjzlc.unknownuser.log.WARNING.20180614-210006.1: read-only file system

[hjacobs]

Here what's going on in those /tmp files:

$ zkubectl exec -it external-dns-8d54c9b9f-wdd4p -n kube-system /bin/sh
/ $ ls /tmp/
external-dns.INFO                                                                    external-dns.external-dns-8d54c9b9f-wdd4p.unknownuser.log.INFO.20180614-211938.1
external-dns.WARNING                                                                 external-dns.external-dns-8d54c9b9f-wdd4p.unknownuser.log.WARNING.20180614-211938.1
/ $ cat /tmp/external-dns.INFO 
Log file created at: 2018/06/14 21:19:38
Running on machine: external-dns-8d54c9b9f-wdd4p
Binary: Built with gc go1.10.2 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0614 21:19:38.527865       1 client_config.go:517] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
/ $ cat /tmp/external-dns.WARNING 
Log file created at: 2018/06/14 21:19:38
Running on machine: external-dns-8d54c9b9f-wdd4p
Binary: Built with gc go1.10.2 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0614 21:19:38.527865       1 client_config.go:517] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

These files are apparently produced by glog: https://github.com/golang/glog/blob/23def4e6c14b4da8ac2ed8007337bc5eb5007998/glog_file.go :disappointed:

[hjacobs]

client-go seems to be the culprit: https://github.com/kubernetes/client-go/issues/358

[mikkeloscar]

client-go seems to be the culprit: kubernetes/client-go#358

glog really, like always...

[luxalpa]

I can't be the only one who finds it hilarious that the kubernetes client is incompatible with kubernetes.

Anyway, 2 possible workarounds:

1. You can vendor-override your own glog package where you fix that error.

2. There is some command line flags that you can set for glog to disable logging

[Quentin-M]

Are those files even rotated? Or are they going to grow out of control if we never restart external-dns?

[hjacobs]

This was closed accidentally I think --- we now know that the new client-go version no longer uses glog, yeah :smile: see https://github.com/kubernetes/client-go/issues/358#issuecomment-460777391

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.