External-dns: Unable to decode an event from the watch stream

[richstokes]

Using Image: registry.opensource.zalan.do/teapot/external-dns:latest

I have started getting the below errors. I am not sure anything has changed, other than the pod restarted so it will have pulled the latest version. Maybe there have been breaking changes? Any suggestions?

Running on AWS/kops.

kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-26T00:04:52Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6", GitCommit:"b1d75deca493a24a2f87eb1efde1a569e52fc8d9", GitTreeState:"clean", BuildDate:"2018-12-16T04:30:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

ERROR: logging before flag.Parse: E0419 22:58:35.974014       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0419 22:58:35.974232       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0419 22:58:35.974304       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0419 22:58:36.932700       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0419 22:58:36.932821       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0419 22:58:36.971535       1 reflector.go:322] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:99: Failed to watch *v1.Node: unknown (get nodes)
ERROR: logging before flag.Parse: E0419 22:58:37.976358       1 reflector.go:322] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:99: Failed to watch *v1.Node: unknown (get nodes)
time="2019-04-19T22:58:38Z" level=info msg="All records are already up to date"
ERROR: logging before flag.Parse: E0419 22:58:38.981814       1 reflector.go:322] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:99: Failed to watch *v1.Node: unknown (get nodes

[njuettner]

@richstokes

I think the scope of nodes need to be adjusted:

Could you please try to modify RBAC of external-dns:

...
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list", "get", "watch"]
...

[njuettner]

This will only fix Failed to watch ....
Those other log lines are coming from K8s, see my last comment:
https://github.com/kubernetes-incubator/external-dns/issues/980#issuecomment-485082551

I will create a PR for that.

[richstokes]

@njuettner - I updated that RBAC, now getting output like this:

ERROR: logging before flag.Parse: E0421 23:47:06.073391       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:47:06.073535       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
time="2019-04-21T23:47:07Z" level=info msg="All records are already up to date"
ERROR: logging before flag.Parse: E0421 23:47:35.107802       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:47:35.107952       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0421 23:47:35.108677       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:47:35.108755       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0421 23:47:35.125188       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:47:35.125298       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0421 23:47:36.073960       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:47:36.074088       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0421 23:48:05.108242       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:48:05.108380       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0421 23:48:05.108988       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:48:05.109056       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0421 23:48:05.125603       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:48:05.125710       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
ERROR: logging before flag.Parse: E0421 23:48:06.074370       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:48:06.074510       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)
time="2019-04-21T23:48:07Z" level=info msg="Desired change: CREATE tester22.xxxxx.com A"
time="2019-04-21T23:48:07Z" level=info msg="Desired change: CREATE tester22.xxxxx.com TXT"
time="2019-04-21T23:48:08Z" level=info msg="2 record(s) in zone xxxxx.com. were successfully updated"
ERROR: logging before flag.Parse: E0421 23:48:35.108678       1 round_trippers.go:291] CancelRequest not implemented
ERROR: logging before flag.Parse: E0421 23:48:35.108811       1 streamwatcher.go:109] Unable to decode an event from the watch stream: net/http: request canceled (Client.Timeout exceeded while reading body)

Looks like its working, just spamming a bunch of junk the process. Is this stable/safe to use in production do you think? (Currently I only have it deployed on our test cluster).

[njuettner]

It will be will fixed with this PR https://github.com/kubernetes-incubator/external-dns/pull/991

[njuettner]

@richstokes Did you try to use the latest version of ExternalDNS? With v0.5.14 this should be fixed. Let me know if I can close the issue.

[richstokes]

@njuettner - just grabbed the latest build. All seems good! Thank you