Client-go: watcher error after 20m

[mheese]

When I create a watch on pods e.g. like this:

import (
...
  "k8s.io/client-go/1.4/kubernetes"
  "k8s.io/client-go/1.4/pkg/api"
  apiv1 "k8s.io/client-go/1.4/pkg/api/v1"
  "k8s.io/client-go/1.4/pkg/labels"
  "k8s.io/client-go/1.4/pkg/selection"
  "k8s.io/client-go/1.4/pkg/util/sets"
  "k8s.io/client-go/1.4/pkg/watch"
  ...
)
...
podsClient := clientset.Core().Pods("default")
watcher, err := podsClient.Watch(api.ListOptions{...})
...
select {
...
  case ev := <-watcher.ResultChan():
  ...
}

I have the trouble that after exactly 20 minutes, the ResultChan() receives infinite messages, but they are all empty. Trying to print the returned ev from the channel shows that it is an empty/uninitialized map map[Type: Object:<nil>].

Is this expected behaviour? Am I supposed to renew the watch after some time?

[caesarxuchao]

It is expected. Server will timeout the watch connection, see code. Though I'm not sure why it's 20 mins, it timeout after 30-60 mins if you didn't change the default configs of your server, because the default MinRequestTimeout is 30 mins.

Trying to print the returned ev from the channel shows that it is an empty/uninitialized map map[Type: Object:]

I guess you have a loop around the select? I think you need to check if watcher.ResultChan() is closed,

 case ev, ok := <-watcher.ResultChan():
    if !ok {
       // break the loop
    }

[caesarxuchao]

You can take a look at this for loop in reflector.go as a reference on how to renew watch.

[mheese]

Thanks @caesarxuchao . I'm giving that a try. I'm 99.999% certain that my problems are solved. Somehow it simply did not occur to me that the channel might be closed at that point, and I actually thought that it would panic otherwise.

With regards to the 20m: I'm testing/developing with minikube which is using localkube I believe? I definitely have not changed any default values. I don't even know how to do that.

[mheese]

I'm going to close this issue as it is not a bug, but expected behaviour.

Additionally, just to point it out again if people are running into the same error/misbehaviour: you have to test if the channel is closed.

So the solution is very simple as @caesarxuchao pointed out:

case ev, ok := <-watcher.ResultChan():
    if !ok {
       // break the loop
    }

Note, that after you recreate the watch, you will receive a lot of ADDED messages for all the pods/objects that match your watch.

Thanks again for the help @caesarxuchao!

[scaszoo]

You could override the timeout in ListOptions

timeout := int64(60 * 60 * 24) // 24 hours
watcher, err := client.CoreV1().Pods("default").Watch(metav1.ListOptions{ TimeoutSeconds: &timeout, })