Tf-operator: tf-job-operator pod restarts

[ljainker]

The tf-job-operator pod restarts.

Logs of the previous instance of the tf-job-operator,

{"filename":"app/server.go:76","level":"info","msg":"EnvKubeflowNamespace not set, use default namespace","time":"2020-06-02T07:42:44Z"} {"filename":"app/server.go:80","level":"info","msg":"Using cluster scoped operator","time":"2020-06-02T07:42:44Z"} {"filename":"app/server.go:86","level":"info","msg":"[API Version: v1 Version: v0.1.0-alpha Git SHA: Not provided. Go Version: go1.13.5 Go OS/Arch: linux/amd64]","time":"2020-06-02T07:42:44Z"} W0602 07:42:44.410868 1 client_config.go:548] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. {"filename":"tf-operator.v1/main.go:42","level":"info","msg":"Setting up client for monitoring on port: 8443","time":"2020-06-02T07:42:44Z"} {"filename":"tensorflow/controller.go:120","level":"info","msg":"Creating TFJob controller","time":"2020-06-02T07:42:44Z"} {"filename":"tensorflow/controller.go:127","level":"info","msg":"Creating Job controller","time":"2020-06-02T07:42:44Z"} I0602 07:42:44.431328 1 leaderelection.go:187] attempting to acquire leader lease default/tf-operator... I0602 07:42:44.468134 1 leaderelection.go:196] successfully acquired lease default/tf-operator {"filename":"tensorflow/controller.go:187","level":"info","msg":"Starting TFJob controller","time":"2020-06-02T07:42:44Z"} {"filename":"tensorflow/controller.go:190","level":"info","msg":"Waiting for informer caches to sync","time":"2020-06-02T07:42:44Z"} {"filename":"tensorflow/controller.go:196","level":"info","msg":"Starting 1 workers","time":"2020-06-02T07:42:44Z"} {"filename":"tensorflow/controller.go:202","level":"info","msg":"Started workers","time":"2020-06-02T07:42:44Z"} I0602 09:46:14.926830 1 leaderelection.go:231] failed to renew lease default/tf-operator: failed to tryAcquireOrRenew context deadline exceeded {"filename":"app/server.go:179","level":"fatal","msg":"leader election lost","time":"2020-06-02T09:46:14Z"}

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| kind/bug | 0.79 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[fulei-9]

I have the same issues, tf opertor installed by kfctl, kubeflow version 0.71 and 1.0 also have the same issues

[johnugeorge]

@fulei-9 @ljainker Are you consistently seeing it?

/cc @gaocegege

[gaocegege]

I will have a look.

[gaocegege]

Maybe you could have a look at https://github.com/kubedb/project/issues/581#issuecomment-629592937

The problem is

I0602 09:46:14.926830 1 leaderelection.go:231] failed to renew lease default/tf-operator: failed to tryAcquireOrRenew context deadline exceeded

Your lease cannot be renewed, I think it may be related to network.

[fulei-9]

I0607 18:54:56.730451 1 leaderelection.go:231] failed to renew lease default/pytorch-operator: failed to tryAcquireOrRenew context deadline exceeded
{"filename":"app/server.go:168","level":"fatal","msg":"leader election lost","time":"2020-06-07T18:54:56Z"}

[fulei-9]

the operator can work,but pod always restart

[ljainker]

Hi @johnugeorge

Even I am facing the same issue, the pod gets restarted with the below error

I0608 07:26:53.478306 1 leaderelection.go:231] failed to renew lease default/tf-operator: failed to tryAcquireOrRenew context deadline exceeded
{"filename":"app/server.go:179","level":"fatal","msg":"leader election lost","time":"2020-06-08T07:26:53Z"}

[gaocegege]

I think it is caused by the network. Where are you running Kubeflow?

[ljainker]

Hi @gaocegege,

I am running Kubeflow on Kubernetes cluster.

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

[gaocegege]

Yeah I know. I was asking the situation of the kubernetes cluster. Is it running locally in your laptop or in a data center or in a public cloud?

[fulei-9]

my cluster in data center

[ljainker]

Mine is on-premise, installed on Openstack platform

[ljainker]

Hi @gaocegege ,

As mentioned in https://github.com/SeldonIO/seldon-core/issues/1910, this issue is caused by network issues similar to what is mentioned in https://github.com/kubernetes/kubernetes/issues/74340.
We are checking with our internal team who can support us with network issue.

Thank you very much for the support :)

I will now be closing this ticket.

[gaocegege]

If you have any progress or problem ,do not hesitate to post here. Thanks for your issue.