External-storage: [nfs-server-provisioner] container restarts when api server is failing

Created on 19 Jun 2019 · 22Comments · Source: kubernetes-retired/external-storage

I posted https://github.com/helm/charts/issues/14912 to the helm charts repo, but this might be a better place as it is here that the quay.io/kubernetes_incubator/nfs-provisioner image is maintained.

The following is a copy/paste of the issue.

Describe the bug

The container restarts frequently when the api server is not responding but it should not have to do it.

Version of Helm and Kubernetes:

helm 2.14.1
kubernetes 1.12.7-1

Which chart:

latest stable/nfs-server-provisioner

What happened:

I have a cluster with latency and sometimes failure issues of the api server. I have submitted a bug to my cloud provider that manages the control plane.
My issue with nfs-server-provisioner is that during periods of failures of the api server I seel logs like this and the container restarts :

I0619 07:05:11.965197       1 leaderelection.go:231] failed to renew lease nfs/cluster.local-nfs-data-nfs-server-provisioner: failed to tryAcquireOrRenew context deadline exceeded
F0619 07:05:11.966172       1 controller.go:646] leaderelection lost

I have lots of problems with the PVCs based on this provisionner: binding timeouts, lost file writes, etc.
I am not certain but I think it is related to these frequent restarts of the nfs server.

What you expected to happen:

The nfs server should tolerate api server failures.

How to reproduce it (as minimally and precisely as possible):

I do not manage the control plane of my cluster. But I suppose that manually stopping the api server should recreate the issue.

Anything else we need to know:

I understand restarting the container in this case is probably the expected behavior, but I think it is wrong.

replication is not relevant in the case of nfs-server-provisioner and I don't see the point of a leader election
stability issues of the api server should not propagate to a storage provisioner and then to a bunch of volumes used by a bunch of services

Source

albanm

👀2

Most helpful comment

This frequently happens to us as well.

Looks like it Fatals deliberately when it loses the leader lock: https://github.com/kubernetes-incubator/external-storage/blob/efs-provisioner-v2.4.0/vendor/github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/controller/controller.go#L646

Since sig-storage-lib-external-provisioner is a shared library it's probably hard to just change it.

How about we allow to turn off leader election via an environment variable or flag. The code is already there but it's not possible to configure it from the outside. That should be easy to change and the default could be left at true.

linki on 6 May 2020

👍3

All 22 comments

@albanm I'm seeing a very similar issue - which cloud provider are you seeing the latency with?

acejam on 23 Jun 2019

We encountered exactly the same issue on Microsoft AKS.

andysiu on 6 Aug 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 4 Nov 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 4 Dec 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 3 Jan 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot on 3 Jan 2020

Just faced with the same issue

/remove-lifecycle rotten
/reopen

log:

I0218 15:53:25.950633       1 main.go:64] Provisioner cluster.local/nextcloud-nfs-server-provisioner specified
I0218 15:53:25.950676       1 main.go:88] Setting up NFS server!
I0218 15:53:26.069124       1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0218 15:53:26.069142       1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0218 15:53:26.070153       1 server.go:134] Running NFS server!
F0218 15:57:37.202820       1 controller.go:847] leaderelection lost
goroutine 1 [running]:
k8s.io/klog.stacks(0xc000147500, 0xc00055a5a0, 0x45, 0x116)
        /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:883 +0xb8
k8s.io/klog.(*loggingT).output(0x2174a40, 0xc000000003, 0xc000140c40, 0x20ddcb3, 0xd, 0x34f, 0x0)
        /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:834 +0x330
k8s.io/klog.(*loggingT).printf(0x2174a40, 0x3, 0x1556377, 0x13, 0x0, 0x0, 0x0)
        /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:715 +0x14b
k8s.io/klog.Fatalf(...)
        /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:1284
github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/controller.(*ProvisionController).Run.func2()
        /home/fedora/go/pkg/mod/github.com/kubernetes-sigs/[email protected]+incompatible/controller/controller.go:847 +0x5b
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1(0xc0003bc5a0)
        /home/fedora/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:193 +0x40
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc0003bc5a0, 0x176eb40, 0xc0003ce380)
        /home/fedora/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:202 +0x15e
k8s.io/client-go/tools/leaderelection.RunOrDie(0x176eb80, 0xc0000da048, 0x177b640, 0xc0003c12c0, 0x37e11d600, 0x2540be400, 0x77359400, 0xc000411ca0, 0x15e1070, 0x0, ...)
        /home/fedora/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:214 +0x96
github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/controller.(*ProvisionController).Run(0xc0001526c0, 0xc00018c3c0)
        /home/fedora/go/pkg/mod/github.com/kubernetes-sigs/[email protected]+incompatible/controller/controller.go:839 +0x41e

kvaps on 18 Feb 2020

@kvaps: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

k8s-ci-robot on 18 Feb 2020

My case was connected with the network outages https://github.com/cloudnativelabs/kube-router/issues/846, however I think it would be nice to add some option to not fall down when nfs-server-provisioner can't reach the Kubernetes API, or completely disable the leader election

kvaps on 19 Feb 2020

@wongma7 sorry to disturb you but do we have any specific reason to having leader election for nfs-server-provisioner pods, or it was made just because of common library usage?

According the docs it can't be used with more than one replicas:

There must always be one service per pod or provisioning will fail, meaning the deployment cannot be scaled beyond 1 replica. To scale (where multiple instances do leader election), new deployment and service pairs need to be created with new names, matching labels+selectors and SERVICE_NAME variables.

kvaps on 19 Feb 2020

/reopen

linki on 6 May 2020

@linki: Reopened this issue.

In response to this:

/reopen

k8s-ci-robot on 6 May 2020

This frequently happens to us as well.

Since sig-storage-lib-external-provisioner is a shared library it's probably hard to just change it.

linki on 6 May 2020

👍3

Just had this issue. I've been using the provisioner for just a few days. @linki is there a workaround in the meantime to prevent this from happening? Thanks

vitobotta on 26 May 2020

Stuck with this same issue on a new deployment of K3S inside LXC - this is likely a symptom of something else, however I would also like to see the ability to turn off leader election - especially since HA is not a thing here..

$ kubectl logs -f pod/nfs-server-nfs-server-provisioner-0
I0621 04:07:31.175070       1 main.go:64] Provisioner cluster.local/nfs-server-nfs-server-provisioner specified
I0621 04:07:31.175102       1 main.go:88] Setting up NFS server!
I0621 04:07:31.755345       1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0621 04:07:31.755366       1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0621 04:07:31.793518       1 server.go:134] Running NFS server!
F0621 11:31:54.903796       1 controller.go:847] leaderelection lost
goroutine 1 [running]:
k8s.io/klog.stacks(0xc000428700, 0xc000510000, 0x45, 0x118)
    /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:883 +0xb8
k8s.io/klog.(*loggingT).output(0x2174a40, 0xc000000003, 0xc00010c070, 0x20ddcb3, 0xd, 0x34f, 0x0)
    /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:834 +0x330
k8s.io/klog.(*loggingT).printf(0x2174a40, 0x3, 0x1556377, 0x13, 0x0, 0x0, 0x0)
    /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:715 +0x14b
k8s.io/klog.Fatalf(...)
    /home/fedora/go/pkg/mod/k8s.io/[email protected]/klog.go:1284
github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/controller.(*ProvisionController).Run.func2()
    /home/fedora/go/pkg/mod/github.com/kubernetes-sigs/[email protected]+incompatible/controller/controller.go:847 +0x5b
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1(0xc0000e0c60)
    /home/fedora/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:193 +0x40
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc0000e0c60, 0x176eb40, 0xc00016e340)
    /home/fedora/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:202 +0x15e
k8s.io/client-go/tools/leaderelection.RunOrDie(0x176eb80, 0xc000044088, 0x177b640, 0xc00040eb40, 0x37e11d600, 0x2540be400, 0x77359400, 0xc00039ce50, 0x15e1070, 0x0, ...)
    /home/fedora/go/pkg/mod/k8s.io/[email protected]/tools/leaderelection/leaderelection.go:214 +0x96
github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/controller.(*ProvisionController).Run(0xc0000e4d80, 0xc0001843c0)
    /home/fedora/go/pkg/mod/github.com/kubernetes-sigs/[email protected]+incompatible/controller/controller.go:839 +0x41e
main.main()
    /home/fedora/go/src/github.com/kubernetes-incubator/external-storage/nfs/cmd/nfs-provisioner/main.go:143 +0x5d8

Routhinator on 21 Jun 2020

I am also seeing the same issue. After nfs server provisioner is restarted, my application pods are down. Restarting the pod is giving issues that unable to mount volumes. Till this issue is closed, is there any workaround to bring my application pods up and running?
Many thanks in advance

deepayr on 24 Jul 2020

@deepayr do you use IPVS?
Check it out: https://github.com/kubernetes/kubernetes/issues/84322

kvaps on 24 Jul 2020

No. Its on VPC Gen2 IBM Cloud Managed openshift cluster.

deepayr on 24 Jul 2020

I've got the same issue, which results in volume mount errors forcing me to manually kill my pods. Workaround or permanent solution would be great!

mattb290 on 7 Aug 2020

In the duplicate issue here, I discussed my workaround using a fork. I still use this fork without issues but I haven't updated it from upstream since I created it.

albanm on 7 Aug 2020

Thanks for reporting the issue!

This repo is no longer being maintained and we are in the process of archiving this repo. Please see https://github.com/kubernetes/org/issues/1563 for more details.

If your issue relates to nfs provisioners, please create a new issue in https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner or https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.

Going to close this issue in order to archive this repo. Apologies for the churn and thanks for your patience! :pray: