External-storage: [efs-provisioner] EFS volumes causes pods to become stuck in terminating status

[rishka]

I am using the EFS provisioner to add persistent storage for some postgres pods.

When they shut down the pods get stuck in terminating status. These pods would shut down properly before I added the EFS volume.

I can leave them there to troubleshoot, and can be reached on the k8s slack as @adam_p

[wongma7]

Can you please reproduce this and get the output of kubectl describe pod? And possibly kubelet logs? If unmount is failing, events like 'Operation failed for volume...' with relevant information should show up as events on the pods stuck terminating.

[rishka]

kubectl get pod pg-2252090818-74k53

NAME                                          READY     STATUS        RESTARTS   AGE
pg-2252090818-74k53   0/1       Terminating   0          5d

kubectl describe pod pg-2252090818-74k53

Name:               pg-2252090818-74k53
Node:               ...
Start Time:         Tue, 18 Jul 2017 17:10:23 -0400
Labels:             app=review-pg
                pod-template-hash=2252090818
Annotations:            kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"review","name":"pg-2252090818","uid":"7b126ba4-6bfa-11e7-...
Status:             Terminating (expires Fri, 21 Jul 2017 16:24:47 -0400)
Termination Grace Period:   30s
IP:
Controllers:            ReplicaSet/pg-2252090818
Containers:
  postgres:
    Container ID:   docker://3fa04db56ef9313115c6c287f04cfb76df8449e23b5d297675bec940f28f8c7c
    Image:      postgres:9.6.1
    Image ID:       docker-pullable://postgres@sha256:0842a7ef786aa2658623085160cb38451eb3d40856e7d222ae0069b6e6296877
    Port:       5432/TCP
    State:      Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:      False
    Restart Count:  0
    Environment:
      PGDATA:   /var/lib/postgresql/data/pgdata
    Mounts:
      /var/lib/postgresql/data/pgdata from pgdata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-chg2r (ro)
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  pgdata:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc
    ReadOnly:   false
  default-token-chg2r:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-chg2r
    Optional:   false
QoS Class:  BestEffort
Node-Selectors: <none>
Tolerations:    node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:     <none>

[wongma7]

No events for 5 days :( will have to see kubelet logs, can grep them for the pod name and/or pvc name, or send them in full. I'm on slack as @mawong

[wongma7]

In the logs we found unmount failed with 'stale file handle.' I think the issue is the PVC and pod are deleted at the same time (helm delete) and this usually means the PVC directory gets deleted before pod Unmount goes through, so pod Unmount fails indefinitely on a folder that no longer exists hence 'stale file handle.'

the workaround is to always delete the pod before the pvc.

Since helm is pretty official and the idea of cleaning up pods+PVCs at the same time isn't unique to it either, a better solution is needed...the best one IMO is to wait for https://github.com/kubernetes/kubernetes/pull/46573 but something cheaper could be done on the provisioner's end too, need to think about it.

[szesch]

I just ran into this. A PVC and a deploy are defined in the same file. kubectl delete -f deploy.yaml results in the pods never terminating. It doesn't look like there's much traction on that kubernetes ticket. Any suggestions for a workaround?

[rishka]

kubectl delete po --grace-period=0 --force POD_WITH_PVC for the stuck ones is how I am handling it

[wongma7]

@szesch instead of deleting by file (-f) you can delete by pod name then pvc name.

also the ticket is scheduled for fix in 1.9, so only a little longer.

[djdevin]

Using OpenShift and also seeing this on project deletion. It likely deletes the PVC before the pods.

[wstrange]

Just ran into this with the nfs-provisioner as well. Kube 1.9.7 / GKE

[geerlingguy]

And with nfs-client on AWS EKS (though harder to get to kubelet logs on that platform...).

After digging into the kubelet logs, though, I found that the problem in my case was stale NFS file handle, and then found https://github.com/kubernetes/kubernetes/issues/71584, so this is a bug with the core kubernetes nfs support, not nfs-client, sorry about the noise!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.