Longhorn: [FEATURE] Improve node failure handling

[rsoika]

Just for information: I read this problem description and tried it out.
Deleting the volumeattachment

$ kubectl delete volumeattachment csi-08d9842e.......

This leads into a situation where the volume can be attached to the new pod and the pod starts. Ok still the old pod is listed in 'terminating' status.

Maybe there is a way to delete the volumeattachment in such a scenario after a while automatically?

[yasker]

Thanks @rsoika . If deleting the volumeattachment works in this scenario, we can do that. It's better than deleting a pod from Longhorn. We will investigate.

[rsoika]

I now reproduced the same behavior with a ceph cluster (octopus release). I think this is a general kubernetes problem and not a problem from longhorn.

I try to describe the problem now once again just for documentation (this link is no longer working).

Environment:

• Kubernetes Cluster 1.18.1.
• 3 worker nodes (node1,node2,node3)

How to reproduce:

1. POD (e.g. a posgress dabase) is deployed on node1. The POD defies a PersistentVolumeClaim for storage (hosted on longhorn)
2. Hard shutdown from node1
3. Kubernetes (after 5 minutes) recognize lost node and starts a rescheduling of the POD to node2
4. POD on node2 does not start because the volume is still attached to the terminating POD on node1 which is in a durable terminating status

Solution: manually delete the volumeattachment or kill the terminating pod.

[yasker]

Thanks for the update @rsoika . I've updated the doc link (since we've moved our docs to https://longhorn.io/docs/).

[yasker]

Btw, we're totally aiming to solve this in the Longhorn's framework, if Kubernetes cannot solve it by itself.

[rsoika]

It looks like there are discussions since nearly 2 years about this issue:

https://github.com/kubernetes/kubernetes/issues/65392
https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/issues/181

I am not a kubernets expert - just a 'normal' user not understanding enough to give further help. But there still seems to be no solution.

[rsoika]

as you may have seen, I have had continued the discussions about the 'node-fancing' topic in the kubernetes-sigs project. It seems that there will be no solution in kubernetes in the near future. If you find a way to address this in Longhorn, I think this would be a unique feature compared to other solutions.

[yasker]

@rsoika We've decided to add support to auto-detach volume from failed deployment pod in the next release, which will be tracked at https://github.com/longhorn/longhorn/issues/820 , since this scenario is relatively straightforward and doesn't require extra permissions from Kubernetes.

We will start investigating other options after the GA.

[yasker]

We also want to implement the automatically recovery for the stateful set.

[longhorn-io-github-bot]

Pre-merged Checklist

• [x] Does the PR include the explanation for the fix or the feature?

• [x] Is the backend code merged (Manager, Engine, Instance Manager, BackupStore etc)?
  The PR is at https://github.com/longhorn/longhorn-manager/pull/640

• [x] Is the reproduce steps/test steps documented?

• [x] Which areas/issues this PR might have potential impacts on?
  Area restore/recovery. Might have conflicts in a large cluster (> 30 nodes)

• [x] If the fix introduces the code for backward compatibility Has an separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

• [x] If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at

• [x] if labeled: require/doc Has the necessary document PR submitted or merged?
  The Doc issue/PR is at https://github.com/longhorn/website/pull/183

• [x] If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case?
  The automation skeleton PR is at
  The automation test case PR is at

• [x] if labeled: require/automation-engine Has the engine integration test been merged?
  The engine automation PR is at

• [x] if labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan PR is at https://github.com/longhorn/longhorn-tests/pull/395

[PhanLe1010]

The fix for StatefulSet
To reproduce

1. Set up a cluster of 3 nodes.
2. Install Longhorn and set Default Replica Count = 2
3. create a SetfullSet. Ex:

kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/statefulset.yaml

1. Find the node which contains one pod of the StatefullSet. Power off the node
2. After about 5 minutes, checking kubectl shows that the pod stuck in terminating state forever.

[yasker]

What are the validation steps?

[PhanLe1010]

It is in the manual test plan here https://github.com/longhorn/longhorn-tests/wiki/Manual-Test-Plan#improve-node-failure-handling

I will modify it and create a PR to move it to longhorn-test repo after we agree on the discussion https://github.com/longhorn/longhorn-manager/pull/640#issuecomment-670236555

[PhanLe1010]

I updated manual test plan at the PR https://github.com/longhorn/longhorn-tests/pull/395

[yasker]

This issue has been in Review for 21 days. What's blocking this issue moving to Merged?

[PhanLe1010]

I think this one should be good to merge. I just re-requested reviews from Joshua and Shuo.

[khushboo-rancher]

Verified on Longhorn master - 08-27-2020

Validation - Pass

Scenarios Tested:

1. The default value for setting Pod Deletion Policy When Node is Down is do-nothing
2. The setting doesn't take empty value.
3. The setting takes input do-nothing, delete-statefulset-pod, delete-deployment-pod and delete-both-statefulset-and-deployment-pod, anything other than these inputs throws an error.

level=error msg="Error in request: fail to set settings with invalid node-down-pod-deletion-policy: value dummy of settings node-down-pod-deletion-policy is invalid: value dummy is not a valid choice, available choices [do-nothing delete-statefulset-pod delete-deployment-pod delete-both-statefulset-and-deployment-pod]"

1. The test cases from https://longhorn.github.io/longhorn-tests/manual/pre-release/node/improve-node-failure-handling/
2. Pods with scaling/upgrade policy as Start upgraded pods only when old ones are manually deleted, Rolling: stop old pods, then start new and Kill ALL pods, then start new. With all these policies, pods behavior is intact as mentioned https://longhorn.github.io/longhorn-tests/manual/pre-release/node/improve-node-failure-handling/
3. No impact on pods don't have longhorn volume attached.