Tf-operator: Pod stuck in unknown status when kubernetes node is down

[chaoleili]

I run tfjobs on kubernets using tf-operator. I found that when kubernetes worker node down or node in NotReady status. the worker or ps pod of tfjobs will be stuck in unknown status and will not rebuild. So the entire tensorflow train job will block until the worker node Ready again. So, should we consider using ReplicaSet or Deployment to create worker and ps instead of just using Pod?

[gaocegege]

We have some discussions about it while we decided to used pod because we need to support some custom restart policy, I think we should handle it in the operator.

[chaoleili]

How about adding a reschedule function in the operator when the pod in the unknown status?

[gaocegege]

reschedule is not the scope of the operator, maybe we could discuss with @k82cn

[gaocegege]

Ref https://github.com/kubernetes/kubernetes/issues/55713

It seems a bug in Kubernetes kubelet.

[chaoleili]

:) Thanks for your ref.

[k82cn]

How about adding a reschedule function in the operator when the pod in the unknown status?

IMO, NotReady is a common state, and should be handled by k8s; in tf-operator, it already has restart policy which are framework related :)

[chaoleili]

I got that, thanks for your reply ！:)

[gaocegege]

Then I am closing the issue