Tf-operator: TFJob and Worker Pods are not deleted after the Job has completed

[asahalyft]

Hello Team - After applying the tf-operator and successfully running a Distributed TFJob. I see the worker pods still hanging around in the Completed state and the TFJob is also not deleted. I cannot spawn a new Job with the same name unless I manually cleanup all of them.

e.g. After the job has completed I do the following
kubectl describe tfjob tf2-keras-mnist-tfjob-gpu -n asaha and I still see the tfjob hanging around even after nearly am hour.

TFJob Status:

Status:
  Completion Time:  2021-02-09T01:27:16Z
  Conditions:
    Last Transition Time:  2021-02-09T01:26:48Z
    Last Update Time:      2021-02-09T01:26:48Z
    Message:               TFJob tf2-keras-mnist-tfjob-gpu is created.
    Reason:                TFJobCreated
    Status:                True
    Type:                  Created
    Last Transition Time:  2021-02-09T01:26:51Z
    Last Update Time:      2021-02-09T01:26:51Z
    Message:               TFJob tf2-keras-mnist-tfjob-gpu is running.
    Reason:                TFJobRunning
    Status:                False
    Type:                  Running
    Last Transition Time:  2021-02-09T01:27:16Z
    Last Update Time:      2021-02-09T01:27:16Z
    Message:               TFJob tf2-keras-mnist-tfjob-gpu successfully completed.
    Reason:                TFJobSucceeded
    Status:                True
    Type:                  Succeeded
  Replica Statuses:
    Worker:
      Succeeded:  2
  Start Time:     2021-02-09T01:26:48Z
Events:
  Type    Reason                   Age                    From         Message
  ----    ------                   ----                   ----         -------
  Normal  SuccessfulCreatePod      5m43s                  tf-operator  Created pod: tf2-keras-mnist-tfjob-gpu-worker-0
  Normal  SuccessfulCreatePod      5m43s                  tf-operator  Created pod: tf2-keras-mnist-tfjob-gpu-worker-1
  Normal  SuccessfulCreateService  5m43s                  tf-operator  Created service: tf2-keras-mnist-tfjob-gpu-worker-0
  Normal  SuccessfulCreateService  5m43s                  tf-operator  Created service: tf2-keras-mnist-tfjob-gpu-worker-1
  Normal  ExitedWithCode           5m15s (x3 over 5m15s)  tf-operator  Pod: asaha.tf2-keras-mnist-tfjob-gpu-worker-1 exited with code 0
  Normal  ExitedWithCode           5m15s                  tf-operator  Pod: asaha.tf2-keras-mnist-tfjob-gpu-worker-0 exited with code 0
  Normal  TFJobSucceeded           5m15s                  tf-operator  TFJob tf2-keras-mnist-tfjob-gpu successfully completed.

Also, the worker pods are still lurking around.

TFJob Worker Pod Status:

(base) asaha-mbp151:exploration asaha$ 
(base) asaha-mbp151:exploration asaha$ kubectl get pods -n asaha | grep tfjob
tf2-keras-mnist-tfjob-gpu-worker-0                 0/1     Completed   0          5m14s
tf2-keras-mnist-tfjob-gpu-worker-1                 0/1     Completed   0          5m14s
(base) asaha-mbp151:exploration asaha$ 

For completeness I applied the following TFJob Yaml.

# https://github.com/kubeflow/tf-operator/blob/master/examples/v1/distribution_strategy/keras-API/multi_worker_strategy-with-keras.py
# ------- Create GPU Distributed TFJob ------ #
apiVersion: kubeflow.org/v1
kind: TFJob
metadata:
  name: tf2-keras-mnist-tfjob-gpu
spec:
  cleanPodPolicy: All
  tfReplicaSpecs:
    Worker:
      replicas: 2
      restartPolicy: Never
      template:
        metadata:
          labels:
            lyft.com/ml-platform: ""   
        spec:
          containers:
            - name: tensorflow
              image: "173840052742.dkr.ecr.us-east-1.amazonaws.com/lyftlearnhorovod:17d83c623429a793a9c526f4d98550c70156961f"
              command: 
                - python
              args: 
                - "/mnt/user-home/mpi-horovod-exploration/multi_worker_strategy-with-keras.py"
                - "--saved_model_dir"
                - "/tmp/tfjob-mnist/saved_model/"
                - "--checkpoint_dir"
                - "/tmp/tfjob-mnist/checkpoint"
              env:
              - name: NCCL_DEBUG
                value: "INFO"
              - name: NCCL_SOCKET_IFNAME
                value: "eth0"
              resources:
                limits:
                  nvidia.com/gpu: 1
              volumeMounts:
              - mountPath: /mnt/user-home
                name: nfs
          volumes:
          - name: nfs
            persistentVolumeClaim:
              claimName: asaha
          tolerations: 
            - key: lyft.net/gpu
              operator: Equal
              value: dedicated
              effect: NoSchedule

TF Operator Logs:

(base) asaha-mbp151:exploration asaha$ kubectl logs -f tf-operator-54dbf7666-r9qz4 -n tf-operator

{"filename":"tensorflow/status.go:216","job":"asaha.tf2-keras-mnist-tfjob-gpu","level":"info","msg":"Finished updating TFJobs Status \"tf2-keras-mnist-tfjob-gpu\" (6.430419ms)","time":"2021-02-09T01:27:16Z","uid":"b14f519e-1105-4a51-98d0-46e50b128d5a"}
{"filename":"tensorflow/controller.go:294","job":"asaha.tf2-keras-mnist-tfjob-gpu","level":"info","msg":"Finished syncing tfjob \"asaha/tf2-keras-mnist-tfjob-gpu\" (6.845854ms)","time":"2021-02-09T01:27:16Z"}
{"filename":"tensorflow/job.go:152","level":"info","msg":"Updating tfjob: tf2-keras-mnist-tfjob-gpu","time":"2021-02-09T01:27:16Z"}
{"filename":"common/job.go:92","level":"info","msg":"Reconciling for job tf2-keras-mnist-tfjob-gpu","time":"2021-02-09T01:27:16Z"}
{"filename":"k8sutil/k8sutil.go:102","level":"info","msg":"Ignoring inactive pod asaha/tf2-keras-mnist-tfjob-gpu-worker-1 in state Succeeded, deletion time \u003cnil\u003e","time":"2021-02-09T01:27:16Z"}
{"filename":"tensorflow/pod.go:130","job":"asaha.tf2-keras-mnist-tfjob-gpu","level":"info","msg":"Pod: asaha.tf2-keras-mnist-tfjob-gpu-worker-1 exited with code 0","time":"2021-02-09T01:27:16Z","uid":"b14f519e-1105-4a51-98d0-46e50b128d5a"}
{"filename":"tensorflow/status.go:108","job":"asaha.tf2-keras-mnist-tfjob-gpu","level":"info","msg":"TFJob=tf2-keras-mnist-tfjob-gpu, ReplicaType=Worker expected=1, running=1, failed=0","time":"2021-02-09T01:27:16Z","uid":"b14f519e-1105-4a51-98d0-46e50b128d5a"}
{"filename":"tensorflow/controller.go:294","job":"asaha.tf2-keras-mnist-tfjob-gpu","level":"info","msg":"Finished syncing tfjob \"asaha/tf2-keras-mnist-tfjob-gpu\" (305.397µs)","time":"2021-02-09T01:27:16Z"}
...
...
{"filename":"tensorflow/job.go:152","level":"info","msg":"Updating tfjob: tf2-keras-mnist-tfjob-gpu","time":"2021-02-09T01:34:12Z"}
{"filename":"common/job.go:92","level":"info","msg":"Reconciling for job tf2-keras-mnist-tfjob-gpu","time":"2021-02-09T01:34:12Z"}
{"filename":"tensorflow/controller.go:294","job":"asaha.tf2-keras-mnist-tfjob-gpu","level":"info","msg":"Finished syncing tfjob \"asaha/tf2-keras-mnist-tfjob-gpu\" (217.145µs)","time":"2021-02-09T01:34:12Z"}
{"filename":"tensorflow/job.go:152","level":"info","msg":"Updating tfjob: tf2-keras-mnist-tfjob-gpu","time":"2021-02-09T01:34:42Z"}
{"filename":"common/job.go:92","level":"info","msg":"Reconciling for job tf2-keras-mnist-tfjob-gpu","time":"2021-02-09T01:34:42Z"}
{"filename":"tensorflow/controller.go:294","job":"asaha.tf2-keras-mnist-tfjob-gpu","level":"info","msg":"Finished syncing tfjob \"asaha/tf2-keras-mnist-tfjob-gpu\" (248.45µs)","time":"2021-02-09T01:34:42Z"}

[gaocegege]

Thanks for the issue! I will have a look ASAP.

[gaocegege]

Hi, which API version are you using? Maybe you need to set cleanPodPolicy like this if you are using v1:

spec:
  runPolicy:
    cleanPodPolicy: All

[gaocegege]

BTW, would you mind adding your adoption to https://github.com/kubeflow/tf-operator/blob/master/docs/adopters.md?

[asahalyft]

I see. Thanks @gaocegege that worked. There is a difference between how tf-operator and the mpi-operator expect the cleanPodPolicy to be specified.

I had been using below with tf-operator which is what we also use with the mpi-operator

spec:
  cleanPodPolicy: All

But after changing to

spec:
  runPolicy:
    cleanPodPolicy: All

I see the tfjob worker pods are now being cleaned up. But, the TFJob still hangs around. Is that designed like that? Do, we need to clean up the TFJob manually?

Sure, I will be happy to update the adoption once we finish our evaluation and testing and incorporate TFJob in our production system.

[terrytangyuan]

Do you mind adding your organization to MPI operator's adopters as well? https://github.com/kubeflow/mpi-operator/blob/master/ADOPTERS.md

[asahalyft]

Do you mind adding your organization to MPI operator's adopters as well? https://github.com/kubeflow/mpi-operator/blob/master/ADOPTERS.md

Sure, I will be happy to update the adoption once we finish our evaluation and testing and incorporate MPI Operator in our production system.

[gaocegege]

Sure, I will be happy to update the adoption once we finish our evaluation and testing and incorporate TFJob in our production system.

Sure, thanks.

I had been using below with tf-operator which is what we also use with the mpi-operator

https://github.com/kubeflow/mpi-operator/issues/324 I am working on it. I think we will have the same layout with MPIJob.

I see the tfjob worker pods are now being cleaned up. But, the TFJob still hangs around. Is that designed like that? Do, we need to clean up the TFJob manually?

Yeah, we will not delete TFJob when the TFJob is succeeded.

[asahalyft]

Thanks @gaocegege for the explanation.