Python: Deleting job with delete_namespaced_job() does not delete pods

[nomastmas]

When deleting a job with BatchV1Api.delete_namespaced_job, the job resource is deleted, but the pod(s) remain.

From this documentation, my assumption was to try Ophaned, Background, or Foreground as the propagation_policy, but none of them did anything to the pod after the job was deleted.

Upon further inspection of the pod metadata, both finalizers and owner_references are set with None.

Here's the gist of the script I used to reproduce this.

[mbohlool]

is kubectl works for the same operation as you expected?

[nomastmas]

@mbohlool yes it does

[mbohlool]

can you run kubectl with -v 9 and paste a working example here? What I am interested in is which API it calls and if we can do the same API calls with the python client.

[nomastmas]

@mbohlool https://pastebin.com/qgiGhFXK

[mbohlool]

From the logs you pasted it look like kubectl is doing it manually. It deletes the pod first:

curl -k -v -XDELETE  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.4.7+92b4f97 (darwin/amd64) kubernetes/92b4f97" https://cat.bug/api/v1/namespaces/sharedservices/pods/e2e-prospects-95rx0

and then deletes the job:

curl -k -v -XDELETE  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.4.7+92b4f97 (darwin/amd64) kubernetes/92b4f97" https://cat.bug/apis/batch/v1/namespaces/sharedservices/jobs/e2e-prospects

python client does not add any logic on top of the kubernetes API server so if kubectl is doing it that way, you should do the same in your client.

[nomastmas]

ah okay, I was operating under a faulty assumption then. Would you kindly provide an example for propagation_policy? My assumption was that the policy would tell the garbage collector how to clean up the pods.

[mbohlool]

I think I know what the problem is. By loking at your code, you are passing the parameter as query parameter and most probably hitting this bug in API server: https://github.com/kubernetes/kubernetes/issues/43329

Try encoding the parameter in the body (set it in V1DeleteOptions).

[alanbchristie]

Sorry, I know this topic is closed but I'm running into this problem now and this and #43329 don't really help. Can you explain precisely how you fixed this i.e. what do you mean by _Try encoding the parameter in the body (set it in V1DeleteOptions)._?

Do I need to set anything in the V1DeleteOptions() body object and what, if any, other arguments need to be passed to delete_namespaced_job?

[drdivano]

@alanbchristie

See #423

The following code works for me:
body = client.V1DeleteOptions(propagation_policy='Background')
resp = api.delete_namespaced_deployment(name=item.metadata.name, body=body, namespace=self.namespace)

[alanbchristie]

Thanks @drdivano, at the moment the code's doing the following, which seems to work (some lines removed for clarity, like the try/catch and error handling blocks): -

_BATCH_API.delete_namespaced_job(name=self._job_name,
                                 namespace=_NAMESPACE,
                                 body=body)
_CORE_API.delete_namespaced_pod(name=self._pod_name,
                                namespace=_NAMESPACE,
                                body=body)

I've made a note of your suggestion and will try that at next time I spin it up.

Thanks for the advice.

[flyhighzy]

Thanks all, I met the same problem and got solved by your solutions.
By the way, since propagation_policy parameter has no effect by passing the parameter as query parameter, could @thomaswtsang update doc and remind developers to put it in the body?

[prateek2408]

@flyhighzy - I totally agree, the documentation needs to update as propagation policy is part of body

[hui-zheng]

thank you for all the discussion.

I had the same problem and confirmed that it is resolved by setting

        body = client.V1DeleteOptions(propagation_policy='Background')

agree that we shall remove the propagation_policy parameter in delete_namespaced_job() as it has no effect and cause confusion.

[daganida88]

@alanbchristie

See #423

The following code works for me:
body = client.V1DeleteOptions(propagation_policy='Background')
resp = api.delete_namespaced_deployment(name=item.metadata.name, body=body, namespace=self.namespace)

Solved it for me also