Python: list_event_for_all_namespaces does not return DELETE events

[pzoxiuv]

What happened (please include outputs or screenshots):
Watching list_event_for_all_namespaces does not return any events when a Pod is deleted.

I have two scripts: one prints all events with list_event_for_all_namespaces, and the other creates and then deletes a Pod. The watcher prints events while the Pod is starting (pulling container, pulled container, starting container, etc.). But I don't get any event when the pod is deleted.

What you expected to happen:

In the apiserver audit log there are events with type='DELETED' - I expect to see those events returned by list_event_for_all_namespaces.

How to reproduce it (as minimally and precisely as possible):
test.yaml:

import yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-testing
spec:
  restartPolicy: Never
  containers:
  - image: busybox
    name: pod-testing
    command: ["sleep", "10"]

watch.py:

from kubernetes import client, config, watch

config.load_kube_config()
v1 = client.CoreV1Api()
w = watch.Watch()
for event in w.stream(v1.list_event_for_all_namespaces):
    print(event)

Start watching, python watch.py. In a separate terminal, do kubectl apply -f test.yaml, then after the Pod is started do kubectl delete -f test.yaml.

Anything else we need to know?:

Environment:

• Kubernetes version (kubectl version): 1.16
• OS (e.g., MacOS 10.13.6): Centos 7.7.1908
• Python version (python --version): 3.6.8
• Python client version (pip list | grep kubernetes): 10.0.1

[roycaihw]

list_event_for_all_namespaces lists the Kubernetes core v1 Event API objects. When you watch list_event_for_all_namespaces, you will get ADDED/MODIFIED/DELETED events for the v1 Event objects.

To watch ADDED/MODIFIED/DELETED events for the v1 Pod objects, you need to watch list_pod_for_all_namespaces instead.

from kubernetes import client, config, watch

config.load_kube_config()
v1 = client.CoreV1Api()
w = watch.Watch()
for event in w.stream(v1.list_pod_for_all_namespaces):
    print(event)

/close

[k8s-ci-robot]

@roycaihw: Closing this issue.

In response to this:

list_event_for_all_namespaces lists the Kubernetes core v1 Event API objects. When you watch list_event_for_all_namespaces, you will get ADDED/MODIFIED/DELETED events for the v1 Event objects.

To watch ADDED/MODIFIED/DELETED events for the v1 Pod objects, you need to watch list_pod_for_all_namespaces instead.

from kubernetes import client, config, watch

config.load_kube_config()
v1 = client.CoreV1Api()
w = watch.Watch()
for event in w.stream(v1.list_pod_for_all_namespaces):
  print(event)

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alanjcastonguay]

There are several concepts confusingly called events.

• The list_event_for_all_namespaces retrieves v1.Event resources that might be associated with a v1.Pod or any other resource in some namespace.
• A watch+stream over list_event_for_all_namespaces will receive a stream of ADDED/MODIFIED/DELETED events about changes to that collection of v1.Event objects, _not about changes to the collection of v1.Pod objects_.

A demonstration

We can watch for v1.Event resources:

$ kubectl get events -v=6 --watch
I0302 12:59:51.546532   34283 loader.go:375] Config loaded from file:  /Users/alan/.kube/config
I0302 12:59:51.591488   34283 round_trippers.go:443] GET https://kubernetes.docker.internal:6443/api/v1/namespaces/default/events?limit=500 200 OK in 27 milliseconds
I0302 12:59:51.595758   34283 round_trippers.go:443] GET https://kubernetes.docker.internal:6443/api/v1/namespaces/default/events?resourceVersion=2358340&watch=true 200 OK in 3 milliseconds
LAST SEEN   TYPE     REASON      OBJECT            MESSAGE

While that watch is running, create a pod with kubectl apply -f pod-testing.yaml. These events appear. You my recognize the as the same events that appear at the bottom of kubectl describe pod

0s          Normal   Scheduled   pod/pod-testing   Successfully assigned default/pod-testing to docker-desktop
0s          Normal   Pulling     pod/pod-testing   Pulling image "busybox"
0s          Normal   Pulled      pod/pod-testing   Successfully pulled image "busybox"
0s          Normal   Created     pod/pod-testing   Created container pod-testing
0s          Normal   Started     pod/pod-testing   Started container pod-testing

While that watch is running, you can delete a v1.Event resource. They're in the same namespace.

$ kubectl get events -o name
event/pod-testing.15f88e8b53022dc8
event/pod-testing.15f88e8be6d88448
event/pod-testing.15f88e8c84237f8c

$ kubectl delete event pod-testing.15f88e8c84237f8c
event "pod-testing.15f88e8c84237f8c" deleted

The watch stream emits a record for the Deletion of the v1.Event object. This appears as an item from the w.stream(v1.list_event_for_all_namespaces) iterable too.

...
2m39s       Normal   Started     pod/pod-testing   Started container pod-testing

While the watch is still running, delete the pod. The watch emits nothing, because NO v1.Event resources were deleted or modified or added.

$ kubectl delete -f pod-testing.yaml 
pod "pod-testing" deleted

Indeed, you can list the v1.Event resources. They two we saw earlier are still present.

$ kubectl get events
LAST SEEN   TYPE     REASON      OBJECT            MESSAGE
13m         Normal   Scheduled   pod/pod-testing   Successfully assigned default/pod-testing to docker-desktop
13m         Normal   Pulling     pod/pod-testing   Pulling image "busybox"

That list can be filtered the same way kubectl describe pod pod-testing -v=10 does, by passing a fieldSelector for the involvedObject.{name, namespace,uid} keys.

I0302 13:35:32.833715   34401 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.16.2 (darwin/amd64) kubernetes/c97fe50" 'https://kubernetes.docker.internal:6443/api/v1/namespaces/default/events?fieldSelector=involvedObject.name%3Dpod-testing%2CinvolvedObject.namespace%3Ddefault%2CinvolvedObject.uid%3D55cdd4f6-9d8c-4cd0-8d96-0adcd54e793e'

aka this, without the watch

$ kubectl get events --field-selector involvedObject.name=pod-testing,involvedObject.namespace=default,involvedObject.uid=55cdd4f6-9d8c-4cd0-8d96-0adcd54e793e
LAST SEEN   TYPE     REASON      OBJECT            MESSAGE
2m24s       Normal   Scheduled   pod/pod-testing   Successfully assigned default/pod-testing to docker-desktop
2m23s       Normal   Pulling     pod/pod-testing   Pulling image "busybox"
2m22s       Normal   Pulled      pod/pod-testing   Successfully pulled image "busybox"
2m22s       Normal   Created     pod/pod-testing   Created container pod-testing
2m22s       Normal   Started     pod/pod-testing   Started container pod-testing

Next steps

• If you want a watch+stream to see a v1.Pod be deleted, use list_pod_for_all_namespaces.
• If you want the events associated with a v1.Pod, you can use list_namespaced_event with a fieldSelector citing the v1.Pod involvedObject.

[roycaihw]

Thanks @alanjcastonguay for the detailed explanation!

[alanjcastonguay]

The v1.Event resources listed above will (probably) be deleted automatically. The default TTL is 1 hour after creation. If you left the kubectl get events -v=10 --watch running, you would see this;

60m         Normal   Scheduled   pod/pod-testing   Successfully assigned default/pod-testing to docker-desktop
60m         Normal   Pulling     pod/pod-testing   Pulling image "busybox"

And because they were deleted kubectl get events doesn't show those two v1.Event objects anymore.

$ kubectl get events 
LAST SEEN   TYPE     REASON      OBJECT            MESSAGE