Kubectl: "kubectl get events --watch" prints old events after some time

Created on 20 Aug 2019 · 18Comments · Source: kubernetes/kubectl

For example:

Run kubectl get events --all-namespaces --watch
A load of events will be printed
Wait 10 seconds
Only new events should be printed

At this point I assume anything printed will be new events, right? But after this point, even if I wait for 5 minutes, I will see events that were last seen 60m ago, e.g:

kube-system   60m         Normal    Pulling                        pod/node-problem-detector-vmcf2                        pulling image "k8s.gcr.io/node-problem-detector:v0.7.0"
kube-system   60m         Normal    Pulled                         pod/node-problem-detector-vmcf2                        Successfully pulled image "k8s.gcr.io/node-problem-detector:v0.7.0"
kube-system   60m         Normal    Created                        pod/node-problem-detector-vmcf2                        Created container
kube-system   60m         Normal    Started                        pod/node-problem-detector-vmcf2                        Started container

arekubectl kinbug prioritP2 sicli

Source

max-rocket-internet

👍9

Most helpful comment

The issue is that it prints the event of deletion of the related event. There is a cleanup mechanism that deletes events older than 60m, this deletion is an event itself, and it gets printed by the watch.

Using a Java client I was able to filter out these deletion events.

Hope this helps!

eliaoggian on 16 Apr 2020

❤5

All 18 comments

Please add information on client version and OS/arch, following the template for a kubectl issue.

/sig cli
/area kubectl
/kind bug
/priority P2

seans3 on 20 Aug 2019

Please add information on client version and OS/arch

Gosh, sorry I forgot to include that:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-20T04:52:26Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.10-eks-2e569f", GitCommit:"2e569fd887357952e506846ed47fc30cc385409a", GitTreeState:"clean", BuildDate:"2019-07-25T23:13:33Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

following the template for a kubectl issue.

There isn't one. There needs to be some files in here. Should I create a PR for this?

max-rocket-internet on 21 Aug 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 19 Nov 2019

/remove-lifecycle stale

max-rocket-internet on 19 Nov 2019

Still not fixed:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-19T14:00:14Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

max-rocket-internet on 19 Nov 2019

I got the same issue:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-16T16:29:58Z", GoVersion:"go1.12.14", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:39:02Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

PSanetra on 17 Dec 2019

I was often seeing 60m old events every few minutes, so I've had to manually filter them out with jq:

kubectl get events --no-headers --watch-only --field-selector involvedObject.name=temp-db-restorer-nue -ojson \
| TZ=UTC jq -r '. | select ( .lastTimestamp | fromdateiso8601 > now - 60 ) | "'"$amber"'\( .kind ):\t\( .lastTimestamp )\t\( .reason )\t\( .involvedObject.fieldPath )\t\( .message )'"$reset"'"'

System (using GKE):

$ kubectl version                                                                                                                                              
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:53:53Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.14", GitCommit:"56d89863d1033f9668ddd6e1c1aea81cd846ef88", GitTreeState:"clean", BuildDate:"2019-11-07T19:12:22Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

ZimbiX on 17 Jan 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 16 Apr 2020

Seems like this is happening for a few people...

/remove-lifecycle stale

brianpursley on 16 Apr 2020

👍3

Using a Java client I was able to filter out these deletion events.

Hope this helps!

eliaoggian on 16 Apr 2020

❤5

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 15 Jul 2020

/remove-lifecycle stale

max-rocket-internet on 15 Jul 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 13 Oct 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 12 Nov 2020

Ok, so in that case how do I filter out these events via the kubectl cli?

nivekastoreth on 11 Dec 2020

/remove-lifecycle rotten

max-rocket-internet on 14 Dec 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

fejta-bot on 14 Mar 2021

/remove-lifecycle stale