Kubectl: Kubectl returns pods in various states when only those in Running state are selected via --field-selector, or -o jsonpath

[wknapik]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
field-selector, field selector, status phase

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  AWS
• OS (e.g. from /etc/os-release):
  Arch Linux
• Kernel (e.g. uname -a):
  Linux something 4.16.3-1-ARCH #1 SMP PREEMPT Thu Apr 19 09:17:56 UTC 2018 x86_64 GNU/Linux
• Install tools:
  ?
• Others:
  ?

What happened:

kubectl get pods --field-selector=status.phase=Running --namespace some-ns

returns pods in other states than Running.

What you expected to happen:
I expected only pods in state Running to be returned.

How to reproduce it (as minimally and precisely as possible):

kubectl get pods --field-selector=status.phase=Running --namespace some-ns

Anything else we need to know:
That's way too deep... I don't think I can tackle that question here...

[wknapik]

Selecting pods like this

kubectl get pods --namespace some-ns -o jsonpath='{.items[?(@.status.phase=="Running")]}'

also doesn't work as expected. Pods in states like Terminating and ContainerCreating are returned.
In fact, if you run

kubectl get pods --namespace some-ns -o jsonpath='{.items[?(@.status.phase=="Running")].status.phase}'

you'll see everything described as Running, even though

kubectl get pods --namespace some-ns

will show the same pods in other states.

I feel like I'm missing some key piece of information here, but the cheatsheet here https://kubernetes.io/docs/reference/kubectl/cheatsheet/ actually says

# Get all running pods in the namespace
$ kubectl get pods --field-selector=status.phase=Running

[pashau]

we have the same problem.
kubectl get pods -n namespace -o jsonpath='{.items[*].status.containerStatuses[?(@.restartCount!=0)].image}'
returns (images of) Pods that had multiple Restarts. I would like to do something like this with --filed-selector kubectl get pods -n namespace --watch=true --field-selector=status.containerStatuses.restartCount!=0 | ts '[%Y-%m-%d %H:%M:%.S]'

just using this: kubectl get pods -n namespace --watch=true | ts '[%Y-%m-%d %H:%M:%.S]' | grep -v " 0 "

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wknapik]

/remove-lifecycle stale

[seans3]

/kind feature
/sig cli
/area kubectl
/priority P3

[wknapik]

@seans3 I'd say that's a bug, not a feature request. The code doesn't do what the documentation says it would. And it doesn't error out, it returns incorrect results.

[rafi-greenberg]

+1

I'm seeing this bug.

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-10T11:44:36Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.7-gke.6", GitCommit:"06898a4d0f2b96f82b43d9e59fa2825bd3d616a2", GitTreeState:"clean", BuildDate:"2018-10-02T17:32:01Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

[seans3]

/kind bug
/priority-remove P3
/priority P1

[seans3]

/remove-priority P3

[seans3]

/remove-kind feature

[vithati]

/assign vithati

[k8s-ci-robot]

@vithati: GitHub didn't allow me to assign the following users: vithati.

Note that only kubernetes members and repo collaborators can be assigned.
For more information please see the contributor guide

In response to this:

/assign vithati

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[DovAmir]

+1

[DovAmir]

Actually, this is not a selector error,
for some reason, a pod in a terminating state does not have a "Terminating" value in its yaml.
doing the following command on a pod in terminating state
kubectl get pod -l app=myapp -o yaml
returns:

status:
...phase: Running

By looking at the yaml,
the only diffrenece between a running pod and a terminating pod
is that the terminating pod has the following 2 fields in metadata.
It is not possible to use selectors to select/filter by these 2 fields

metadata:
...deletionGracePeriodSeconds: 30
...deletionTimestamp:'2018-12-11T15:42:14Z'

my workaround hack to getting running pods that are NOT terminating is with the following command :

kubectl get pod | grep Running

[vithati]

This looks like not a bug. kubectl get pods output STATUS column not shows status.phase. kubectl get pods displays table format of PodList object and uses status.containerStatuses states to display STATUS column data.

Pod phase valid states are pending, running, succeeded, failed and unknown. From this document 'Running' state depends on 'restartPolicy'.

Jsonpath applies on 'PodList' object, not on kubectl output. status.containerStatuses gives whether pod containers running or not. Field selectors vary by Kubernetes resource types. status.containerStatuses not supported in pods field selectors.

[vithati]

--field-selector, or -o jsonpath depends on object data, not on kubectl get table format output. I think adding more information to command comment helps to resolve this issue.

In https://kubernetes.io/docs/reference/kubectl/cheatsheet/
'#Get all running pods in the namespace' change to '#Get all running pods in the namespace and Running state depends on 'restartPolicy'. Refer https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#example-states for pod status.phase states'

[idealhack]

/close

feel free to reopen if it still an issue

[k8s-ci-robot]

@idealhack: Closing this issue.

In response to this:

/close

feel free to reopen if it still an issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wknapik]

I see no PRs referencing this issue, which means it's still valid. Why was this closed ?

[soltysh]

Closing per previous comment.

[wknapik]

@soltysh which previous comment ?

Nothing has changed since I reported this, afaik. kubectl still reports something else than the user is expecting (and what the documentation is saying it would) and there's still no way to get pods in state Running without scraping unstructured output. I don't understand why this was closed.

[howardjohn]

I think this should be open again.

The docs clearly state

Get all running pods in the namespace
$ kubectl get pods --field-selector=status.phase=Running

This command doesn't get all pods in running state.

Call it a bug, a doc error, or even a UX issue of confusing command, but something seems missing here.

I should be able to query for running pods without doing some hack like kubectl get pods | grep Running

[sassanh]

Currently there's no easy way to select running pods (as running means in English language)
So please reopen this issue and do not close it until API supports this feature as it seems to be a basic feature to list running pods.

This is my dirty workaround for now: kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].state.running},{.items[*].metadata.name}' --field-selector=status.phase==Running | sed 's/ /\n/' | grep startedAt | awk -F',' '{print $2}'

I'm sure you agree this is not how it's supposed to be.

[vmrm]

# Get all running pods in the namespace kubectl get pods --field-selector=status.phase=Running
from https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-running-pods
isn't working as expected. Please reopen this issue.

[gdankov]

Can you please reopen this issue?

This is still not working as documented!

[wknapik]

@idealhack

feel free to reopen if it still an issue

This should never have been closed. The issue is still there and, iuuc, nothing's been done to address it. Please reopen.

[idealhack]

/reopen

but I'll defer more discussion to @kubernetes/sig-cli-bugs

[k8s-ci-robot]

@idealhack: Reopened this issue.

In response to this:

/reopen

but I'll defer more discussion to @kubernetes/sig-cli-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alahijani]

As a workaround for now I'm relying on the READY column showing n/n:

kubectl get pod | awk -F' *|/' '$2 == $3 && $4 == "Running"'

[brianpursley]

I looked into this issue a little bit today and here is what I found out.

First of all, for reference, here is how I am reproducing:

kubectl run nginx --image=nginx && sleep 1 && kubectl delete deploy nginx && sleep 1 && kubectl get pods --field-selector=status.phase=Running

The output is:

deployment.apps/nginx created
deployment.apps "nginx" deleted
NAME                     READY   STATUS        RESTARTS   AGE
nginx-6db489d4b7-k4k4j   1/1     Terminating   0          2s

At least by my understanding, this is the problem: I told kubectl only to get pods with --field-selector=status.phase=Running but I see the pod in a Terminating status.

The reason this happens is that the value displayed in the table's status column is constructed on-the-fly based on many different factors. It is not the actual status.phase of the pod, which contributes to it, but there are other factors as well.

In this situation, the code reports a status of "Terminating" because when the pod is being deleted, the DeletionTimestamp != nil. See code excerpt below to see where this occurs:

https://github.com/kubernetes/kubernetes/blob/fe2fdcd6953a877664cbdb784888c0e7f636ce40/pkg/printers/internalversion/printers.go#L780

        if pod.DeletionTimestamp != nil && pod.Status.Reason == node.NodeUnreachablePodReason {
        reason = "Unknown"
    } else if pod.DeletionTimestamp != nil {
        reason = "Terminating"
    }

Technically the pod is still in the Running status phase even though Terminating is displayed as the Status in the table. So the field selector is technically correct, although it is confusing.

---

So given that this issue has been open for a long time, I am trying to understand a couple of things:

1. Is this still an issue where the code needs to be changed, and if so, what is the proposed new behavior? I think this issue is stuck because it is not clear what needs to change.

2. Could this be addressed without a code change? For example, by improving the documentation to explain the difference between the displayable status and status.phase can be explained?

Feel free to respond, anyone who has more information or an opinion on this issue.

[wknapik]

@brianpursley getting running pods from a script should be easy and reliable. Currently it's not. You could change the documentation, but that would not help users with that problem.

And yes, the current documentation is actively misleading and should be changed, but that's not where the work should stop.

IMO.

[brianpursley]

Thanks @wknapik

By "running pods" do you mean you do not want to see the ones that are still running, but in the process of terminating? Because that is what it is showing you now. When it says Terminating in the kubectl get output, that is technically still a pod that is in a Running status phase, it just shows you "Terminating" because it knows it is in the process of terminating, but that is not an actual status.

I'm not trying to be difficult, just trying to define exactly what the expected behavior should be so that the issue can potentially be worked on.

I see one of the workarounds above is to use awk to examine the Ready column. Is it that you are really interested in knowing how many pods are completely ready? Ready being defined as 100% of the containers in the pod have ready == true.

[brianpursley]

I was curious to see if there was a way using only kubectl to get a list of the ready pods, and this is what I came up with (inspired by alahijani's awk workaround above)

It uses a go-template to print the name of pods only if all of its containers are ready.

It's definitely more complex than the awk workaround, but if you are looking for a pure kubectl way, or want to extend it with other conditions or fields, this go-template approach might be an option.

kubectl get pods -o=go-template --template='{{range .items}}{{$ready:=true}}{{range .status.containerStatuses}}{{if not .ready}}{{$ready = false}}{{end}}{{end}}{{if $ready}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}'

[vosmax]

I've simplified @brianpursley solution a bit.

Find NOT READY pods:
kubectl -n default get pods -o=go-template --template='{{range $i := .items}}{{range .status.containerStatuses}}{{if not .ready}}{{$i.metadata.name}}{{"\n"}}{{end}}{{end}}{{end}}'

Find READY pods:
kubectl -n default get pods -o=go-template --template='{{range $i := .items}}{{range .status.containerStatuses}}{{if .ready}}{{$i.metadata.name}}{{"\n"}}{{end}}{{end}}{{end}}'

[GMZwinge]

@brianpursley and @vosmax: the kubectl commands using the go-template don't work in my case. They handle the pods with a STATUS of Terminating as if they had a STATUS of Running.

[vosmax]

@GMZwinge this is normal behavior. While termination is going, container has status Running, because it got termination signal and trying kindly complete the main thread. When it's terminated pod changes his phase on state Terminated.
Here you may find answers on your questions.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[wknapik]

/remove-lifecycle rotten

[DirkTheDaring]

Get running pods and do not show Terminating pods (.metata.deletiion != nil)

kubectl get pods --field-selector=status.phase=Running --template {{range .items}}{{ if not .metadata.deletionTimestamp }}{{.metadata.name}}{{"\n"}}{{end}}{{end}}

[lallinger-arbeit]

I understand the issue with people wanting to get pods which are in terminating state, but at least there should be a flag to add to tell kubectl it really should only output pods which also show running state when calling kubectl get pods.

Thanks @DirkTheDaring, your solution outputs the desired information!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[szpak]

It would still be nice to have it possible without any "templating magic".

/remove-lifecycle rotten

[danielyaa5]

Ya this is hella dumb, status says terminating not running... lets not overcomplicate things or at least update documentation

[danielyaa5]

 ➜ kubectl get pods --field-selector=status.phase=Running --template {{range .items}}{{ if not .metadata.deletionTimestamp }}{{.metadata.name}}{{"\n"}}{{end}}{{end}}
error: name cannot be provided when a selector is specified

[danielyaa5]

This did work for me though kubectl get pods --template '{{range .items}}{{ if not .metadata.deletionTimestamp }}{{.metadata.name}}{{"\n"}}{{end}}{{end}}'