kubectl get pod -l job-name --watch -o jsonpath produces no output

[starpit]

What happened:

When doing a watch with a label selector and a jsonpath output, I get no output. When using the default output, or json output, or no label-selector, or no watch... all these other combinations work as expected. Only this combination:

kubectl get pod -l job-name -o jsonpath='{range .items[*]}{.metadata.name}{"|"}{.kind}{"|"}{.apiVersion}{"|"}{.metadata.namespace}{"|\n"}{end}' --watch

Remove the --watch, or remove the -l, or remove the -o jsonpath, and things work as expected.

What you expected to happen:

I expect a live table of my jobs, using the custom jsonpath formatting. But I can only get a live table if I avoid using jsonpath.

How to reproduce it (as minimally and precisely as possible):

First, make sure you have some pods for jobs. You can use this example job spec:

kubectl apply -f https://kubernetes.io/examples/controllers/job.yaml

Then execute the above command to list the job pods. Without --watch, you should see the job-pod. With --watch, you should see nothing. With --watch and without -o jsonpath, you should see something.

Anything else we need to know?:

Environment:

• Kubernetes client and server versions (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-14T11:09:22Z", GoVersion:"go1.14.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9+IKS", GitCommit:"dbe4e8a5c169e0a6a8a9b7b183561fa61e161985", GitTreeState:"clean", BuildDate:"2020-07-16T01:30:57Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Also tested: client 1.18.3. I believe the problem also exists in 1.16 and minikube.

• Cloud provider or hardware configuration: ibmcloud and minikube
• OS (e.g: cat /etc/os-release): macOS 10.15.6

[starpit]

i see. the discrepancy here is that when not doing a watch, the jsonpath needs the {range items[*]}...{end} iterator. but, when doing a watch, if you have that iterator, the watcher produces no output.

instead, to switch from non-watch to watch, you need to add --watch, and you need to modify the jsonpath to exclude the range iterator. was this intentional?

[brianpursley]

Using kubectl 1.18.0 and a 1.19 cluster, I see the same thing, it just "hangs"

But strangely, with kubectl 1.20.0, I actually get an error message:

error: unable to output the provided object: error executing jsonpath "{range .items[*]}{.metadata.name}{\"|\"}{.kind}{\"|\"}{.apiVersion}{\"|\"}{.metadata.namespace}{\"|\\n\"}{end}": Error executing template: not in range, nothing to end. Printing more information for debugging the template:
    template was:
        {range .items[*]}{.metadata.name}{"|"}{.kind}{"|"}{.apiVersion}{"|"}{.metadata.namespace}{"|\n"}{end}
    object given to jsonpath engine was:
        map[string]interface {}{"apiVersion":"v1", "kind":"Pod", "metadata":map[string]interface {}{"annotations":map[string]interface {}{"cni.projectcalico.org/podIP":"172.16.230.33/32", "cni.projectcalico.org/podIPs":"172.16.230.33/32"}, "creationTimestamp":"2020-08-19T00:43:26Z", "generateName":"pi-", "labels":map[string]interface {}{"controller-uid":"abb705c7-3dc4-41cf-a7fc-0c262f666941", "job-name":"pi"}, "managedFields":[]interface {}{map[string]interface {}{"apiVersion":"v1", "fieldsType":"FieldsV1", "fieldsV1":map[string]interface {}{"f:metadata":map[string]interface {}{"f:generateName":map[string]interface {}{}, "f:labels":map[string]interface {}{".":map[string]interface {}{}, "f:controller-uid":map[string]interface {}{}, "f:job-name":map[string]interface {}{}}, "f:ownerReferences":map[string]interface {}{".":map[string]interface {}{}, "k:{\"uid\":\"abb705c7-3dc4-41cf-a7fc-0c262f666941\"}":map[string]interface {}{".":map[string]interface {}{}, "f:apiVersion":map[string]interface {}{}, "f:blockOwnerDeletion":map[string]interface {}{}, "f:controller":map[string]interface {}{}, "f:kind":map[string]interface {}{}, "f:name":map[string]interface {}{}, "f:uid":map[string]interface {}{}}}}, "f:spec":map[string]interface {}{"f:containers":map[string]interface {}{"k:{\"name\":\"pi\"}":map[string]interface {}{".":map[string]interface {}{}, "f:command":map[string]interface {}{}, "f:image":map[string]interface {}{}, "f:imagePullPolicy":map[string]interface {}{}, "f:name":map[string]interface {}{}, "f:resources":map[string]interface {}{}, "f:terminationMessagePath":map[string]interface {}{}, "f:terminationMessagePolicy":map[string]interface {}{}}}, "f:dnsPolicy":map[string]interface {}{}, "f:enableServiceLinks":map[string]interface {}{}, "f:restartPolicy":map[string]interface {}{}, "f:schedulerName":map[string]interface {}{}, "f:securityContext":map[string]interface {}{}, "f:terminationGracePeriodSeconds":map[string]interface {}{}}}, "manager":"kube-controller-manager", "operation":"Update", "time":"2020-08-19T00:43:26Z"}, map[string]interface {}{"apiVersion":"v1", "fieldsType":"FieldsV1", "fieldsV1":map[string]interface {}{"f:metadata":map[string]interface {}{"f:annotations":map[string]interface {}{".":map[string]interface {}{}, "f:cni.projectcalico.org/podIP":map[string]interface {}{}, "f:cni.projectcalico.org/podIPs":map[string]interface {}{}}}}, "manager":"calico", "operation":"Update", "time":"2020-08-19T00:43:34Z"}, map[string]interface {}{"apiVersion":"v1", "fieldsType":"FieldsV1", "fieldsV1":map[string]interface {}{"f:status":map[string]interface {}{"f:conditions":map[string]interface {}{"k:{\"type\":\"ContainersReady\"}":map[string]interface {}{".":map[string]interface {}{}, "f:lastProbeTime":map[string]interface {}{}, "f:lastTransitionTime":map[string]interface {}{}, "f:reason":map[string]interface {}{}, "f:status":map[string]interface {}{}, "f:type":map[string]interface {}{}}, "k:{\"type\":\"Initialized\"}":map[string]interface {}{".":map[string]interface {}{}, "f:lastProbeTime":map[string]interface {}{}, "f:lastTransitionTime":map[string]interface {}{}, "f:reason":map[string]interface {}{}, "f:status":map[string]interface {}{}, "f:type":map[string]interface {}{}}, "k:{\"type\":\"Ready\"}":map[string]interface {}{".":map[string]interface {}{}, "f:lastProbeTime":map[string]interface {}{}, "f:lastTransitionTime":map[string]interface {}{}, "f:reason":map[string]interface {}{}, "f:status":map[string]interface {}{}, "f:type":map[string]interface {}{}}}, "f:containerStatuses":map[string]interface {}{}, "f:hostIP":map[string]interface {}{}, "f:phase":map[string]interface {}{}, "f:podIP":map[string]interface {}{}, "f:podIPs":map[string]interface {}{".":map[string]interface {}{}, "k:{\"ip\":\"172.16.230.33\"}":map[string]interface {}{".":map[string]interface {}{}, "f:ip":map[string]interface {}{}}}, "f:startTime":map[string]interface {}{}}}, "manager":"kubelet", "operation":"Update", "time":"2020-08-19T00:44:30Z"}}, "name":"pi-7rq96", "namespace":"default", "ownerReferences":[]interface {}{map[string]interface {}{"apiVersion":"batch/v1", "blockOwnerDeletion":true, "controller":true, "kind":"Job", "name":"pi", "uid":"abb705c7-3dc4-41cf-a7fc-0c262f666941"}}, "resourceVersion":"15316162", "selfLink":"/api/v1/namespaces/default/pods/pi-7rq96", "uid":"694521d6-1598-43a3-a417-a87f8980cca1"}, "spec":map[string]interface {}{"containers":[]interface {}{map[string]interface {}{"command":[]interface {}{"perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"}, "image":"perl", "imagePullPolicy":"Always", "name":"pi", "resources":map[string]interface {}{}, "terminationMessagePath":"/dev/termination-log", "terminationMessagePolicy":"File", "volumeMounts":[]interface {}{map[string]interface {}{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount", "name":"default-token-hgc4x", "readOnly":true}}}}, "dnsPolicy":"ClusterFirst", "enableServiceLinks":true, "nodeName":"k8s-worker-1", "priority":0, "restartPolicy":"Never", "schedulerName":"default-scheduler", "securityContext":map[string]interface {}{}, "serviceAccount":"default", "serviceAccountName":"default", "terminationGracePeriodSeconds":30, "tolerations":[]interface {}{map[string]interface {}{"effect":"NoExecute", "key":"node.kubernetes.io/not-ready", "operator":"Exists", "tolerationSeconds":300}, map[string]interface {}{"effect":"NoExecute", "key":"node.kubernetes.io/unreachable", "operator":"Exists", "tolerationSeconds":300}}, "volumes":[]interface {}{map[string]interface {}{"name":"default-token-hgc4x", "secret":map[string]interface {}{"defaultMode":420, "secretName":"default-token-hgc4x"}}}}, "status":map[string]interface {}{"conditions":[]interface {}{map[string]interface {}{"lastProbeTime":interface {}(nil), "lastTransitionTime":"2020-08-19T00:43:27Z", "reason":"PodCompleted", "status":"True", "type":"Initialized"}, map[string]interface {}{"lastProbeTime":interface {}(nil), "lastTransitionTime":"2020-08-19T00:44:30Z", "reason":"PodCompleted", "status":"False", "type":"Ready"}, map[string]interface {}{"lastProbeTime":interface {}(nil), "lastTransitionTime":"2020-08-19T00:44:30Z", "reason":"PodCompleted", "status":"False", "type":"ContainersReady"}, map[string]interface {}{"lastProbeTime":interface {}(nil), "lastTransitionTime":"2020-08-19T00:43:26Z", "status":"True", "type":"PodScheduled"}}, "containerStatuses":[]interface {}{map[string]interface {}{"containerID":"docker://a66a7d6253325eb2a5c111d44a489689757b67283fe63bcdff67162ea8de2eb4", "image":"perl:latest", "imageID":"docker-pullable://perl@sha256:d3ada65696d21e0c2fdbb82c5595a72e32a72a126291713b511974b0575a9142", "lastState":map[string]interface {}{}, "name":"pi", "ready":false, "restartCount":0, "started":false, "state":map[string]interface {}{"terminated":map[string]interface {}{"containerID":"docker://a66a7d6253325eb2a5c111d44a489689757b67283fe63bcdff67162ea8de2eb4", "exitCode":0, "finishedAt":"2020-08-19T00:44:27Z", "reason":"Completed", "startedAt":"2020-08-19T00:44:20Z"}}}}, "hostIP":"192.168.200.12", "phase":"Succeeded", "podIP":"172.16.230.33", "podIPs":[]interface {}{map[string]interface {}{"ip":"172.16.230.33"}}, "qosClass":"BestEffort", "startTime":"2020-08-19T00:43:27Z"}}

But just like @starpit said, if I don't use --watch then it prints the output as expected:

pi-7rq96|Pod|v1|default|

[starpit]

thanks for looking into this. i think the label-selector is all a red herring. this perhaps is more simply an oddity with watches silently failing when given a ranged jsonpath, for any kind (not just jobs) and with or without a selector.

this also seems to be independent of watch versus watch-only. either mode of ranged jsonpath watches seems to remain persistently silent.

when i look at --v 9 output, i see some odd, though perhaps innocuous, messages. the number of such messages matches exactly the number of expected (but not emitted) rows in the "initial state" part of the watch.

I0818 21:21:35.912284 51208 table_printer.go:45] Unable to decode server response into a Table. Falling back to hardcoded types: attempt to decode non-Table object

[soltysh]

I think it's being resolved in https://github.com/kubernetes/kubernetes/pull/90137

[eddiezane]

/priority backlog

[soltysh]

@starpit I did some testing and I think the problem is that you're actually working with different resources depending on what you request. Regular get returns a list of pods, but a watch returns objects one by one, thus you should be seeing errors about template not matching desired output like so:

error: unable to output the provided object: error executing jsonpath "{range .items[*]}{.metadata.name}{\"|\"}{.kind}{\"|\"}{.apiVersion}{\"|\"}{.metadata.namespace}{\"|\\n\"}{end}": Error executing template: not in range, nothing to end. Printing more information for debugging the template:
    template was:
        {range .items[*]}{.metadata.name}{"|"}{.kind}{"|"}{.apiVersion}{"|"}{.metadata.namespace}{"|\n"}{end}
    object given to jsonpath engine was:
        map[string]interface {}{"apiVersion":"v1", "kind":"Pod", "metadata":map[string]interface...

So I think this is more of a problem with the set of flags and your expectations. Probably worth a better explanation about possible flag combination or a check.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.