kubectl describe hpa not working with v1.13.0

Created on 6 Dec 2018 · 34Comments · Source: kubernetes/kubectl

Is this a request for help? no.

What keywords did you search in Kubernetes issues before filing this one? "hpa"

Is this a BUG REPORT or FEATURE REQUEST? : Bug report

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-eks", GitCommit:"6bf27214b7e3e1e47dce27dcbd73ee1b27adadd0", GitTreeState:"clean", BuildDate:"2018-12-04T13:33:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

I installed using instructions from https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl - i.e. in my Ubuntu Desktop:

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Environment:

Cloud provider or hardware configuration: AWS EKS

What happened:

When using kubectl v1.13.0 I cannot describe hpa:

$ kubectl describe hpa my-hpa
Error from server (NotFound): the server could not find the requested resource

I tried kubectl describe HorizontalPodAutoscaler talentdesk as well, same issue.

What you expected to happen:

Before I upgraded kubectl to v1.13 , I had v1.10 and it worked ok.

$ kubectl describe hpa my-hpa
Name:                                                  my-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"my-hpa","namespace":"default"},"spec":{"...
...

Anything else we need to know:

My HPA manifests use apiVersion: autoscaling/v2beta1:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
...

I noticed the hpa documentation mentions v2beta2 but I'm using v2beta1 - I tried to upgrade my manifest but kubernetes v1.10 doesn't support v2beta2 - this is the error I get:

$ kubectl apply -f my-hpa.yaml
error: unable to recognize "my-hpa.yaml": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"

I'm stuck on kubernetes v1.10 for now because AWS EKS doesn't yet allow upgrades.

arekubectl kinbug prioritP1 sicli

Source

tomfotherby

👍4

Most helpful comment

Any chance we can bump the priority on this? There's breakage in a vital feature of kubectl.

Vlaaaaaaad on 11 Feb 2019

👍9

All 34 comments

/sig cli
/area kubectl
/kind bug
/priority P1

seans3 on 7 Dec 2018

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew

For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and should work with v1.2, v1.3, and v1.4 clients.

server 1.13's describe cmd is actually relying on v2beta2 hpa api which doesn't exist on 1.10 server. for a backward-compatibility we should downgrade the hpa describer to v2beta1 (i suppose?)

yue9944882 on 7 Dec 2018

👍1

/assign

yue9944882 on 7 Dec 2018

@yue9944882 - Thank you for explaining the situation.

For now I downgraded my kubectl back one version and all is fine. i.e. kubectl v1.12.3 works with kubernetes v1.10.11 with regards to hpa describe.

This is how I downgraded kubectl ( collapsed because not useful for most folks because Ubuntu-specific )

# Find what versions of kubectl are available:
$ sudo apt-cache policy kubectl

# Install the previous version from the latest
$ sudo apt install kubectl=1.12.3-00

# check the new version
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-eks", GitCommit:"6bf27214b7e3e1e47dce27dcbd73ee1b27adadd0", GitTreeState:"clean", BuildDate:"2018-12-04T13:33:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

# Now hpa describe works
$ k describe hpa my-hpa
Name:                                                     my-hpa
Namespace:                                                default
Labels:                                                   <none>

tomfotherby on 7 Dec 2018

For now I downgraded my kubectl back one version and all is fine. i.e. kubectl v1.12.3 works with kubernetes v1.10.11 with regards to hpa describe.

yes, downgrading kubectl to 1.12 would be a solution for this case. while 1.13 kubectl doesn't have compatibility to 1.11 as it documented^^^. so we could call it low-severity bug @seans3 wdyt?

yue9944882 on 7 Dec 2018

/remove-priority P1
/priority P3

seans3 on 7 Dec 2018

I'll close this. It's too minor to worry about and since AWS EKS has a 1-click upgrade from 1.10 to 1.11.5 the issue doesn't affect me anymore.

tomfotherby on 14 Dec 2018

I am having this issue with kubectl 1.13.1 with EKS version 1.11.5. A colleague with kubectl 1.11.3 can successfully describe HPAs, I cannot.

Vlaaaaaaad on 17 Dec 2018

I am having this issue with kubectl 1.13.1 with EKS version 1.11.5. A colleague with kubectl 1.11.3 can successfully describe HPAs, I cannot.

yes, u r hitting this very issue. plz downgrade ur kubectl to 1.12 or 1.11 for a workaround

yue9944882 on 17 Dec 2018

If this is still happening why was the issue closed?

I firmly believe it should be open until the issue is fixed.

Vlaaaaaaad on 17 Dec 2018

/reopen

yue9944882 on 17 Dec 2018

👍4

@yue9944882: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot on 17 Dec 2018

Thanks for re-opening this issue, Yes, I confirm the issue still occurs for me too:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-eks-6bad6d", GitCommit:"6bad6d9c768dc0864dab48a11653aa53b5a47043", GitTreeState:"clean", BuildDate:"2018-12-06T23:13:14Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

$ k describe hpa my-hpa
Error from server (NotFound): the server could not find the requested resource

The workaround of downgrading kubectl to 1.12 is still needed.

tomfotherby on 17 Dec 2018

👍6

kubectl should use autoscaling/v1 to describe HPA (or fall back to v1 if v2beta1 isn't available)

liggitt on 19 Dec 2018

👍1

Error from server (NotFound): horizontalpodautoscalers.autoscaling "my-hpa" not found

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-eks-6bad6d", GitCommit:"6bad6d9c768dc0864dab48a11653aa53b5a47043", GitTreeState:"clean", BuildDate:"2018-12-06T23:13:14Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

khteh on 5 Feb 2019

Any chance we can bump the priority on this? There's breakage in a vital feature of kubectl.

Vlaaaaaaad on 11 Feb 2019

👍9

This isn't just an issue with AWS EKS, it also affects Google's GKE, using the latest Kubernetes master (currently v1.11.7-gke.6):

kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.7-gke.6", GitCommit:"144b681e3870074abfdaaf176e48b1b13360d7e5", GitTreeState:"clean", BuildDate:"2019-02-09T00:08:19Z", GoVersion:"go1.10.7b4", Compiler:"gc", Platform:"linux/amd64"}

kubectl describe hpa nginx-ingress-controller
Error from server (NotFound): the server could not find the requested resource

Looking at https://**.***.***.***/openapi/v2, it seems autoscaling/v2beta1/ is supported but not autoscaling/v2beta2/.

mans0954 on 21 Feb 2019

cc @mars1024, is right WIP on this. will see a pull soon 🐛

yue9944882 on 21 Feb 2019

🎉3

Fixed by https://github.com/kubernetes/kubernetes/pull/75597, closing

yue9944882 on 28 Mar 2019

/reopen

luckymagic7 on 10 Apr 2019

@luckymagic7: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

k8s-ci-robot on 10 Apr 2019

hi all!
my kubectl version looks below:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

And I still can't get the description from the hpa resource
Error from server (NotFound): the server could not find the requested resource

luckymagic7 on 10 Apr 2019

@luckymagic7 what's your server version?

yue9944882 on 10 Apr 2019

@yue9944882 my server is 1.9.8

luckymagic7 on 10 Apr 2019

https://github.com/kubernetes/kubectl/issues/568#issuecomment-445162832

that's a gap between the client and servers. we provides compatibility up to 3 releases, which is, 1.14 kubectl can work w/ 1.12 servers at best

yue9944882 on 10 Apr 2019

@yue9944882 ok! I have to downgrade my client. THANKS!

luckymagic7 on 10 Apr 2019

I am having the same issue with the following versions:

Client 1.14.1
Server 1.12.7-gke10

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-19T22:12:47Z", GoVersion:"go1.12.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.10", GitCommit:"8d9b8641e72cf7c96efa61421e87f96387242ba1", GitTreeState:"clean", BuildDate:"2019-04-12T22:59:24Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

huguesalary on 2 May 2019

Hi there. We had this same issue in Azure's AKS with:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.12", GitCommit:"c757b93cf034d49af3a3b8ecee3b9639a7a11df7", GitTreeState:"clean", BuildDate:"2018-12-19T11:16:52Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.12", GitCommit:"c757b93cf034d49af3a3b8ecee3b9639a7a11df7", GitTreeState:"clean", BuildDate:"2018-12-19T11:04:29Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

and

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:41:57Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

The thing is, for the latter case, we tried with the following hpa configuration:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
  labels:
    cell: gscreverseproxyapi
    tier: backend
  name: gscreverseproxyapi
  namespace: production
spec:
  maxReplicas: 6
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gscreverseproxyapi
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

and this seems to work, but if we run a describe, we'll get

# kubectl get hpa gscreverseproxyapi -nproduction -o yaml
Output:
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2019-05-28T12:55:07Z","reason":"ReadyForNewScale","message":"recommended
(...)
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"labels":{"cell":"gscreverseproxyapi","tier":"backend"},"name":"gscreverseproxyapi","namespace":"production"},"spec":{"maxReplicas":6,"metrics":[{"resource":{"name":"cpu","target":{"averageUtilization":80,"type":"Utilization"}},"type":"Resource"}],"minReplicas":2,"scaleTargetRef":
(...)
status:
  currentCPUUtilizationPercentage: 0
  currentReplicas: 2
  desiredReplicas: 2

Note that apiVersion is autoscaling/v1 but latter on we got autoscaling/v2beta2 last-applied-configuration. This happens also if we do an edit, and if you change autoscaling/v1 with autoscaling/v2beta2 on the fly, it breaks, but looks like it's working

EDIT: the above seems related to https://github.com/kubernetes/kubernetes/issues/71556

Hope this info helps someone.

Azrrael on 28 May 2019

Still having this issue.

kubectl version

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-gke.0", GitCommit:"569511c9540f78a94cc6a41d895c382d0946c11a", GitTreeState:"clean", BuildDate:"2019-08-21T23:28:44Z", GoVersion:"go1.11.13b4", Compiler:"gc", Platform:"linux/amd64"}

As a workaround you can do

kubectl get --raw "/apis/autoscaling/v1/horizontalpodautoscalers" | jq -C . | less

{
  "kind": "HorizontalPodAutoscalerList",
  "apiVersion": "autoscaling/v1",
  "metadata": {
    "selfLink": "/apis/autoscaling/v1/horizontalpodautoscalers",
    "resourceVersion": "30461"
  },
  "items": [
    {
      "metadata": {
        "name": "example-hpa-external-metrics",
        "namespace": "default",
        "selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/example-hpa-external-metrics",
        "uid": "1e070646-efc4-11e9-abc2-42010a800099",
        "resourceVersion": "24778",
        "creationTimestamp": "2019-10-16T03:22:00Z",
        "annotations": {
          "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2019-10-16T03:22:15Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2019-10-16T03:22:15Z\",\"reason\":\"FailedGetExternalMetric\",\"message\":\"the HPA was unable to compute the replica count: unable to get external metric default/custom.googleapis.com|nginx-ingress-internal-controller|nginx_connections_total/nil: unable to fetch metrics from external metrics API: the server could not find the descriptor for metric custom.googleapis.com/nginx-ingress-internal-controller/nginx_connections_total: googleapi: Error 404: Could not find descriptor for metric 'custom.googleapis.com/nginx-ingress-internal-controller/nginx_connections_total'., notFound\"}]",
          "autoscaling.alpha.kubernetes.io/metrics": "[{\"type\":\"External\",\"external\":{\"metricName\":\"custom.googleapis.com|nginx-ingress-internal-controller|nginx_connections_total\",\"targetValue\":\"1\"}}]",
          "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"autoscaling/v2beta1\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"example-hpa-external-metrics\",\"namespace\":\"default\"},\"spec\":{\"maxReplicas\":5,\"metrics\":[{\"external\":{\"metricName\":\"custom.googleapis.com|nginx-ingress-internal-controller|nginx_connections_total\",\"targetValue\":1},\"type\":\"External\"}],\"minReplicas\":1,\"scaleTargetRef\":{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"name\":\"flask\"}}}\n"
        }
      },
      "spec": {
        "scaleTargetRef": {
          "kind": "Deployment",
          "name": "flask",
          "apiVersion": "apps/v1"
        },
        "minReplicas": 1,
        "maxReplicas": 5
      },
      "status": {
        "currentReplicas": 1,
        "desiredReplicas": 0
      }
    }
  ]
}

Ozrlz on 16 Oct 2019

/reopen
/remove-priority P3
/priority P1

seans3 on 16 Oct 2019

@seans3: Reopened this issue.

In response to this:

/reopen
/remove-priority P3
/priority P1

k8s-ci-robot on 16 Oct 2019

I just ran into this with kubectl and master both at v1.14.7.

mdlinville on 30 Oct 2019

OK, I'm going to close this in favor of the more focused issue: https://github.com/kubernetes/kubectl/issues/750

Check your kubectl version ($ kubectl version). If it is more than +/- 1 version compared to the server, then you probably have a version skew problem. Kubernetes only guarantees that kubectl will work with an APIServer that is +/- 1 version. For example: kubectl 1.14 is only guaranteed to work with APIServers version 1.13, 1.14, and 1.15. Solution: update kubectl to match your API server. Most of the complaints in this issue are solved by this step.
If you have a kubectl 1.14 version, then you have a real bug. Please follow the bug #750 for progress on the fix.

/close

seans3 on 30 Oct 2019

@seans3: Closing this issue.

In response to this:

OK, I'm going to close this in favor of the more focused issue: https://github.com/kubernetes/kubectl/issues/750

Check your kubectl version ($ kubectl version). If it is more than +/- version compared to the server, then you probably have a version skew problem. Kubernetes only guarantees that kubectl will work with an APIServer that is +/- version. Solution: update kubectl to match your API server. Most of the complaints in this issue are solved by this step.

If you have a kubectl 1.14 version, then you have a real bug. Please follow the bug #750 for progress on the fix.

/close