Kube-state-metrics: Cannot list resource "ingresses" in API group "extensions" after permission adjustment

[kehao95]

/kind bug

What happened:

Recent permission adjustment seem to caused the issue

kube-state-metrics log output:

 kube-state-metrics E1020 20:01:05.097248       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:mo
 nitoring:kube-state-metrics" cannot list resource "ingresses" in API group "extensions" at the cluster scope
 kube-state-metrics E1020 20:01:06.099540       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:mo
 nitoring:kube-state-metrics" cannot list resource "ingresses" in API group "extensions" at the cluster scope
 kube-state-metrics E1020 20:01:07.103837       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:mo
 nitoring:kube-state-metrics" cannot list resource "ingresses" in API group "extensions" at the cluster scope
 kube-state-metrics E1020 20:01:08.105993       1 reflector.go:156] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:108: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:mo
 nitoring:kube-state-metrics" cannot list resource "ingresses" in API group "extensions" at the cluster scope

How to reproduce it (as minimally and precisely as possible):
I am using kube-prometheus project to manage the stack deployment with no version pinning. I sync with upstream and update cluster regularly, living at the edge.
This issue happened after I updated from 10/7/2020 to 10/19/2020.

I noticed this commit might caused this issue.

Manually add resources:ingresses back to apiGroups:extensions in kube-state-metrics-clusterRole.yaml will hush the issue.

  - apiGroups:
    - extensions
    resources:
    - daemonsets
    - deployments
    - replicasets
+   - ingresses

Anything else we need to know?:

my jsonnetfile.lock.json file

{
  "version": 1,
  "dependencies": [
    {
      "source": {
        "git": {
          "remote": "https://github.com/brancz/kubernetes-grafana.git",
          "subdir": "grafana"
        }
      },
      "version": "d7c1a53462ecd533593c60e5277b92fbf7ea7623",
      "sum": "8OnIwMhzWtgoWYHNrDlkzUAMr/CPsWKauYEv0vnH1zs="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/etcd-io/etcd.git",
          "subdir": "Documentation/etcd-mixin"
        }
      },
      "version": "15f507f6b543888eb721ead2fa3c1a72ad0b6d5c",
      "sum": "L+PGlPK9mykGCJ9TIoEWdhMBjz+9lKuQ4YZ8fOeP9sk="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/grafana/grafonnet-lib.git",
          "subdir": "grafonnet"
        }
      },
      "version": "72462d419ecb43f31c8ec85ee9500c62b2a4e229",
      "sum": "sFuToRtBG3qbNPTxKRACGbvBzyS9/Pv8pf9OlM/yoaE="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/grafana/jsonnet-libs.git",
          "subdir": "grafana-builder"
        }
      },
      "version": "b5e45051995755ea373ea67642f8e5f54fcb8dd7",
      "sum": "mD0zEP9FVFXeag7EaeS5OvUr2A9D6DQhGemoNn6+PLc="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/ksonnet/ksonnet-lib.git",
          "subdir": ""
        }
      },
      "version": "0d2f82676817bbf9e4acf6495b2090205f323b9f",
      "sum": "h28BXZ7+vczxYJ2sCt8JuR9+yznRtU/iA6DCpQUrtEg=",
      "name": "ksonnet"
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/kubernetes-monitoring/kubernetes-mixin.git",
          "subdir": ""
        }
      },
      "version": "8a98e9c6fab000ef090b8d313292043696a8b3bb",
      "sum": "btFPZfE2paWZdvLtFwv4gfDoygj1axt7Q4ACGSdeuJ8="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/kubernetes-monitoring/kubernetes-mixin.git",
          "subdir": "lib/promgrafonnet"
        }
      },
      "version": "8a98e9c6fab000ef090b8d313292043696a8b3bb",
      "sum": "zv7hXGui6BfHzE9wPatHI/AGZa4A2WKo6pq7ZdqBsps="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/kubernetes/kube-state-metrics.git",
          "subdir": "jsonnet/kube-state-metrics"
        }
      },
      "version": "ad65f2bf3d55a998e60e90743e3cb4418db51e06",
      "sum": "ySP+bI2ZMLPt/sguSh9WrwI5H5dasaNFRE8Uo9PcZrI="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/kubernetes/kube-state-metrics.git",
          "subdir": "jsonnet/kube-state-metrics-mixin"
        }
      },
      "version": "ad65f2bf3d55a998e60e90743e3cb4418db51e06",
      "sum": "Yf8mNAHrV1YWzrdV8Ry5dJ8YblepTGw3C0Zp10XIYLo="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus-operator/kube-prometheus.git",
          "subdir": "jsonnet/kube-prometheus"
        }
      },
      "version": "3b38fcc3509cfb779c09c6d32c6b408b3f284ab2",
      "sum": "dZQGAlVZyL7wgI2/QmPs+ptLNPB/H065Nr1BZtSUEtE="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus-operator/prometheus-operator.git",
          "subdir": "jsonnet/mixin"
        }
      },
      "version": "59a2a22dba363be52d3ce25f6e130214cc50ed2c",
      "sum": "+Q45oBC7O8g7KQOaiKhGglwndAMWRlLTR94KUI8Q1Ko="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus-operator/prometheus-operator.git",
          "subdir": "jsonnet/prometheus-operator"
        }
      },
      "version": "cd331ce9bb58bb926e391c6ae807621cb12cc29e",
      "sum": "nM1eDP5vftqAeQSmVYzSBAh+lG0SN6zu46QiocQiVhk="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus/node_exporter.git",
          "subdir": "docs/node-mixin"
        }
      },
      "version": "9c9c63630572ac706dcd0bd0e79dc03179f6d02c",
      "sum": "GE8EJdQvnaaj31avW1OaJGY6xP+Vd6MqMu7/GgudRDQ="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus/prometheus.git",
          "subdir": "documentation/prometheus-mixin"
        }
      },
      "version": "0a7fdd3b76960808c3a91d92267c3d815c1bc354",
      "sum": "CGxvaHkP7z/gnsLB/8Imvt/AnW+9nJUnTcL+fvIAZUs=",
      "name": "prometheus"
    }
  ],
  "legacyImports": false
}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.15-gke.500", GitCommit:"f7db507aabec3b78cba0c27c616f4974213db6fd", GitTreeState:"clean", BuildDate:"2020-09-21T09:20:41Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

• Kube-state-metrics image version
  quay.io/coreos/kube-state-metrics:v1.9.7

[tariq1890]

I agree that we should add it back. Maintain compatibility for both apiGroups

[tariq1890]

Would you like to contribute a PR for this @kehao95 ?

/help
/good-first-issue

[lilic]

You can use a release branch for jsonnet to pin and not master in kube-prometheus , so I would suggest that instead. Not sure we need to add permissions for an API we are no longer using in master. Can you explain the reason a bit more @tariq1890? @kehao95 is using 1.9.7 image of kube-state-metrics but files from master, due to breaking changes this will of course not work.

Instead, this is my fault for not pinning this to the release-2.0 branch in kube-promethus, will do the fix unless you want to? https://github.com/prometheus-operator/kube-prometheus/blob/895dcbe1a7c41f112b1de2efd121947337e1ffef/jsonnet/kube-prometheus/jsonnetfile.json#L60-L66 :)

cc @brancz

[tariq1890]

Thanks for the clarification @lilic.

I thought it'd be better maintain compatibility for apiGroups as many folks will still be on the ingress under extensions api.

But if our minimum supported K8s does not include extensions, then we should not go ahead with this PR

[lilic]

@tariq1890 the support was removed when we migrated away from them which is specified in this issue https://github.com/kubernetes/kube-state-metrics/issues/1249 the logs were very noisy when I had it deployed on my cluster.

Only time this issue makes sense is if we revert the change fixed by https://github.com/kubernetes/kube-state-metrics/issues/1249. Support for older ingress API group still works in 1.9.x releases so folks who do not have new APIs can use that, and 2.0 was not cut yet, by the time it will be there will be already k8s 1.20 which means that API will have been deprecated in two releases. WDYT?

[lilic]

https://github.com/prometheus-operator/kube-prometheus/pull/736 @kehao95 opened PR to pin against latest stable kube-state-metrics release which should solve your problem :)

[brancz]

I agree with lili that on master we should only have support for ingress in the networking API. And even regardless of opinion, we can only support one version, because of how kube-state-metrics works.

As lili mentioned, the latest v1.9.x supports the old ingress API, so until people migrate to a Kubernetes version with the new ingress API available, they should use that.

[lilic]

PR in kube-prometheus was updated to pin to release-1.9 of kube-state-metrics so closing this issue, if you have any more problems feel free to open a new issue, thanks!