Service-catalog: extension-apiserver-authentication not found

[jicowan]

I tried installing service catalog onto a k8s v1.10 cluster from the helm chart svc-cat/catalog. The API server fails to start with the following error:

01:31:44.138448       1 authentication.go:232] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
Error: configmaps "extension-apiserver-authentication" not found

There is a configmap in the kube-system namespace called catalog.v1 but so far as I can tell the chart didn't create a configmap called extension-apiserver-authentication.

[jboyd01]

Helm should be creating the rbac to allow Catalog API server to read this. Within charts/catalog/templates/rbac.yaml:

- apiVersion: {{template "rbacApiVersion" . }}
  kind: RoleBinding
  metadata:
    name: "servicecatalog.k8s.io:apiserver-authentication-reader"
    namespace: kube-system
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: extension-apiserver-authentication-reader
  subjects:
  - apiGroup: ""
    kind: ServiceAccount
    name: "{{ .Values.apiserver.serviceAccount }}"
    namespace: "{{ .Release.Namespace }}"

Can you see if this was created?

kubectl get roles -n kube-system extension-apiserver-authentication-reader -o yaml

kubectl get rolebinding servicecatalog.k8s.io:apiserver-authentication-reader -n kube-system -o yaml

If you remove catalog (helm delete --purge catalog) and redeploy adding --debug to the helm install command, you'll get more verbose output that might help narrow in on the issue

[jicowan]

Yes, the role and the role binding were created. It also created 2 service accounts in the catalog namespaces for API server and the controller manager. Both pods are failing to start. I am trying to install this on an AWS EKS cluster.

[jicowan]

Apparently EKS doesn't support custom API deployments and service catalog yet. Closing for now.

[jboyd01]

@jicowan - we've been hearing of some other users unsuccessfully trying to use Catalog on EKS. Can you provide a quick summary on what you found out?

[MHBauer]

If the configmap extension-apiserver-authentication does not exist, the features for api-aggregation are not enabled.

[incognick]

I'm running into this same issue trying to install the stable/metrics-server on EKS. Same error Could not create the API server: configmaps "extension-apiserver-authentication" not found

[iamakulov]

A relevant issue for metrics-server on EKS: https://github.com/kubernetes-incubator/metrics-server/issues/80

[zsolt-keseru-epam]

Just ran into this issue yesterday, and now found that EKS announced an update on this today:
https://aws.amazon.com/blogs/opensource/horizontal-pod-autoscaling-eks/
TLDR: Good news: Newly created clusters should work.

[jwitrick]

Please read their blog carefully. It looks like to get around this issue you have to either:
1) deploy a new EKS cluster
or
2) Wait for AWS to patch all existing clusters over the next few weeks.

[bobhenkel]

We stood up a new cluster which is on eks.2 and did a ARK restore on it. We were told that eks1 would not be upgraded and we would need to create a new cluster and migrate to it if we wanted to get HPA up and running. This was before they actually implemented eks.2 so maybe they have figured out how to at some point upgrade eks.1. Personally, I wouldn't hold my breath on the upgrade path from eks.1 to eks.2.

[Vikash082]

I am facing similar issue but not AWS. I am running the cluster in VM. What is the resolution for this ?

I am getting error, while accessing this:

$ kubectl get rolebinding servicecatalog.k8s.io:apiserver-authentication-reader -n kube-system -o yaml
Error from server (NotFound): rolebindings.rbac.authorization.k8s.io "servicecatalog.k8s.io:apiserver-authentication-reader" not found

[bergbrains]

I get the same error, @Vikash082. Did you find out how to add the group or whatever needs to be done?

[Vikash082]

@bergbrains if i remember correctly, i had to upgrade the k8s version both client and server.