Keda: Apiservice (v1beta1.external.metrics.k8s.io) status FailedDiscoveryCheck

[Anbu2506]

i tried to setup keda using helm , upon doing that i got an status error "(v1beta1.external.metrics.k8s.io) status FailedDiscoveryCheck" . keda has issue creating HPA for scaledobjects.

status:
Conditions:
Last Transition Time: 2020-01-30T12:35:04Z
Message: no response from https://10.233.46.104:443: Get https://10.233.46.104:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Reason: FailedDiscoveryCheck
Status: False
Type: Available
Events:

[zroubalik]

So KEDA is not able to create HPA for your ScaledObject?

Which k8s version are you using?

[Anbu2506]

@zroubalik kindly find the details below and am using calico as cni solution,

kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:24:33Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

[Anbu2506]

@zroubalik below are logs of keda-operator pod ,

{"level":"error","ts":1580440365.2607787,"logger":"controller_scaledobject","msg":"Failed to create new HPA resource","Request.Namespace":"default","Request.Name":"rabbitmq-consumer","HPA.Namespace":"default","HPA.Name":"keda-hpa-rabbitmq-consumer","error":"error getting scaler for trigger #0: error parsing rabbitmq metadata: no host setting given","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/home/ahmed/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/kedacore/keda/pkg/controller/scaledobject.(ReconcileScaledObject).reconcileDeploymentType\n\tkeda/pkg/controller/scaledobject/scaledobject_controller.go:203\ngithub.com/kedacore/keda/pkg/controller/scaledobject.(ReconcileScaledObject).Reconcile\n\tkeda/pkg/controller/scaledobject/scaledobject_controller.go:146\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/ahmed/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/ahmed/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/ahmed/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/ahmed/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/ahmed/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/ahmed/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580440365.2608767,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"scaledobject-controller","request":"default/rabbitmq-consumer","error":"error getting scaler for trigger #0: error parsing rabbitmq metadata: no host setting given","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/home/ahmed/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/ahmed/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/ahmed/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/ahmed/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/ahmed/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/ahmed/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/ahmed/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}

[zroubalik]

@Anbu2506 thanks for the feedback. What I can see in the log is that you might have a mistake in the scaler config:
error getting scaler for trigger #0: error parsing rabbitmq metadata: no host setting given

[ghost]

I am facing the same issue with the apiservice. The error appears just after deploying keda (no need to create any scaledobject to recreate the issue). I have installed it using the yaml files and the helm chart with same results.

Conditions:
Last Transition Time: 2020-02-04T09:35:32Z
Message: no response from https://172.31.32.4:6443: Get https://172.31.32.4:6443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Reason: FailedDiscoveryCheck
Status: False
Type: Available

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.17", GitCommit:"bdceba0734835c6cb1acbd1c447caf17d8613b44", GitTreeState:"clean", BuildDate:"2020-01-17T23:10:13Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: keda-operator
    namespace: keda
  version: v1beta1
  versionPriority: 100

$ kubectl get svc keda-operator -o yaml
apiVersion: v1
kind: Service
metadata:
  name: keda-operator
  namespace: keda
spec:
  clusterIP: 172.31.50.64
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 6443
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: keda-operator

$ kubectl logs -f keda-operator-cc8778dfb-pczdm -c keda-operator-metrics-apiserver -n keda
I0131 15:02:29.414181       1 provider.go:46] keda_metrics_adapter/provider "msg"="starting"  
I0131 15:02:29.414282       1 main.go:84] keda_metrics_adapter "msg"="starting adapter..."  
I0131 15:02:29.668258       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0131 15:02:30.072932       1 secure_serving.go:116] Serving securely on [::]:6443

[zroubalik]

@pmorillas-copado thanks for the input, where exactly do you see the error? And if you try to deploy ScaledObject anyway, do you see any problems during the deployment?

[Anbu2506]

@zroubalik Keda Metrics apiserver status showing false(FailedDiscoveryCheck). If i describe it am getting an error " Message: no response from https://10.233.6.240:443: Get https://10.233.6.240:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)".

And am running kubernetes cluster inside a proxy. So do i have to worry about this error , if so what changes i have to make ? . Kindly help , am breaking my head for past few days. Thanks

kubectl get apiservice
NAME SERVICE AVAILABLE AGE
v1beta1.external.metrics.k8s.io keda/keda-operator-metrics-apiserver False (FailedDiscoveryCheck) 21h
v1beta1.policy Local True 3d
v1beta1.rbac.authorization.k8s.io Local True 3d
v1beta1.scheduling.k8s.io Local True 3d
v1beta1.storage.k8s.io Local True 3d
v1beta2.apps Local True 3d
v2beta1.autoscaling Local True 3d
v2beta2.autoscaling Local True 3d
lake@lake:~$

[mashrin]

Try this to check the status:

1. kubectl get apiservice --all-namespaces (Find the apiservice name for the service keda/keda-metrics-apiserver).
2. kubectl get apiservice -o yaml (Check for the status)
   Example: kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml

If the status is False, then there seems to be an issue and proxy network might be the primary reason for it.

The following steps can be a solution:

1. kubectl get services --all-namespaces
   (Find the cluster IP for the keda-metrics-apiserver and keda-operator-metrics)
2. In the /etc/kubernetes/manifests/kube-apiserver.yaml - add the cluster IPs found in the previous step in no_proxy variable.
3. sudo systemctl daemon-reload (Reload systemd manager configuration)
4. sudo systemctl restart kubelet (Restart kubelet)
5. Check the API service status and the pods now. Should work!

[Anbu2506]

Try this to check the status:

1. kubectl get apiservice --all-namespaces (Find the apiservice name for the service keda/keda-metrics-apiserver).
2. kubectl get apiservice -o yaml (Check for the status)
   Example: kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml

If the status is False, then there seems to be an issue and proxy network might be the primary reason for it.

The following steps can be a solution:

1. kubectl get services --all-namespaces
   (Find the cluster IP for the keda-metrics-apiserver and keda-operator-metrics)
2. In the /etc/kubernetes/manifests/kube-apiserver.yaml - add the cluster IPs found in the previous step in no_proxy variable.
3. sudo systemctl daemon-reload (Reload systemd manager configuration)
4. sudo systemctl restart kubelet (Restart kubelet)
5. Check the API service status and the pods now. Should work!

@mashrin you made my day.. thanks a lot

[zroubalik]

@mashrin this is great, do you think you can create a FAQ section covering this topic?
@tomkerkhove FYI^

[mashrin]

@zroubalik Sure, I will do it now.

[mashrin]

@zroubalik Done: https://github.com/kedacore/keda-docs/pull/83

Let me know if you want any edits.

[przmk0]

hi Guys,
I face exactly the same issue. One thing that is different is that I'm using Docker Desktop on MacOS. I'm using corporate network, but all other things seem to be working just fine.

1. Keda was installed using yaml manifests
2. Keda Pods are running fine:
   
3. kubectl version:
   ```Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
   Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

API log:
```I0622 09:09:11.035725       1 main.go:77] keda_metrics_adapter "msg"="KEDA Version: 1.4.1"
I0622 09:09:11.036045       1 main.go:78] keda_metrics_adapter "msg"="KEDA Commit: 1ec79ecd8e720748aaf9ca152cb11dd055889ee2"
I0622 09:09:11.036083       1 main.go:79] keda_metrics_adapter "msg"="Go Version: go1.13.3"
I0622 09:09:11.036093       1 main.go:80] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I0622 09:09:13.442514       1 provider.go:47] keda_metrics_adapter/provider "msg"="starting"
I0622 09:09:13.442736       1 main.go:96] keda_metrics_adapter "msg"="starting adapter..."
I0622 09:09:13.825607       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0622 09:09:14.319622       1 secure_serving.go:116] Serving securely on [::]:6443

here is my ScaledObject manifest:
```apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: validator-scaler
spec:
pollingInterval: 5
cooldownPeriod: 30
maxReplicaCount: 10
scaleTargetRef:
deploymentName: validator
triggers:

• type: rabbitmq
  metadata:
  # Required
  host: amqp://guest:[email protected]:5672/%2f
  queueName: testqueue
  queueLength: "20"

There're no HPAs created:
`kubectl get hpa`
`No resources found in default namespace.`

`kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml` returns:
```apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"v1beta1.external.metrics.k8s.io","app.kubernetes.io/part-of":"keda-operator","app.kubernetes.io/version":"1.4.1"},"name":"v1beta1.external.metrics.k8s.io"},"spec":{"group":"external.metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"keda-metrics-apiserver","namespace":"keda"},"version":"v1beta1","versionPriority":100}}
  creationTimestamp: "2020-06-19T14:04:34Z"
  labels:
    app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
    app.kubernetes.io/part-of: keda-operator
    app.kubernetes.io/version: 1.4.1
  name: v1beta1.external.metrics.k8s.io
  resourceVersion: "653209"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.external.metrics.k8s.io
  uid: 94e5d290-c59c-4693-a8ad-f9827220e7f6
spec:
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: keda-metrics-apiserver
    namespace: keda
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-06-22T12:15:08Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

Keda operator logs point to the issue described by @Anbu2506. Don't think hints provided by @mashrin could be applied here...

{"level":"error","ts":1592838893.306561,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"scaledobject-controller","request":"default/rabbitmq-scaledobject","error":"error getting scaler for trigger #0: error parsing rabbitmq metadata: no host setting given","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88"}

[przmk0]

Above issue was caused by invalid host setting - it needs to be resolved via Secret or ConfigMap. There is no option to provide connection string explicitly.

[FullyScaled]

We run into the same problem. The only difference is that there seems to be some issues with the certificates.

$ k describe apiservices v1beta1.external.metrics.k8s.io
Name:         v1beta1.external.metrics.k8s.io
Namespace:
Labels:       app.kubernetes.io/instance=keda
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=v1beta1.external.metrics.k8s.io
              app.kubernetes.io/part-of=keda-operator
              app.kubernetes.io/version=2.0.0
Annotations:  meta.helm.sh/release-name: keda
              meta.helm.sh/release-namespace: keda
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2021-02-02T01:50:55Z
  Resource Version:    88348376
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.external.metrics.k8s.io
  UID:                 9e0098e1-d982-4aa4-99b2-c2466d84c767
Spec:
  Group:                     external.metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            keda-operator-metrics-apiserver
    Namespace:       keda
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2021-02-02T01:50:55Z
    Message:               failing or missing response from https://100.96.3.203:6443/apis/external.metrics.k8s.io/v1beta1: Get https://100.96.3.203:6443/apis/external.metrics.k8s.io/v1beta1: remote error: tls: unknown certificate authority
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available
Events:                    <none>

Since we run on a gardener managed kubernetes cluster (azure) we cannot just change the no_proxy variable. Is this the same issue as described here?