Katib: Failed calling webhook when running random example experiment

[Felihong]

Hi there,

my Katib failed to submit the the random algorithm example https://github.com/kubeflow/katib/blob/master/examples/v1alpha3/random-example.yaml:

Internal error occurred: 
failed calling webhook "mutating.experiment.katib.kubeflow.org": 
Post https://katib-controller.kubeflow.svc:443/mutate-experiments?timeout=30s: 
ssh: rejected: connect failed (Connection refused)

The same experiment worked just fine before several days and I didn't change anything. All deployments kati-controller, katib-db-manager etc. are in healthy states. Any ideas what is going wrong here?

Thanks.

Environment:

• Kubeflow version: v1.0
• Kubernetes version: (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", 
GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| bug | 0.61 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

Can you check your webhook status?
kubectl get MutatingWebhookConfiguration

[Felihong]

Hi @andreyvelich , thanks for your reply.

The katib-mutating-webhook-config is listed in Webhook configurations:

kubectl get MutatingWebhookConfiguration

NAME                                                 CREATED AT
admission-webhook-mutating-webhook-configuration     2020-03-08T16:34:33Z
cert-manager-webhook                                 2020-03-08T16:32:44Z
inferenceservice.serving.kubeflow.org                2020-03-08T16:36:05Z
istio-sidecar-injector                               2020-03-08T16:32:21Z
katib-mutating-webhook-config                        2020-03-08T16:37:42Z
pod-ready.config.common-webhooks.networking.gke.io   2020-03-08T16:30:29Z
seldon-mutating-webhook-configuration-kubeflow       2020-03-08T16:35:22Z
webhook.serving.knative.dev                          2020-03-08T16:34:45Z

Below the yaml file:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  generation: 1
  name: katib-mutating-webhook-config
  resourceVersion: "5990"
  selfLink: /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/katib-mutating-webhook-config
  uid: [ID]
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: [Certificat]
    service:
      name: katib-controller
      namespace: kubeflow
      path: /mutate-experiments
  failurePolicy: Fail
  name: mutating.experiment.katib.kubeflow.org
  namespaceSelector:
    matchExpressions:
    - key: control-plane
      operator: DoesNotExist
  rules:
  - apiGroups:
    - kubeflow.org
    apiVersions:
    - v1alpha3
    operations:
    - CREATE
    - UPDATE
    resources:
    - experiments
    scope: '*'
  sideEffects: Unknown
  timeoutSeconds: 30
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: [Certificat]
    service:
      name: katib-controller
      namespace: kubeflow
      path: /mutate-pods
  failurePolicy: Ignore
  name: mutating.pod.katib.kubeflow.org
  namespaceSelector:
    matchLabels:
      katib-metricscollector-injection: enabled
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: Unknown
  timeoutSeconds: 30

Thanks for your help!

[andreyvelich]

Can you describe your Kubeflow namespace, check it you have label katib-metricscollector-injection=enabled there?
Also, where did you see this error, that you showed above?

If it doesn't help, try to reinstall Katib deployment.
You can use this script: https://github.com/kubeflow/katib/blob/master/scripts/v1alpha3/deploy.sh to install Katib.
This will delete validating and mutating webhook.
Katib controller will create new, when you deploy it.

[Felihong]

Hi @andreyvelich , I re-deployed Katib and now it works fine. Thanks a lot for your help!

The error was occurred as a pop-out on top of the page, when I tried to submit the experiment yaml file. I also checked the kubeflow namespace and katib-metricscollector was injected:

kubectl describe namespace kubeflow
Name:         kubeflow
Labels:       control-plane=kubeflow
              katib-metricscollector-injection=enabled

Any ideas what may cause this happen?

[andreyvelich]

@Felihong Maybe on your cluster you somehow modified this webhook.
How did you install Kubeflow?

[Felihong]

@andreyvelich actually not that I remember :(

My kubeflow cluster is hosted on GCP and I deployed v1.0 using the official config URL https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml

[andreyvelich]

@Felihong Ok, if you have this problem again, let me know.