Katib: Standalone Katib deployment: creating a new experiment fails due to the MutatingWebhook timing out

[kylepad]

/kind bug

Hi I am trying to setup a standalone deployment of Katib v1beta1 on GKE. It is very possible I am doing something totally wrong here, or missing something obvious and that's why I'm coming to you for help.

TL;DR: I have tried deploying katib via the deploy.sh script and also via Terraform. No matter how I deploy everything seems fine, all of katib-controller, katib-db-manager, katib-mysql and katib-ui pods are up and running, logs look clean. Then I try and submit one of the sample experiments and I get this timeout error:

$ kubectl apply -f examples/v1beta1/grid-example.yaml 
    Error from server (InternalError): error when creating "examples/v1beta1/grid-example.yaml": 
    Internal error occurred: failed calling webhook "mutating.experiment.katib.kubeflow.org": 
    Post https://katib-controller.kubeflow.svc:443/mutate-experiments?timeout=30s: context deadline exceeded

What I've tried:

I tried following all the debugging steps in #1160 (closest to this issue AFAIK) and didn't really get anywhere.
The webhook itself seems to be setup (admittedly I cant say if it's correct or not):

$ kubectl describe MutatingWebhookConfiguration katib-mutating-webhook-config
Name:         katib-mutating-webhook-config
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  admissionregistration.k8s.io/v1beta1
Kind:         MutatingWebhookConfiguration
Metadata:
  Creation Timestamp:  2020-07-09T00:24:58Z
  Generation:          1
  Resource Version:    17496
  Self Link:           /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/katib-mutating-webhook-config
  UID:                 9eaca06d-c17a-11ea-ac7b-42010a000066
Webhooks:
  Admission Review Versions:
    v1beta1
  Client Config:
    Ca Bundle:  <hidden>
    Service:
      Name:        katib-controller
      Namespace:   kubeflow
      Path:        /mutate-experiments
  Failure Policy:  Fail
  Name:            mutating.experiment.katib.kubeflow.org
  Namespace Selector:
    Match Expressions:
      Key:       control-plane
      Operator:  DoesNotExist
  Rules:
    API Groups:
      kubeflow.org
    API Versions:
      v1beta1
    Operations:
      CREATE
      UPDATE
    Resources:
      experiments
    Scope:          *
  Side Effects:     Unknown
  Timeout Seconds:  30
  Admission Review Versions:
    v1beta1
  Client Config:
    Ca Bundle: <hidden>
    Service:
      Name:        katib-controller
      Namespace:   kubeflow
      Path:        /mutate-pods
  Failure Policy:  Ignore
  Name:            mutating.pod.katib.kubeflow.org
  Namespace Selector:
    Match Labels:
      Katib - Metricscollector - Injection:  enabled
  Rules:
    API Groups:

    API Versions:
      v1
    Operations:
      CREATE
    Resources:
      pods
    Scope:          *
  Side Effects:     Unknown
  Timeout Seconds:  30
Events:             <none>

I've tried multiple Kubernetes versions from: 1.14.10-gke.45 - 1.16.9-gke.6

Some things I have not tried:

• installing via the GCP kubeflow script (the whole point is to get a standalone katib deployment)
• installing v1alpha3 (not against it, but ideally we want the newest version)

Any and all help is appreciated! 😄

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/katib | 0.94 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

@kylepad Thank you for the issue.
According to this: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites, admission webhooks v1beta1 (which we are using) should work on Kubernetes > v1.9 by default on GKE if you didn't disable it manually.

To follow this: https://github.com/kubeflow/katib/issues/1160, did you check that Kubeflow namespace has katib-metricscollector-injection=enabled label?

Deploy script should work fine, since it deletes webhooks before deploying Controller.

[kylepad]

Hey thanks for a quick response, the label is in fact set on the kubeflow namespace:

$ kubectl describe namespace kubeflow
Name:         kubeflow
Labels:       katib-metricscollector-injection=enabled
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"katib-metricscollector-injection":"enabled"},"name":"kubeflo...
Status:       Active

Resource Quotas
 Name:                       gke-resource-quotas
 Resource                    Used  Hard
 --------                    ---   ---
 count/ingresses.extensions  0     100
 count/jobs.batch            0     5k
 pods                        4     1500
 services                    4     500

No resource limits.

[andreyvelich]

@kylepad

1. Do you have something installed in your cluster besides Katib that can block webhooks?
2. Run: kubectl api-versions to check if admissionregistration.k8s.io/v1beta1 is among the results.
3. Try to expose katib-controller service (webhook server) kubectl port-forward svc/katib-controller -n kubeflow 8080:443 and run wget https://localhost:8080/mutate-experiments --no-check-certificate. Check the response, please

[andreyvelich]

@kylepad If you are using Private GKE cluster (you can check here: https://console.cloud.google.com/kubernetes, clicking on your cluster), it might be the same problem as here: https://github.com/kubernetes/kubernetes/issues/79739.

[kylepad]

Hey thanks for your help! We are using a Private GKE Cluster with a VPC. Creating a firewall rule allowing traffic via TCP:8443 and specifying the source range as the master plane CIDR fixed the problem 👍

$ kubectl apply -f examples/v1beta1/grid-example.yaml 
experiment.kubeflow.org/grid-example created

Might be a good idea to document this somewhere. Could save somebody else a lot of time 🤷‍♂️