Website: Kubeflow instllation issue on kubernetes cluster V1.18.6

[mmlmahesh]

Hi Team,

I'm trying to install kubeflow on my kubernetes cluster and getting the below issue.could you please look into the issue and help.

kfctl version : 1.1(latest version)

ERRO[0712] Permanently failed applying application kubeflow-apps: (kubeflow.error): Code 500 with message: Apply.Run : [error when creating "/tmp/kout657170533": CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[containerSpec].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property], error when creating "/tmp/kout657170533": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request, error when creating "/tmp/kout657170533": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: dial tcp 10.110.30.14:443: connect: connection refused] filename="kustomize/kustomize.go:270"
Error: failed to apply: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 500 with message: Apply.Run : [error when creating "/tmp/kout657170533": CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[containerSpec].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property], error when creating "/tmp/kout657170533": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request, error when creating "/tmp/kout657170533": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: dial tcp 10.110.30.14:443: connect: connection refused]
Usage:
kfctl apply -f ${CONFIG} [flags]

Flags:
--context string Optional kubernetes context to use when applying resources. Currently not used by KFDef resources.
-f, --file string Static config file to use. Can be either a local path:
export CONFIG=./kfctl_gcp_iap.yaml
or a URL:
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws.v1.0.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.0.yaml
kfctl apply -V --file=${CONFIG}
-h, --help help for apply
-V, --verbose verbose output default is false

kfctl exited with error: failed to apply: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 500 with message: Apply.Run : [error when creating "/tmp/kout657170533": CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[containerSpec].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property], error when creating "/tmp/kout657170533": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request, error when creating "/tmp/kout657170533": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: dial tcp 10.110.30.14:443: connect: connection refused]ubuntu@ip-172-31-0-165:~/dummy$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
ubuntu@ip-172-31-0-165:~/dummy$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-31-0-165 Ready master 5h1m v1.18.6
ip-172-31-40-53 NotReady 4h58m v1.18.6
ubuntu@ip-172-31-0-165:~/dummy$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:56:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
ubuntu@ip-172-31-0-165:~/dummy$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/kfctl | 0.87 |
| kind/bug | 0.86 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[DavidSpek]

This seems to be a duplicate of https://github.com/kubeflow/manifests/issues/1543, and I was able to fix it and successfully deployed on Charmed Kubernetes 1.19. Here is the pull request for the fix: https://github.com/kubeflow/manifests/pull/1558. If you edit the kfctl_istio_dex_1.1.0.yaml locally and change near the bottom:

repos:
  - name: manifests
    uri: https://github.com/DavidSpek/manifests/archive/v1.1-branch.tar.gz
  version: v1.1-branch

You should be able to deploy with kfctl apply -V -f ./kfctl_istio_dex_1.1.0.yaml

[cells2numbers]

Thanks @DavidSpek, I was running into the same error, this solved it

[jbottum]

@mmlmahesh is this issue resolved ?

[DavidSpek]

@jbottum To my knowledge it is only resolved if my repository from above is used to work around the seldon issue.

[cliveseldon]

Seldon 1.3 is compatible with k8s 1.18. However, we do this in the Helm install on the upstream source by using a new v1 CRD.

If its ok for kubeflow 1.2 to not work on k8s <=1.15 then we can ship just this in kubeflow 1.2 otherwise would need some switch for the v1 and v1beta1 versions.

[Soonmok]

thanks @DavidSpek .
I solved this issue thanks to your pull request patch.
I hope your patch will be merged into kubeflow repo soon

[kevinleewy]

Has this been incorporated into v1.2? Otherwise, is there a workaround similar to the one for v1.1 by @DavidSpek?

[DavidSpek]

@kevinleewy This has been solved in Kubeflow 1.2 as the seldon version has been updated.

[aliasmee]

+1