Kfserving: CustomResourceDefinition "inferenceservices.serving.kubeflow.org" is invalid, during installation of v0.4.0

[ShilpaGopal]

/kind bug

What steps did you take and what happened:
I am trying to install standalone kfserving on minikube. I have installed Istio, knative and cert-manger. When I install kfserving
I notice the log saying "The CustomResourceDefinition "inferenceservices.serving.kubeflow.org" is invalid".

namespace/kfserving-system created
mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org created
role.rbac.authorization.k8s.io/leader-election-role created
clusterrole.rbac.authorization.k8s.io/kfserving-manager-role created
clusterrole.rbac.authorization.k8s.io/kfserving-proxy-role created
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/kfserving-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/kfserving-proxy-rolebinding created
configmap/inferenceservice-config created
secret/kfserving-webhook-server-secret created
service/kfserving-controller-manager-metrics-service created
service/kfserving-controller-manager-service created
service/kfserving-webhook-server-service created
statefulset.apps/kfserving-controller-manager created
certificate.cert-manager.io/serving-cert created
issuer.cert-manager.io/selfsigned-issuer created
validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org created
The CustomResourceDefinition "inferenceservices.serving.kubeflow.org" is invalid:
* spec.validation.openAPIV3Schema.properties[spec].properties[canary].properties[explainer].properties[custom].properties[container].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property
* spec.validation.openAPIV3Schema.properties[spec].properties[canary].properties[predictor].properties[custom].properties[container].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property
* spec.validation.openAPIV3Schema.properties[spec].properties[canary].properties[transformer].properties[custom].properties[container].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property
* spec.validation.openAPIV3Schema.properties[spec].properties[default].properties[predictor].properties[custom].properties[container].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property
* spec.validation.openAPIV3Schema.properties[spec].properties[default].properties[transformer].properties[custom].properties[container].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property
* spec.validation.openAPIV3Schema.properties[spec].properties[default].properties[explainer].properties[custom].properties[container].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property

And the pod kfserving-controller-manager-0 ends up in CrashLoopBackOff.

_kubectl logs kfserving-controller-manager-0 -n kfserving-system -c manager_

{"level":"info","ts":1596828783.325014,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kfserving-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1596828783.3255389,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1596828783.4208002,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1596828783.42329,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1596828783.4235485,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":443}
{"level":"info","ts":1596828783.6203442,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"error","ts":1596828785.7815657,"logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"InferenceService.serving.kubeflow.org","error":"no matches for kind \"InferenceService\" in version \"serving.kubeflow.org/v1alpha2\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:88\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:165\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:198\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElectionRunnables.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:477"}
{"level":"error","ts":1596828785.781721,"logger":"entrypoint","msg":"unable to run the manager","error":"no matches for kind \"InferenceService\" in version \"serving.kubeflow.org/v1alpha2\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nmain.main\n\t/go/src/github.com/kubeflow/kfserving/cmd/manager/main.go:100\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

Anything else you would like to add:
With the same setup, I tried installing kfserving 0.2.2, It was successfully installed.

Environment:

• Istio Version: 1.6.7
• Knative Version: v0.16.0
• KFServing Version: v0.4.0
• Kubeflow version: No kubeflow. Standalone installation of kfserving
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube version: v1.12.2
• Kubernetes version: (use kubectl version): v1.18.3
• OS (e.g. from /etc/os-release): OS X

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.87 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[yuzisun]

@ShilpaGopal I think it is known issue that KFServing does not work on kubernetes 1.18, can you try 1.17 ?

[yuzisun]

Oh you might be on kubernetes 1.15? Check the instruction here https://github.com/kubeflow/kfserving#standalone-kfserving-installation.

[ShilpaGopal]

@yuzisun Thanks for your response. It's working fine for Kubernetes version v1.17.9