Kfserving: controller is binding to a priviliged port 443

[robin2008]

/kind bug

What steps did you take and what happened:
I install Kfserving on openshift 4.3 with the steps in guides, and in the end, found that kfserving pod cannot be up.

The logs as below:

oc logs kfserving-controller-manager-0 -c manager
{"level":"info","ts":1598011267.176977,"logger":"entrypoint","msg":"Setting up client for manager"}
{"level":"info","ts":1598011267.177478,"logger":"entrypoint","msg":"Setting up manager"}
{"level":"info","ts":1598011270.674518,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"}
{"level":"info","ts":1598011270.674906,"logger":"entrypoint","msg":"Registering Components."}
{"level":"info","ts":1598011270.6749208,"logger":"entrypoint","msg":"Setting up scheme"}
{"level":"info","ts":1598011270.6750262,"logger":"entrypoint","msg":"Setting up Knative scheme"}
{"level":"info","ts":1598011270.6751091,"logger":"entrypoint","msg":"Setting up Istio schemes"}
{"level":"info","ts":1598011270.6751914,"logger":"entrypoint","msg":"Setting up controller"}
{"level":"info","ts":1598011270.7700293,"logger":"entrypoint","msg":"setting up webhook server"}
{"level":"info","ts":1598011270.7700856,"logger":"entrypoint","msg":"registering webhooks to the webhook server"}
{"level":"info","ts":1598011270.7701104,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-pods"}
{"level":"info","ts":1598011270.7701356,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-inferenceservices"}
{"level":"info","ts":1598011270.7701447,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-inferenceservices"}
{"level":"info","ts":1598011270.7701523,"logger":"entrypoint","msg":"Starting the Cmd."}
{"level":"info","ts":1598011270.7705138,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"kfserving-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1598011270.770591,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1598011270.7705498,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1598011270.7714868,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1598011270.7718337,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"error","ts":1598011270.771932,"logger":"entrypoint","msg":"unable to run the manager","error":"listen tcp :443: bind: permission denied","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nmain.main\n\t/go/src/github.com/kubeflow/kfserving/cmd/manager/main.go:100\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

I can workaround it with adding anyuid scc to kfserving namespaces.

oc adm policy add-scc-to-group anyuid system:serviceaccounts:kfserving-system

What did you expect to happen:
installs kfserving successfully w/o extra scc. Maybe use other unprivileged port, like 9443.

Anything else you would like to add:

Environment:

• Istio Version: 1.6.8
• Knative Version: 0.13.3
• KFServing Version: 0.4.0
• Kubeflow version:
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube version:
• Kubernetes version: (use kubectl version):

oc version
Client Version: 4.3.10-202003300415-6a90d0a
Server Version: 4.3.23
Kubernetes Version: v1.16.2+18cfcc9

• OS (e.g. from /etc/os-release):

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.58 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.58 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[yuzisun]

@robin2008 thanks for the issue! I think there are a few places need to fix
https://github.com/kubeflow/kfserving/blob/465a4e6a7499b9be05de64f7fd2c8f4e98cbd0f3/config/webhook/service.yaml#L9
https://github.com/kubeflow/kfserving/blob/465a4e6a7499b9be05de64f7fd2c8f4e98cbd0f3/config/webhook/service.yaml#L9

When constructing the controller manager we need to set the webhook port like 9443
https://github.com/kubeflow/kfserving/blob/master/cmd/manager/main.go#L65

mgr, err := manager.New(cfg, manager.Options{MetricsBindAddress: metricsAddr, Port: 9443})

[decewei]

@robin2008 thanks for the issue! I think there are a few places need to fix

https://github.com/kubeflow/kfserving/blob/465a4e6a7499b9be05de64f7fd2c8f4e98cbd0f3/config/webhook/service.yaml#L9

https://github.com/kubeflow/kfserving/blob/465a4e6a7499b9be05de64f7fd2c8f4e98cbd0f3/config/webhook/service.yaml#L9

When constructing the controller manager we need to set the webhook port like 9443
https://github.com/kubeflow/kfserving/blob/master/cmd/manager/main.go#L65

mgr, err := manager.New(cfg, manager.Options{MetricsBindAddress: metricsAddr, Port: 9443})

I'm trying to fix this in v0.3.0, as this is a roadbloack for us:
So we changed
1 targetPort: 443 to 9443 in https://github.com/kubeflow/kfserving/blob/0d64937ae90b2910d93a20c07ecb3748f0314413/install/v0.3.0/kfserving.yaml#L7064
2 Add https://github.com/kubeflow/kfserving/blob/0d64937ae90b2910d93a20c07ecb3748f0314413/cmd/manager/main.go#L54

mgr, err := manager.New(cfg, manager.Options{MetricsBindAddress: metricsAddr, Port: 9443})

Now KFServing controller is running, but when I tried to deploy an inference service, we got the following

Error from server (InternalError): error when creating "iae.yaml": Internal error occurred: failed calling webhook "inferenceservice.kfserving-webhook-server.defaulter": Post "https://kfserving-webhook-server-service.kfserving-system.svc:443/mutate-inferenceservices?timeout=30s": dial tcp 10.99.241.116:443: connect: connection refused

And when I check the webhook service, the target port remains at 443

Name:              kfserving-webhook-server-service
Namespace:         kfserving-system
Labels:            <none>
Annotations:       <none>
Selector:          control-plane=kfserving-controller-manager
Type:              ClusterIP
IP:                x.x.x.x
Port:              <unset>  443/TCP
TargetPort:        443/TCP
Endpoints:         x.x.x.x:443
Session Affinity:  None
Events:            <none>

[decewei]

For people using v0.3.0, the webhook service file is here: /config/default/webhook/service.yaml

Change targeport 443 to 9443 in addition to the above changes work.

[adrian555]

Thanks @decewei, could you please assign the issue to yourself and push a PR if you have done the testing? We have done similar thing with KfServing installed through Kubeflow but had no time to test this standalone case yet. Also cc @pugangxa

[decewei]

Thanks @decewei, could you please assign the issue to yourself and push a PR if you have done the testing? We have done similar thing with KfServing installed through Kubeflow but had no time to test this standalone case yet. Also cc @pugangxa

I'm happy to do a PR and testing since it's mostly done for this branch. But I'm not a member of this repo, not sure how I can or should assign an issue to myself. Let me know.

[yuzisun]

@decewei you do not need to be in the org to create the PR, feel free to send over the PR.

[pugangxa]

Thanks @decewei , I am doing the similar things, let me know if you encounter some issue or do not have time to do this.

[pugangxa]

And in my test, released versions' port cannot be changed because we need rebuild manager but released version just use images from "gcr.io/kfserving/kfserving-controller:$TAG". So I can only change the target port from /config/webhook/service.yaml and main.go and then make deploy-dev to make it work(after code merged make deploy should work too)
@yuzisun Do you think we need change /install/${TAG}/kfserving.yaml and rebuild gcr.io/kfserving/kfserving-controller:$TAG for all release versions?

[pugangxa]

There's another issue during reconciling inferenceservice after the change, the log shows:

{"level":"info","ts":1602146302.2823198,"logger":"PredictorReconciler","msg":"Reconciling Predictor","PredictorSpec":{"sklearn":{"storageUri":"gs://kfserving-samples/models/sklearn/iris","runtimeVersion":"v0.4.0","name":"","resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"1","memory":"2Gi"}}},"containerConcurrency":0}}
{"level":"info","ts":1602146302.2828405,"logger":"KsvcReconciler","msg":"Creating knative service","namespace":"kfserving-testing","name":"sklearn-iris-predictor-default"}
{"level":"error","ts":1602146315.8249855,"logger":"v1beta1Controllers.InferenceService","msg":"Failed to reconcile","reconciler":"<*components.Predictor Value>","Name":"sklearn-iris","error":"fails to reconcile predictor: services.serving.knative.dev \"sklearn-iris-predictor-default\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","errorVerbose":"services.serving.knative.dev \"sklearn-iris-predictor-default\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>\nfails to reconcile predictor\ngithub.com/kubeflow/kfserving/pkg/controller/v1beta1/inferenceservice/components.(*Predictor).Reconcile\n\tgithub.com/kubeflow/kfserving/pkg/controller/v1beta1/inferenceservice/components/predictor.go:107\ngithub.com/kubeflow/kfserving/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\tgithub.com/kubeflow/kfserving/pkg/controller/v1beta1/inferenceservice/controller.go:88\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:88\nruntime.goexit\n\truntime/asm_amd64.s:1373","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tgithub.com/go-logr/[email protected]/zapr.go:128\ngithub.com/kubeflow/kfserving/pkg/controller/v1beta1/inferenceservice.(*InferenceServiceReconciler).Reconcile\n\tgithub.com/kubeflow/kfserving/pkg/controller/v1beta1/inferenceservice/controller.go:89\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:88"}

Investigated and seems need update RBAC to add permission for below resource to fix this issue.

- apiGroups:
  - serving.knative.dev
  resources:
  - services/finalizers

My openshift cluster is not stable and will test this later.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/engprod | 0.80 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[yuzisun]

And in my test, released versions' port cannot be changed because we need rebuild manager but released version just use images from "gcr.io/kfserving/kfserving-controller:$TAG". So I can only change the target port from /config/webhook/service.yaml and main.go and then make deploy-dev to make it work(after code merged make deploy should work too)
@yuzisun Do you think we need change /install/${TAG}/kfserving.yaml and rebuild gcr.io/kfserving/kfserving-controller:$TAG for all release versions?

No, releases are immutable. Once this is fixed it will be available for the upcoming 0.5 release.