Kfserving: aggressive probe error: dial tcp 127.0.0.1:8080: connect: connection refused

[iavinas]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

I am getting this error when seeing the logs in the Pytorch server gpu example model in kfserving v0.4.

aggressive probe error: dial tcp 127.0.0.1:8080: connect: connection refused
aggressive probe error: dial tcp 127.0.0.1:8080: connect: connection refused
....

{"level":"error","ts":"2020-10-21T08:15:01.782Z","logger":"queueproxy","caller":"network/error_handler.go:33","msg":"error reverse proxying request; sockstat: sockets: used 16\nTCP: inuse 5 orphan 1 tw 12 alloc 51 mem 5\nUDP: inuse 0 mem 2\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"3372d58","knative.dev/key":{"knative.dev/key":"default/pytorch-cifar10-gpu-predictor-default-2zprm"},"knative.dev/pod":"pytorch-cifar10-gpu-predictor-default-2zprm-deployment-858m8cs5","error":"context canceled","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/[email protected]/network/error_handler.go:33\nnet/http/httputil.(ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:259\nknative.dev/serving/pkg/queue.(appRequestMetricsHandler).ServeHTTP\n\tknative.dev/serving/pkg/queue/request_metric.go:205\nmain.proxyHandler.func1\n\tknative.dev/serving/cmd/queue/main.go:149\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2041\nknative.dev/serving/pkg/queue.ForwardedShimHandler.func1\n\tknative.dev/serving/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2041\nknative.dev/serving/pkg/http/handler.(*timeToFirstByteTimeoutHandler).ServeHTTP.func1\n\tknative.dev/serving/pkg/http/handler/timeout.go:86"}

What did you expect to happen:
I am suspecting if multiple GPU nodes are serving requests or not. It is taking almost the same time for 1 replica vs 3 replica

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Istio Version: 1.7
• Knative Version: v0.17.1
• KFServing Version: 0.4
• Kubeflow version: Usning only KFServing
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube version:
• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.401", GitCommit:"eb94c181eea5290e9da1238db02cfef263542f5f", GitTreeState:"clean", BuildDate:"2020-09-09T00:57:35Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}
• OS (e.g. from /etc/os-release): Ubuntu 20.04

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.99 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[averagua]

I have a similar issue, my main problem is that i cant request to the service. I'm trying to request it from jupyter notebook in the same namespace and avoid de IAP but doest not works. I think that is the same issue, my details are:

• Latest Kubeflow deployment on GCP, v1.12
• Custom KFServing model for simple SKlearn model
• Server run locally

!kubectl logs covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh queue-proxy
results in
{"level":"info","ts":"2020-12-16T07:07:20.577Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
{"level":"info","ts":"2020-12-16T07:07:20.577Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2020-12-16T07:07:20.578Z","logger":"queueproxy","caller":"metrics/exporter.go:142","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"bcda051","knative.dev/key":"default-profile/covid-server-53e4be7e-predictor-default-whn5l","knative.dev/pod":"covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh"}
{"level":"info","ts":"2020-12-16T07:07:20.578Z","logger":"queueproxy","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/internal/serving revision prometheus 5000000000 false 9091 false { false}}. Start the server for Prometheus exporter.","commit":"bcda051","knative.dev/key":"default-profile/covid-server-53e4be7e-predictor-default-whn5l","knative.dev/pod":"covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh"}
{"level":"info","ts":"2020-12-16T07:07:20.578Z","logger":"queueproxy","caller":"metrics/exporter.go:151","msg":"Successfully updated the metrics exporter; old config: ; new config &{knative.dev/internal/serving revision prometheus 5000000000 false 9091 false { false}}","commit":"bcda051","knative.dev/key":"default-profile/covid-server-53e4be7e-predictor-default-whn5l","knative.dev/pod":"covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh"}
aggressive probe error: dial tcp 127.0.0.1:8080: connect: connection refused

!kubectl get inferenceservices -n default-profile

NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
covid-server-53e4be7e http://covid-server-53e4be7e.default-profile.example.com True 100 4h33m
covid-server-592407c43 http://covid-server-592407c43.default-profile.example.com True 100 172m
covid-server-fe8b3e4c3 http://covid-server-fe8b3e4c3.default-profile.example.com True 100 117m

[yuzisun]

I have a similar issue, my main problem is that i cant request to the service. I'm trying to request it from jupyter notebook in the same namespace and avoid de IAP but doest not works. I think that is the same issue, my details are:

• Latest Kubeflow deployment on GCP, v1.12
• Custom KFServing model for simple SKlearn model
• Server run locally

!kubectl logs covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh queue-proxy
results in
{"level":"info","ts":"2020-12-16T07:07:20.577Z","caller":"logging/config.go:108","msg":"Successfully created the logger."}
{"level":"info","ts":"2020-12-16T07:07:20.577Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2020-12-16T07:07:20.578Z","logger":"queueproxy","caller":"metrics/exporter.go:142","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"bcda051","knative.dev/key":"default-profile/covid-server-53e4be7e-predictor-default-whn5l","knative.dev/pod":"covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh"}
{"level":"info","ts":"2020-12-16T07:07:20.578Z","logger":"queueproxy","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/internal/serving revision prometheus 5000000000 false 9091 false { false}}. Start the server for Prometheus exporter.","commit":"bcda051","knative.dev/key":"default-profile/covid-server-53e4be7e-predictor-default-whn5l","knative.dev/pod":"covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh"}
{"level":"info","ts":"2020-12-16T07:07:20.578Z","logger":"queueproxy","caller":"metrics/exporter.go:151","msg":"Successfully updated the metrics exporter; old config: ; new config &{knative.dev/internal/serving revision prometheus 5000000000 false 9091 false { false}}","commit":"bcda051","knative.dev/key":"default-profile/covid-server-53e4be7e-predictor-default-whn5l","knative.dev/pod":"covid-server-53e4be7e-predictor-default-whn5l-deployment-8bklfh"}
aggressive probe error: dial tcp 127.0.0.1:8080: connect: connection refused

!kubectl get inferenceservices -n default-profile

NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
covid-server-53e4be7e http://covid-server-53e4be7e.default-profile.example.com True 100 4h33m
covid-server-592407c43 http://covid-server-592407c43.default-profile.example.com True 100 172m
covid-server-fe8b3e4c3 http://covid-server-fe8b3e4c3.default-profile.example.com True 100

@averagua can you get the logs from the main model server container ?