Kfserving: Triton server Load-balancing & Autoscaling, on prem.

[ontheway16]

Hello, I am looking for a way to establish a load balancing and autoscaling setup, on premise. I want to use Triton server at multiple nodes, containing multiple GPUs.
My kubernetes test cluster is containing 3 physical nodes, one is master and other two is worker, It is installed with kfserving, knative and istio, also installed a Triton predictor, but after spending weeks, now I am not sure if load-balancing and autoscaling is possible at all, on premise.
There is no guide etc on net about how to setup an on prem configuration, almost all existing samples are using gs:// mounts, and load balancing capabilities of cloud resources.
Should I give up ?

[yuzisun]

@ontheway16 KFServing supports on Prem setup, the examples are using gs:// because those buckets are public and easier for users to get started, if you can setup PVCs on prem that works too and here is the example. For load balancing you can take a look at Metalb.

[ontheway16]

Thank you very much, @yuzisun. Does it matter if I never need an external IP for access from outside (wan)? Since object detection inference requests will be send from a script in same LAN, may be from a pod in the cluster.

[yuzisun]

Kfserving supports calling from in cluster too, there is an in cluster url exposed for each inference service. If you are in a pod in the cluster you can call the service with http://$service.$namespace.svc.cluster.local

[alokumar1]

I was having the same issue. I have deployed the triton-server using custom predictor. I was able to call triton server using predictor pod IP and triton client script
but when I curl kfserving url like this as mentioned in doc

# Other environment(On Prem)
export INGRESS_HOST=$(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

then
curl -v http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/densenet_onnx:predict -d @./input.jpg

It throws 404 error

Kfserving supports calling from in cluster too, there is an in cluster url exposed for each inference service. If you are in a pod in the cluster you can call the service with http://$service.$namespace.svc.cluster.local