Kfserving: Manage virtual service route for each model in for multi-model InferenceService

[yuzliu]

When an InferenceService can load multiple models when can we mark an InferenceService ready?
Approach 1: An InferenceService is ready when all the models are ready
pros:
1) k8s can help reject requests coming into this pod as the pod won't be ready

cons:
1) If there are 1000 models in a pod, if one model is not ready then all other models can not receive traffic
2) If a large model need to be loaded, then it will blocks traffic for all other models

Approach 2: An InferenceService is ready as long as an empty model server is up
pros:
1) one model that is not ready won't block the traffic to other models

cons:
1) when an InferenceService need to be scaled up, users may expect that a new replica is only ready when all models are ready

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/engprod | 0.59 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| kind/question | 0.70 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[njhill]

I think we talked about this in the last MMS discussion I was in. I don't think there's a great answer but the only practical approach I can think of given other constraints would be:

• New pods to become ready only once all of the models that were present in the _initial_ copy of the configmap read upon startup (minus any removed concurrently but not including any added)
• New TrainedModels become ready/available (controller status update) only once they are available (loaded) in all pods

Though for both of these there might be more to think about for failure cases, especially if the same model loads successfully in some pods but fails in others.

[yuzisun]

As discussed in the meeting, for the new trained model we can send probes to all the endpoints(pods) in trained model controller, only if they are all successful we then create the virtual service route to the new trained model, in this way we can make sure requests to the new trained model are successful as soon as the route presents otherwise user gets 404 if they send requests before models are ready in all the replicas.

[njhill]

@yuzisun yep I think that's consistent with my suggestion, where "TrainedModels become ready/available" corresponds to "create the virtual service route to the new trained model"