Serving: Exposing model version for models loaded by model_server

[shrutisharmavsco]

My use case is that I would like to access the version for the models loaded by model_server in inception_client.cc so the client can return the model version number in its response. For example, if I have two versions of the same model loaded via model_server using specific model version policy in model_config_file and I change the client code to create two different prediction requests with prediction_request.mutable_model_spec()->mutable_version()->set_value(...); set differently for both prediction requests. In this way, I can run inference on both the models but in order to set the value of prediction_request.model_spec.version from the client, I need to know which versions are loaded. How can I know that?

One potential place where I felt that this information could be available is GetModelMetadata proto However, since GetModelMetadataRequest also takes a model_spec, my understanding is that the intended use of GetModelMetadata() in PredictionService proto is to specify a model_spec with certain name and version and get the corresponding signature_def when multiple versions are loaded. Could there be a way in which we can use GetModelMetadata to also give us which various versions are loaded if we specify just the name in the model_spec?

[promiseofcake]

@chrisolston, any guidance you could provide here or insight to future plans?

[chrisolston]

Unfortunately I don't believe we currently offer a way to discover which version(s) are loaded. We're discussing internally the possibility of adding such a (very sensible :) feature, possibly via GetModelMetadata as you suggest. Stay tuned.

[promiseofcake]

Thanks @chrisolston!

[vitalyli]

Would be good to have version long number baked right into the same response object. Hopefully one extra 64bit long number is not too much of an overhead, but it is indeed very useful to have.

[cchung100m]

Hi @chrisolston,
It's been a while since this topic is raised, Considering the current version of tensor flow serving is 1.8, I’m wondering if it would be possible to get all versions of the specific model served from model server?
I'd like to know what's your suggestion if we want to get all version of serving models from model server? #961
I am looking forward to your reply, thank you

Best regards,

[Harshini-Gadige]

@chrisolston - Hi, any inputs for the above ?

[lilao]

If you know the versions of the model in the model_config_file, you can send inference requests with those versions. If you get not_found error for a request, it means that the version specified in the request is not loaded.

If you don't know the versions of the model, you can try to read the model_config_file in your client to get the model versions and follow the steps in the above paragraph. Alternatively, you can add model version labels to the model_config_file, and send inference requests with those version labels (for details see https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/serving_config.md). In this way, you don't need the specific model versions and can just use version labels instead.

Would this work for you?

[Harshini-Gadige]

@shrutisharmavsco Did you get a chance to try above workaround ? Please update the status. If the issue is resolved, feel free to close this.

[Harshini-Gadige]

Closing this at it is in "awaiting response" status for more than a week. Feel free to add comments(if any), we will reopen the issue. Thanks !