Serving: Unable to apply version_labels to models loaded through S3

[orf]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow Serving installed from (source or binary): Source
• TensorFlow Serving version: 1.11.x (r.11 branch)

Describe the problem

Following the instructions in the documentation does not appear to work. It fails with:

tensorflow_serving | 2019-01-07 18:52:15.947347: I tensorflow_serving/core/basic_manager.cc:280] Unload all remaining servables in the manager.
tensorflow_serving | Failed to start server. Error: Failed precondition: Request to assign label to version 1 of model x, which is not currently available for inference

Exact Steps to Reproduce

Creating a simple config file with this contents:

model_config_list: {
  config: {
    name: "model",
    base_path: "s3://bucket/models/model",
    model_platform: "tensorflow",
    model_version_policy {
      specific {
         versions: 1
      }
    }
    version_labels {
      key: 'stable'
      value: 1
    }
  }
}

(A S3 bucket may not be needed, someone else is describing the same issue on stack overflow using local images)

And try to start the service. It will fail with the logs below.

If you remove the version_labels section the model loads correctly.

Source code / logs

tensorflow_serving | 2019-01-07 18:52:15.947347: I tensorflow_serving/core/basic_manager.cc:280] Unload all remaining servables in the manager.
tensorflow_serving | Failed to start server. Error: Failed precondition: Request to assign label to version x of model x, which is not currently available for inference

[cshjarry]

I have the similar problem...And I found this question in stackoverflow described my problem. But I did not found any useful solutions to this.

[christisg]

A label can be assigned only to a model version that is already loaded and available for serving (i.e. in kAvailable status).

[orf]

Sure, so doesn't that make it almost completely useless in the model config file that's loaded from disk at startup?

And in any case this should be documented.

[christisg]

The presumed use-case for this feature is canary of new model versions, and one of the main benefits of assigning a tentative label to a model version is the ability to dynamically change this association without any changes to the querying clients.
So in order to benefit from this feature one would need to have a mechanism to reload the model config on the fly (with something like HandleReloadConfigRequest mentioned in the serving config documentation).
The use-case of querying a Model Server running with a static model config loaded from disk only once at startup could be easily satisfied without the use of labels.
We can improve the documentation to make the prerequisites for assigning labels more clear. Thanks for your feedback.

[orf]

I appreciate that this is the presumed use case, however this live reloading of the configuration on the fly is perhaps not a deployment model that is common or easy to achieve for many places, particularly if you're using something like kubernetes.

You can also achieve the same effect using health checks to ensure that the model is loaded correctly and is ready to serve before any requests are routed to the service. This is far easier to implement and fits in with common deployment patterns than the way you are presuming it should be done. In fact I would say it's nearly impossible to use labels with S3, which are an awesome feature, without a pretty bespoke deployment process.

I guess the real question is if there is any hard technical limitations that prevent version labels being applied to models that are not ready yet, or are loaded from S3.

[shlomiken]

i think that we are all going in circles here , looking for the same features - but still getting the same answers. (i for example opened this one #1300 which was closed almost immediately ).
what we need is very very simple .

1. have a static model list configured in models.conf - where we can easily set our labels. (yes for models that are in the local file system and ready to be loaded)
2. dynamic model discovery - ability to discover new models (not versions ) in realtime - may it be local file system or S3 (under the deceptive model_base_path parameter - why do you even need it if you specify everything in config file,its very confusing to new comers)

unfortunately i cannot write c++ code - if i could i would suggest a PR yesterday
hopefully - this will be addressed , as the number of issues i see on dynamic loading of models is getting bigger.
just for you to understand the workaround and its problems here.
i needed to create script that
a. sync S3 buckets (cause the S3 integration is poor and query AWS every second)
b. write a python script to update model list
c. i'm stuck with my static models and applying version labels to them
d. thinking on merging protobuf files (static definition + dynamic definition) and then update the model - something i really don't want to do.

[shlomiken]

More than that - presumed use case or not , where did you see a server that its config file is not valid upon start.
cause the models first needs to be loaded and only then can be configured (labels,specific version etc)
you basically say - we don't have a config file , just an interface to be configured at runtime.

[christisg]

Changing this to feature request. We'll evaluate extending the version labeling feature to allow assigning labels statically at load time.
As per current design this assignment is possible only at runtime.

[misterpeddy]

This is fixed in this commit.

[christisg]

To summarize, commit adds a flag which allows assigning labels to versions that are not yet available, but only in cases when the label assigned is new (i.e. it was not yet in use). The latter is a safeguard against user mistake that could lead to loss of queryable label during a version swap.

Also, another commit relevant to the discussions here.