Serving: Honor MODEL_CONFIG_FILE as an environment variable

[DinoSourcesRex]

Feature Request

What

I would like to be able to add a path to the config file using an environment variable MODEL_CONFIG_FILE, in the same way that I can pass MODEL_NAME and MODEL_BASE_PATH currently.

Why

Firstly it would be good to have a common design that can be followed, if MODEL_NAME and MODEL_BASE_PATH are accepted it stands to reason that MODEL_CONFIG_FILE should also be accepted.

My use case

To help understand why this would be helpful let me offer the following usecase:

I currently build an image with all of my models packaged within it. This is so that when it come to serving it I simply need to run docker run, or more specifically docker-compose up - this is for simplicity as I do not need to volume mount or share anything with the container as everything it needs is contained within the image.

Currently it seems as though the only example pertaining to the usage of serving multiple models is through using binding and the base image. (see here). This won't work for me, unfortunately, as it complicates our deployment pipeline.

The solution

I don't know enough about the inner workings of Tensorflow Serving but I would assume if environment variable support can be added for MODEL_NAME and MODEL_BASE_PATH then my assumption is that it should be possible for MODEL_CONFIG_FILE to be added?

Ideally that is the solution!

Alternatives

These are only specific to my use case but, as it stands, I will simply be creating multiple images (one for each model) and serving them individually with an API in front to send each request to the correct model (this is already in place as I needed that anyway, to resolve to the correct url).

[gautamvasudevan]

Is there a reason you can't just specify it on the docker run command line?

[DinoSourcesRex]

@gautamvasudevan Please correct me if I am wrong but specifying it with docker run means that I must have the config file available on the filesystem of whatever is calling the docker run, which is something I would rather avoid.

As mentioned above I build a custom image with all of the models in it, so I'm also unsure if that would interfere with anything but I expect not as it is the same entry point as the base image.

[gautamvasudevan]

Not necessarily. If you build a custom image with your models and config file in it, you can just point it the config file in the docker image.

[DinoSourcesRex]

@gautamvasudevan What would that look like then? If I run docker run -p 8501:8501 custom-image-name --model_config_file=/path/within/container then I simply get an error telling me it cannot find the file on my local file system.

2018-10-29 21:11:04.426739: F tensorflow_serving/model_servers/server.cc:97] Non-OK-status: ParseProtoTextFile(file, &proto) status: Not found: C:/Program Files/Git/config/models.config; No such file or directory
/usr/bin/tf_serving_entrypoint.sh: line 3:     6 Aborted                 tensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=${MODEL_NAME} --model_base_path=${MODEL_BASE_PATH}/${MODEL_NAME} "$@"

[gautamvasudevan]

Hmm, worked for me:

# start a base server to modify
$ docker run -d --name serving_base tensorflow/serving
# add my two savedmodels to the models dir
$ docker cp ./bob serving_base:/models/bob
$ docker cp ./bob2 serving_base:/models/bob2
# add my config to the models dir
$ docker cp models.config serving_base:/models/models.config
# commit this to a new image
$ docker commit serving_base $USER/tfserving_test
$ docker kill serving_base
# run this image and point it to the config file I have in the image
$ docker run -t $USER/tfserving_test --model_config_file=/models/model.config
2018-10-29 22:23:52.272869: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2018-10-29 22:23:52.272929: I tensorflow_serving/model_servers/server_core.cc:517]  (Re-)adding model: bob
2018-10-29 22:23:52.272943: I tensorflow_serving/model_servers/server_core.cc:517]  (Re-)adding model: bob2
...

[DinoSourcesRex]

Just ran into this, it seems. Adding //models/models.config resolved the issue.

As for passing the argument via docker-compose I simply used command: --model_config_file=/models/models.config.

I've got a reasonable workaround now so I suppose adding the environment variable doesn't affect me as much but it would be nice for consistencies sake.

Thanks for the help @gautamvasudevan

[JoseDGarcia]

If anyone still having issues with model_config_file and is using dockerfile:

FROM tensorflow/serving

RUN printf '#!/bin/bash\ntensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=${MODEL_NAME} --model_base_path=${MODEL_BASE_PATH}/${MODEL_NAME} --model_config_file=${MODEL_CONFIG_FILE} "$@"' > usr/bin/tf_serving_entrypoint.sh

ENV MODEL_CONFIG_FILE=/models/models.config

I dont know if this is the better way to solve this, but is working temporally.

[marcos-marques]

If you are using docker-compose to run tf-serving together with another application you can use this example without the need for a Dockerfile:

version: '3'

services:
  tensorflow-serving:
    image: tensorflow/serving:nightly
    container_name: tf_serving_example
    command: --model_config_file=/models/models.config
    volumes:
      - './models/models.config:/models/models.config'
      - './models/saved_model/:/models/my_model'
    ports:
      - '8500:8500' 
      - '8501:8501'
  app:
    build: app
    volumes:
      - './app:/app'
    working_dir: /app
    command: flask run
    ports:
      - '5000:5000`

In the models.config file you can configure something similar to this example:

model_config_list: {
  config: {
    name:"my_model"
    base_path:"/models/my_model"
    model_platform: "tensorflow"
    model_version_policy:{
      specific: {
          versions: 3
          versions: 2
          versions: 1
        }

      }
  },
}

[lbustelo]

+1 on this request
I don't understand why you would allow folks to build custom images that embed their models and then expect the outside world to have to pass in a configuration or know where that file exist inside the container.

Accepting ENV variables would allow fully encapsulated knowledge of the layout inside the container.

[a7744hsc]

This is not a complex feature and as others said it is counterintuitive since other configs can be set via ENV.

Is there a reason why we don't want this feature?