Serving: Run multiple models

[prateekgupta11]

I am struggling to run tensorflow serving with two models via the docker image. Below is what i have done to load two models

docker pull tensorflow/serving

docker run -d --name serving_base tensorflow/serving

docker cp models/model1 serving_base:/models/model1

docker cp models/model1 serving_base:/models/model2

docker commit serving_base new_serving

docker kill serving_base

I am able to run the model1 and model2 individually by below

docker run -p 8500:8500 -e MODEL_NAME=model1 -t new_serving
or
docker run -p 8500:8500 -e MODEL_NAME=model2 -t new_serving

what should i do to run with both models at the same time.

[gautamvasudevan]

A change was submitted to the docker serving images last night that should make this easier. You can use a model config file like

model_config_list: {
  config: {
    name: "model1",
    base_path: "/models/model1",
    model_platform: "tensorflow"
  },
  config: {
    name: "model2",
    base_path: "/models/model2",
    model_platform: "tensorflow"
  }
}

And then follow this example:

https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/docker.md#passing-additional-arguments

Note: since this just changed yesterday, you'll have to pull the tensorflow/serving:nightly image

[prateekgupta11]

Thanks Gautam, i am now able to run multiple models with the new docker image.

[DinoSourcesRex]

@prateekgupta11 Would it be possible to get an example of the dockerfile, config file and folder structure you are using?

I have outlined my problem here in this stackoverflow question. It looks to me like I did everything in the expected way?

[RajezMariner]

Thanks Gautam, i am now able to run multiple models with the new docker image.

Hi Prateek,

I would like to know the way to access the two different model end points from the client side.

I am assuming the docker consists of two different models running inside the container.

Thanks,
Rajesh