Serving: make model_config_file auto poll

[yupbank]

Hey, right now the model_base_path is auto polling, but not for the model_config_file itself.

It would be really helpful to add one option to let the user to switch it on.

[Harshini-Gadige]

@yupbank - Hi, is this feature added now ? Any PR related to this ?

[elvys-zhang]

ReloadConfig API will help. You could call it to reload your config file to load new model without restarting your serving.

[gitnana]

ReloadConfig API will help. You could call it to reload your config file to load new model without restarting your serving.

Hi, could you please give more details about reloading model_config_file? Is this implemented by run the cmd 'tensorflow_model_server --model_config_file=/path/to/config_file' inside docker container? Thx!

[yupbank]

we finally decide to use k8s configmap to address that. basically, we make the model_config_file into a configmap, whenever there is an update to the model_config_file, k8s would fire a new round of deployment

[elvys-zhang]

@gitnana
Yes. First you should install tensorflow_serving_api. Supposse your serving has already started by command 'tensorflow_model_server --model_config_file=/path/to/config_file', and you want to add a mnist model to it, you may call it like this

    hostport = your_host + ':' + your_port
    with open("/path/to/config_file", "r") as f:
        config_ini = f.read()
    model_server_config = model_server_config_pb2.ModelServerConfig()
    model_server_config = text_format.Parse(text=config_ini, message=model_server_config)
    new_config = model_server_config.model_config_list.config.add()
    new_config.name = "mnist"
    new_config.base_path = "/your/path/to/mnist_model"
    new_config.model_platform = "tensorflow"
    channel = grpc.insecure_channel(hostport)
    stub = model_service_pb2_grpc.ModelServiceStub(channel)
    request = model_management_pb2.ReloadConfigRequest()
    request.config.CopyFrom(model_server_config)
    response = stub.HandleReloadConfigRequest(request, 10)
    if response.status.error_code == 0:
        print("Reload done") 

After the model reloaded, you could save your new model in this way

    loaded_config = text_format.MessageToString(model_server_config)
    with open("/path/to/config_file", "w+") as f:
        f.write(loaded_config)

Hope it helps

[gitnana]

@gitnana
Yes. First you should install tensorflow_serving_api. Supposse your serving has already started by command 'tensorflow_model_server --model_config_file=/path/to/config_file', and you want to add a mnist model to it, you may call it like this

    hostport = your_host + ':' + your_port
    with open("/path/to/config_file", "r") as f:
        config_ini = f.read()
    model_server_config = model_server_config_pb2.ModelServerConfig()
    model_server_config = text_format.Parse(text=config_ini, message=model_server_config)
    new_config = model_server_config.model_config_list.config.add()
    new_config.name = "mnist"
    new_config.base_path = "/your/path/to/mnist_model"
    new_config.model_platform = "tensorflow"
    channel = grpc.insecure_channel(hostport)
    stub = model_service_pb2_grpc.ModelServiceStub(channel)
    request = model_management_pb2.ReloadConfigRequest()
    request.config.CopyFrom(model_server_config)
    response = stub.HandleReloadConfigRequest(request, 10)
    if response.status.error_code == 0:
        print("Reload done") 

After the model reloaded, you could save your new model in this way

    loaded_config = text_format.MessageToString(model_server_config)
    with open("/path/to/config_file", "w+") as f:
        f.write(loaded_config)

Hope it helps

thank you, that's very helpful!

[gitnana]

@gitnana
Yes. First you should install tensorflow_serving_api. Supposse your serving has already started by command 'tensorflow_model_server --model_config_file=/path/to/config_file', and you want to add a mnist model to it, you may call it like this

    hostport = your_host + ':' + your_port
    with open("/path/to/config_file", "r") as f:
        config_ini = f.read()
    model_server_config = model_server_config_pb2.ModelServerConfig()
    model_server_config = text_format.Parse(text=config_ini, message=model_server_config)
    new_config = model_server_config.model_config_list.config.add()
    new_config.name = "mnist"
    new_config.base_path = "/your/path/to/mnist_model"
    new_config.model_platform = "tensorflow"
    channel = grpc.insecure_channel(hostport)
    stub = model_service_pb2_grpc.ModelServiceStub(channel)
    request = model_management_pb2.ReloadConfigRequest()
    request.config.CopyFrom(model_server_config)
    response = stub.HandleReloadConfigRequest(request, 10)
    if response.status.error_code == 0:
        print("Reload done") 

After the model reloaded, you could save your new model in this way

    loaded_config = text_format.MessageToString(model_server_config)
    with open("/path/to/config_file", "w+") as f:
        f.write(loaded_config)

Hope it helps

one more question, is this the "ReloadConfig API" you mentioned above?

[elvys-zhang]

@gitnana
Yes. First you should install tensorflow_serving_api. Supposse your serving has already started by command 'tensorflow_model_server --model_config_file=/path/to/config_file', and you want to add a mnist model to it, you may call it like this

    hostport = your_host + ':' + your_port
    with open("/path/to/config_file", "r") as f:
        config_ini = f.read()
    model_server_config = model_server_config_pb2.ModelServerConfig()
    model_server_config = text_format.Parse(text=config_ini, message=model_server_config)
    new_config = model_server_config.model_config_list.config.add()
    new_config.name = "mnist"
    new_config.base_path = "/your/path/to/mnist_model"
    new_config.model_platform = "tensorflow"
    channel = grpc.insecure_channel(hostport)
    stub = model_service_pb2_grpc.ModelServiceStub(channel)
    request = model_management_pb2.ReloadConfigRequest()
    request.config.CopyFrom(model_server_config)
    response = stub.HandleReloadConfigRequest(request, 10)
    if response.status.error_code == 0:
        print("Reload done") 

After the model reloaded, you could save your new model in this way

    loaded_config = text_format.MessageToString(model_server_config)
    with open("/path/to/config_file", "w+") as f:
        f.write(loaded_config)

Hope it helps

one more question, is this the "ReloadConfig API" you mentioned above?

yes

[dsnlevi]

for example, now you have modelA running in tensorflow-serving, and you want to dynamic add modelB. you should use as the following code.

    model_server_config = model_server_config_pb2.ModelServerConfig()
    model_server_config = text_format.Parse(text=config_ini, message=model_server_config)
    modelA_config= model_server_config.model_config_list.config.add()
    modelA_config.name = "modelA"
    modelA_config.base_path = "modelA's  base_path"
    modelA_config.model_platform = "tensorflow"
    modelB_config= model_server_config.model_config_list.config.add()
    modelB_config.name = "modelB"
    modelB_config.base_path = "modelB's  base_path"
    modelB_config.model_platform = "tensorflow"
    channel = grpc.insecure_channel(hostport)
    stub = model_service_pb2_grpc.ModelServiceStub(channel)
    request = model_management_pb2.ReloadConfigRequest()
    request.config.CopyFrom(model_server_config)
    response = stub.HandleReloadConfigRequest(request, 10)

[netfs]

Happy to accept+review PR to add polling support to model config.

[stevewyl]

Based on @dsnlevi's example, if you want to add version policy information into model_config, the following code snippet is for reference:

First, we initialize some config object

from tensorflow_serving.sources.storage_path import file_system_storage_path_source_pb2
from tensorflow_serving.config import model_server_config_pb2

version_policy_config = file_system_storage_path_source_pb2.FileSystemStoragePathSourceConfig().ServableVersionPolicy()
model_server_config = model_server_config_pb2.ModelServerConfig()
modelA_config = model_server_config.model_config_list.config.add()

1. latest

version_policy_config.latest.num_versions = 2
modelA_config.model_version_policy.latest.CopyFrom(version_policy_config.latest)

1. specfic

version_policy_config.specific.versions.extend([1, 2, 3])
modelA_config.model_version_policy.specific.CopyFrom(version_policy_config.specific)
# add some version labels
modelA_config.version_labels['stable'] = 1
modelA_config.version_labels['canary'] = 2

1. all

modelA_config.model_version_policy.all.CopyFrom(version_policy_config.all)

[karlschriek]

This solution requires the client to have access to model.config file, which doesn't really seem optimal to me. For our use case we have multiple clients that request inferences from a load-balanced cluster of servers. It would be much better if the server could accept a gRPC request from any one of the clients to add a model to its model.config, without the client needing to have access to the file

[misterpeddy]

This solution requires the client to have access to model.config file, which doesn't really seem optimal to me. For our use case we have multiple clients that request inferences from a load-balanced cluster of servers. It would be much better if the server could accept a gRPC request from any one of the clients to add a model to its model.config,

@karlschriek I'm not following your use case. Are you saying when the server receives a request for model X, it looks for and adds model X? Because this is probably not what you want, since model load (which could take minutes) would now potentially be in the critical path of serving prediction requests)

without the client needing to have access to the file

[stevewyl]

Hi, guys! Have you ever meet such case, you add a brand-new model after tf serving starts running, it will cause the following error:

<_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Connect Failed"
debug_error_string = "{"created":"@1554968846.411774859","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2267,"referenced_errors":[{"created":"@1554968846.411764989","description":"Pick Cancelled","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":242,"referenced_errors":[{"created":"@1554968846.411480556","description":"Connect Failed","file":"src/core/ext/filters/client_channel/subchannel.cc","file_line":962,"grpc_status":14,"referenced_errors":[{"created":"@1554968846.411439762","description":"Failed to connect to remote host: Connection refused","errno":111,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":207,"os_error":"Connection refused","syscall":"connect","target_address":"ipv4:172.22.0.3:8502"}]}]}]}"

So, it means that tf serving can only succeed in reloading existed models, which are loaded when tf serving starts. Is there any solutions to load totally new models without stopping tf serving?

[liaocz]

how to do it using java?@dsnlevi

[misterpeddy]

Thanks for the great discussion everyone. A simple implementation for this will be added as a part of - Please follow #1301 for updates.