Nmt: Export model to tensorflow serving

[StupidL]

I recently needs to deploy my model to tensorflow serving, but I do not know how to do that exactly even if I read the documentations of exporting a MNIST model. Here is my problem:

• How to create a signature_def_map to the SavedModelBuilder? I do not understand what the inputs and outputs should be.

Can you provide an example that export the pre-trained nmt model to tensorflow serving model?

[luozhouyang]

same issue

[jacks808]

same issue, can anyone help us?

[ptamas88]

same issue, help would be appreciated

[jacks808]

Now I Can answer this: How to create a signature_def_map to the SavedModelBuilder? I do not understand what the inputs and outputs should be.

But I still face some issue on serving model runtime.

The signature_def_map define the input and output of your serving provide.

If you want to export a model to inference from input to a output, that you should define the signature_def_map to find the inference_model input and output. this code could be found at https://github.com/tensorflow/nmt/blob/master/nmt/model_helper.py#L178. And this function should be modified to add some variable to mark what is input and output.

After this . I can export a model to inference input. but I'll keep get <unk> result.

I still work on it. Anything progress will be update here.

[changdongsheng]

@jacks808 Can you share your code?

[luozhouyang]

@changdongsheng @jacks808 @ptamas88 Hey guys, GOOD news! I recently exported the model and serve it in tf serving, and it works well! I made a pull request to this repo. You can check the pull request in pull request#344 export model to tf serving, or you can visit my fork directly: tensorflow/nmt.
And if you need a client demo, I can provide it.

[StupidL]

Thanks a lot! @luozhouyang

[ArashHosseini]

great, thx @luozhouyang

[tuyoisaza]

@luozhouyang Can you pls update this to the actual NMT branch and add to the README How to use it? Thanks!!!

[luckysofia]

@luozhouyang How can I load from exported_model and run inference?
I use tf.saved_model.loader.load(), but there's a bug:
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value dynamic_seq2seq/decoder/output_projection/kernel
[[{{node dynamic_seq2seq/decoder/output_projection/kernel/_100}} = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4_dynamic_seq2seq/decoder/output_projection/kernel", _device="/job:localhost/replica:0/task:0/device:GPU:0"](dynamic_seq2seq/decoder/output_projection/kernel)]]
[[{{node dynamic_seq2seq/decoder/output_projection/kernel/_101}} = _Recv[_start_time=0, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4_dynamic_seq2seq/decoder/output_projection/kernel", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Could you solve this problem? Thanks!!!