Serving: Trying to use Universal Sentence Encoder Lite/2 via Tensorflow Serving

[samitshah]

System information

• Ubuntu 16.04
• Tensorflow v1.9.0
• Tensorflow Serving via Docker v1.11

Describe the problem

I created a SavedModel using the Universal Sentence Encoder Lite version. If I load the SavedModel using tf.saved_model.loader.load, it works perfectly fine.

However, if I try to serve the model using Tensorflow Serving or if I deploy the saved model to Google ML Engine, I'm getting the following error:

"error": "indices[3] = 1 is not in [0, 1)\n\t [[Node: lite_module_apply_default/Encoder_en/KonaTransformer/ClipToMaxLength/GatherV2_1 = GatherV2Taxis=DT_INT32, Tindices=DT_INT64, Tparams=DT_INT64, _output_shapes=[[?]], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]"

Exact Steps to Reproduce

1. Save Checkpoint Files

import tensorflow as tf
import tensorflow_hub as hub

tf.reset_default_graph()
with tf.Graph().as_default() as g:
    qsph_v = tf.placeholder(tf.int64,name="qsph_v")
    qsph_i = tf.placeholder(tf.int64,name="qsph_i")
    qsph_s = tf.placeholder(tf.int64,name="qsph_s")
    lite_module = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-lite/2",name="lite_module")
    q_lite_embs = lite_module(inputs=dict(values=qsph_v,indices=qsph_i,dense_shape=qsph_s))
    tensor = tf.identity(q_lite_embs,name="out") #makes it easier to find outputtensor by name
    saver = tf.train.Saver(max_to_keep=1)
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        save_path = saver.save(sess,"models/model")

1. Create saved_model

from tensorflow.saved_model import signature_constants
from tensorflow.saved_model import tag_constants
import tensorflow as tf
import os

model_name = "usel"
version = "1"
export_dir = os.path.join("saved",model_name,version)
builder = tf.saved_model.builder.SavedModelBuilder(export_dir)

sigs = {}

with tf.Session() as sess:
    new_saver = tf.train.import_meta_graph('models/model.meta')
    new_saver.restore(sess, tf.train.latest_checkpoint("models"))
    g = tf.get_default_graph()
    print("Model restored.")
    # Check the values of the variables

    qsph_v = g.get_tensor_by_name("qsph_v:0")
    qsph_i = g.get_tensor_by_name("qsph_i:0")
    qsph_s = g.get_tensor_by_name("qsph_s:0")

    out = g.get_tensor_by_name("out:0")
    sigs[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = tf.saved_model.signature_def_utils.predict_signature_def(inputs={"qsph_v": qsph_v,"qsph_i":qsph_i,"qsph_s":qsph_s}, outputs={"out": out})

    builder.add_meta_graph_and_variables(sess,
                                        [tag_constants.SERVING],
                                        signature_def_map=sigs)

builder.save()

1. Dockerfile For Tensorflow Serving

FROM tensorflow/serving
COPY saved/usel /usel
ENTRYPOINT tensorflow_model_server --rest_api_port=8501 --model_name=test --model_base_path=/usel

1. Basic Sentence Piece Values, Indices and Shapes to use for recreating error

qsph_v: [1394, 920, 2683],
qsph_i:[[0, 0], [0, 1], [0, 2]],
qsph_s:[1, 3]

[lilao]

Can you use the SavedModel CLI (https://www.tensorflow.org/guide/saved_model#cli_to_inspect_and_execute_savedmodel) to see if you can reproduce the problem with your input on the model? If so, it's a problem with the model and input (instead of model server), and you can check with the tensorflow community (https://github.com/tensorflow/tensorflow/issues). The SavedModel CLI also provides tfdbg integration, with which you can watch the intermediate Tensors and runtime graphs or subgraphs while running the SavedModel.

[samitshah]

Hi Li, thank you very much for the response.

I tried the following command with the SavedModel CLI:

saved_model_cli run --dir saved/usel/1 --tag_set serve --signature_def serving_default --input_exprs 'qsph_i=[[0, 0], [0, 1], [0, 2]];qsph_v=[1394, 920, 2683];qsph_s=[1,3]'

The command gave me the correct output without throwing an error:

Result for output key out:
[[-0.0390547  -0.05131876 -0.05443674 -0.04450405  0.03522946 -0.05135814
   0.03908788 -0.04428833  0.03865962  0.00195224  0.03077702  0.02404998
  -0.04352591 -0.0499149  -0.05417296 -0.05445938  0.05294846  0.01236235
   0.05430653 -0.00121196  0.04785711  0.05392167  0.05399362 -0.01436502
  -0.02218701 -0.01481455 -0.00473549  0.03400161  0.04498927 -0.04528304
   0.05393153 -0.00498955  0.05433089 -0.05366577  0.0536161  -0.05444458
  -0.05162744  0.02540135  0.01210878 -0.0532152   0.05002093 -0.00641442
   0.04113445 -0.054399    0.01971552  0.00744578  0.0542624   0.05444881
   0.05357424 -0.01746205  0.05435497 -0.0539602  -0.05387552 -0.0519681
  -0.00623987  0.05141359 -0.01765086  0.04210559 -0.05172855 -0.04612244
  -0.05409651  0.00837274 -0.0537173  -0.03170831  0.02029242  0.05445331
  -0.05441872  0.0267318  -0.05336414 -0.0005645   0.04136001 -0.05331825
   0.05441893 -0.02752222  0.05433295 -0.04313444  0.05242933  0.05445582
  -0.04925671  0.03437535 -0.05319436 -0.04898577 -0.05442346  0.04917231
   0.05383827  0.05189483 -0.04657027 -0.04173019 -0.05431239 -0.05386469
  -0.05445221  0.05430004 -0.04879037  0.05422398 -0.05234366  0.04549473
   0.02442385 -0.05438514 -0.05138465  0.05333255 -0.0541897   0.05423518
   0.05388683  0.05074567  0.02917385  0.05389898  0.05273515  0.0539897
  -0.05366722  0.02901492 -0.04008795 -0.05239561 -0.04316665  0.05194532
   0.00860965  0.05335913 -0.02769539  0.04685472  0.00679272 -0.04360851
   0.01780804 -0.03687434 -0.05445931 -0.05442847 -0.05039885  0.01251615
  -0.0316577  -0.04345765  0.04640885  0.05407887 -0.05412959 -0.05330467
   0.04658583  0.05441123 -0.05410266 -0.05346956  0.05432043  0.05441266
  -0.04998003  0.05319555 -0.05139231 -0.05218228  0.05053038 -0.05446017
   0.00643735  0.0136572  -0.04707678  0.0124497  -0.00584787  0.03527922
   0.00850004 -0.04847332  0.03102202 -0.05440529 -0.01745486 -0.0196976
  -0.05411869 -0.03094499  0.05119469  0.01358053 -0.04427039  0.05002345
   0.05398628  0.03839658 -0.05249179  0.04163113  0.05435916 -0.05089683
   0.04229518  0.05144932 -0.00980411 -0.04205261 -0.05349262 -0.00099692
   0.04590576  0.02848431  0.05414177 -0.03662258  0.05423737 -0.04216199
  -0.05404306  0.01412615  0.04328381  0.05437591 -0.05377967  0.04473304
  -0.03505219  0.0535838   0.0490193   0.00325101  0.01217517  0.04629558
   0.05440398  0.03231958 -0.04403592 -0.03080818  0.0533475  -0.05421685
  -0.05412471  0.05183288  0.03495123 -0.05427013  0.04786846 -0.05387892
   0.04562405 -0.02623491 -0.01342465 -0.05296112 -0.03589419 -0.05445236
  -0.04346198  0.05418207  0.0543387   0.05435125 -0.05433892  0.03700591
  -0.05004712 -0.03615998 -0.0390476   0.00054944  0.05386348  0.05376244
  -0.05324801  0.02107744  0.0438097   0.00642602  0.05119409 -0.05234864
   0.05224883 -0.05349494 -0.03342161 -0.03935241  0.05144654  0.03911753
  -0.05430007  0.05446016  0.04909403  0.05380749  0.02775652  0.05445988
  -0.05346797  0.05436308 -0.05297933 -0.03744473  0.04904737 -0.04866121
  -0.05195357 -0.01276503  0.04769094 -0.05244059 -0.00614405 -0.05436084
   0.05389296 -0.05355896 -0.04530189  0.03978476  0.04510038 -0.03033797
   0.05110478  0.01878497 -0.04621099  0.04862958 -0.0471683   0.05357021
  -0.05236137  0.05311387  0.05011735  0.05377553  0.05207914 -0.00767752
   0.05025118 -0.04189597 -0.0267998  -0.05289879  0.01835471  0.03760944
  -0.04835671  0.00381059  0.03895815  0.05414993 -0.05263049  0.03932277
   0.03544869 -0.04364847  0.01485131  0.05373975  0.05372592 -0.05428355
   0.04073901  0.05309013 -0.01906774  0.04316043  0.00216111  0.01510141
   0.05288654  0.03668046  0.04655451 -0.03007206 -0.03826308  0.05335871
  -0.04742686  0.00061342  0.05082375  0.05249269  0.04401026  0.0489563
   0.04226424 -0.04535199 -0.04574054  0.05414042  0.05427412 -0.05195138
  -0.0369627  -0.05142293  0.05366909 -0.05390502  0.03357559  0.05446029
   0.05327057  0.01077361 -0.0181316   0.05165812  0.0538209  -0.05153169
   0.03829483  0.01783956 -0.03125625 -0.05097396 -0.05377278 -0.03274406
   0.05444317 -0.02251389 -0.05419633 -0.05168956  0.04981492  0.00561636
  -0.00238015  0.0193407   0.00906236 -0.04389145 -0.05086837 -0.03948119
   0.01790449  0.01412982  0.05444275 -0.0411229   0.04917607  0.05400007
  -0.05269113  0.0018626   0.05433017  0.00390746  0.01763358 -0.05050537
  -0.05314493  0.02363025  0.0243938  -0.04167199 -0.01276046 -0.05386424
   0.05194443  0.00701935 -0.0523864  -0.03229162  0.05446023 -0.05370165
   0.05441637 -0.021365    0.05030903 -0.05438655 -0.05397403 -0.05442204
  -0.03453629  0.05033002  0.05191258  0.05443878  0.05362246 -0.05221505
   0.03397849  0.05132354 -0.00440427  0.01433226 -0.05367564  0.05439983
  -0.05388848 -0.05320518 -0.0528186  -0.05363654 -0.05315236  0.00382126
  -0.05367037  0.04347014 -0.0461202  -0.00594564  0.04431136  0.04998482
  -0.05133199 -0.05140623 -0.01942278 -0.05395672  0.00190637  0.05434731
  -0.03798412  0.04546212 -0.0534295   0.04496037  0.04654883 -0.00376845
   0.05434544 -0.03008544  0.05043663  0.05158393  0.03312892 -0.04513692
   0.05394857 -0.05309227  0.0302054   0.05100184  0.01785039  0.03216236
   0.03575879 -0.03656827  0.04361623 -0.04801746  0.05173651 -0.05331227
   0.05156473  0.04707986  0.05090237 -0.05432165  0.02421831  0.03025488
  -0.05157762 -0.04718632 -0.05092454 -0.05059215  0.04850627 -0.04495431
  -0.00432434 -0.03805459 -0.01231509  0.02651228  0.05446016  0.05331238
  -0.03511122  0.04125041  0.05438256  0.00415673  0.05203027  0.05245123
  -0.05405087  0.05407048  0.04481856 -0.05426499  0.04957487 -0.05396662
   0.0462095  -0.04503505  0.05414419  0.05425871  0.0378555   0.00333885
   0.05294531 -0.05418173  0.05430685  0.05410743  0.02065756  0.05440363
  -0.03114495 -0.05376722  0.04782495 -0.05407592  0.04027341 -0.03072825
  -0.01945762  0.02741491  0.05399194  0.04955262 -0.05442381  0.05194761
  -0.02873114 -0.02094656 -0.01981871  0.05425328  0.02411803  0.05115354
   0.03276411  0.05243343 -0.05395339 -0.05009544  0.02289287  0.04897153
   0.03509337 -0.04922479 -0.00395204 -0.00032889 -0.04984192  0.03010731
   0.04460446  0.03789061  0.00155059 -0.0094902  -0.04958741 -0.03088354
  -0.04963464  0.04973147  0.05445487 -0.00381577  0.05388623 -0.04294363
   0.0495515  -0.05243629]]

[netfs]

(this will not help solve your problem, but help us narrow down i think)

its strange that saved_model_cli works for you but TF Serving (model-server) does not. will it be possible for you to test with latest TF (1.11.0) and TFS (1.11.1)? here is my rationale:

you seem to be using TF 1.9.0 and TFS (serving) 1.11.0. the saved_model_cli that you used will also be at 1.9.0 (given that saved_model_cli is built from TF sources). so testing saved_model_cli is not identical to testing TFS (as that is 1.11.0 for you).

hence requesting to see if you can test with lastest versions of TF and TFS. and report back. that will help us narrow down where the problem is.

thanks!

[samitshah]

Hi netfs,

I upgraded my TF version to 1.11.0. I created a saved_model using that version and tested it using the saved_model_cli. The saved_model_cli still works.

I tested the same saved_model with TS 1.11.1. It still throws the same error.

[netfs]

Thanks for testing this with latest TF and TFS releases!

I will take a closer look first thing, next week and get back to you.

[samitshah]

Sounds good!

[vbardiovskyg]

Hi @samitshah,

can you please show the code that built the predict request? There could actually be an issue there.

[samitshah]

Hi @vbardiovskyg ,

I'm using the REST API endpoint to make predictions. I am using Postman to make the request, but if you need some form of code to replicate the request, I am attaching the request in Python.

import requests

url = "http://localhost:8501/v1/models/test:predict"

payload = "{\n\t\"instances\":[\n\t\t{\n\t\t\t\"qsph_v\": [1394, 920, 2683],\n            \"qsph_i\":[[0, 0], [0, 1], [0, 2]],\n            \"qsph_s\":[1, 3]\n\t\t}\t\n\t]\n}"
headers = {
    'Content-Type': "application/json"    
}

response = requests.request("POST", url, data=payload, headers=headers)

print(response.text)

[samitshah]

Hi @vbardiovskyg
You are absolutely correct. The 0th dimension of the inputs are not the same and as a result, I need to use "inputs" instead of "instances".

Thank you so much for your assistance!

[vbardiovskyg]

Yes, exactly. It's also new for me, but "instances" expects a list of instances, which it batchifies together, so after providing only one instance, this:

qsph_v: [1394, 920, 2683],
qsph_i:[[0, 0], [0, 1], [0, 2]],
qsph_s:[1, 3]

will become:

qsph_v: [[1394, 920, 2683],]
qsph_i:[[[0, 0], [0, 1], [0, 2]]],
qsph_s:[[1, 3]]

all with messed up dimensions.

[netfs]

w.r.t the REST API:

• instances is used to specify inputs in row-format (also used by CMLE)
  https://www.tensorflow.org/serving/api_rest#specifying_input_tensors_in_row_format

• inputs is used to specify in column-format:
  https://www.tensorflow.org/serving/api_rest#specifying_input_tensors_in_column_format
  this is more compact, most generic and works in cases where 0-th dimension is not same across inputs (like in your case).

happy to accept PR to improve our docs (source here) -- if this aspect is not clear from the docs.

[vbardiovskyg]

I think the documentation is good enough :)

[samitshah]

Yeah, the documentation is good enough. And I don't think the error message could be more elaborate or point to this.

I guess it's one of those things that you need to keep in mind, since the successful usage of 'instances' in the past creates the tendency to forget about 'inputs'.