Serving: tensorflow_text support.

[stefanondisponibile]

Are tensorflow_text ops currently fully supported by TensorFlow Serving?
I have some models perfectly running using e.g. Wordpiece tokenization, but I'm failing when trying to serve this new version of the Universal Sentence Encoder due to the following error:

2019-11-14 13:34:17.976685: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at partitioned_function_ops.cc:113 : Not found: Op type not registered 'SentencepieceOp' in binary running on 08305347fd52. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

I'm experiencing this with tensorflow/serving:nightly.

[gowthamkpr]

Please go through the similar issue and it should help you in resolving this problem. Thanks!

[stefanondisponibile]

Thank you @gowthamkpr, but I've been following that issue for months, so I'm aware of that. What I was trying to understand, though, was if there's the plan of including all tensorflow_text ops in TensorFlow Serving or the only solution will remain having to recompile it. I'm asking this because when loading the tf.hub module importing tf_sentencepiece is no longer needed, while tensorflow_text is expected. That's why I found it unexpected.

In the tf_text repo I see operations listed that are not included in the tf serving one (if I'm reading all this properly):

tensorflow_text:

cc_library(
    name = "ops_lib",
    visibility = ["//visibility:public"],
    deps = [
        ":constrained_sequence_op_cc",
        ":normalize_ops_cc",
        ":regex_split_ops_cc",
        ":sentence_breaking_ops_cc",
        **":sentencepiece_tokenizer_cc",**
        ":text_similarity_metric_ops_cc",
        ":unicode_script_tokenizer_cc",
        ":whitespace_tokenizer_cc",
        ":wordpiece_tokenizer_cc",
    ],
)

TensorflowServing (2.0)

SUPPORTED_TENSORFLOW_OPS = if_v2([]) + if_not_v2([
    "@org_tensorflow//tensorflow/contrib:contrib_kernels",
    "@org_tensorflow//tensorflow/contrib:contrib_ops_op_lib",
]) + [
    "@org_tensorflow_text//tensorflow_text:constrained_sequence_op_cc",
    "@org_tensorflow_text//tensorflow_text:sentence_breaking_ops_cc",
    "@org_tensorflow_text//tensorflow_text:unicode_script_tokenizer_cc",
    "@org_tensorflow_text//tensorflow_text:whitespace_tokenizer_cc",
    "@org_tensorflow_text//tensorflow_text:wordpiece_tokenizer_cc",
]

[nrobeR]

Hi Stefano, which tf_serving are you using? It should be included in tf 1.15 or 2.0 release version.

[guillaumekln]

The list of supported TensorFlow Text ops should simply be updated to include the latest ones (eg. SentencepieceOp).

cc @broken who introduced initial support of Text ops in https://github.com/tensorflow/serving/commit/142d0adb5e2975689d80d8fc608c9684e96de078.

[stefanondisponibile]

@nrobeR I'm using today's tensorflow/serving:latest, i.e.:

"Id": "sha256:048f8669e211797288bb308bbca2969d5ae9b5177afa08baab532c7c6da3fcf2",
        "RepoTags": [
            "tensorflow/serving:latest"
        ],
        "RepoDigests": [
            "tensorflow/serving@sha256:091c1d0440815e250114a6d0232ad3cb1d320c64b1ebb75ed8a80184fc25482d"
        ]

This is what I get:

$ docker pull tensorflow/serving

Using default tag: latest
latest: Pulling from tensorflow/serving
22e816666fd6: Pull complete 
079b6d2a1e53: Pull complete 
11048ebae908: Pull complete 
c58094023a2e: Pull complete 
4dee0153a839: Pull complete 
90850b98765d: Pull complete 
de0a35913cb5: Pull complete 
91e09abfcd7f: Pull complete 
Digest: sha256:091c1d0440815e250114a6d0232ad3cb1d320c64b1ebb75ed8a80184fc25482d
Status: Downloaded newer image for tensorflow/serving:latest
docker.io/tensorflow/serving:latest

$ export MODEL_DIR=/path/to/my/model/use
$ docker run -t --rm -p 8501:8501 -v "$MODEL_DIR:/models/use" -e MODEL_NAME=use tensorflow/serving

2019-11-19 09:00:57.714772: I tensorflow_serving/model_servers/server.cc:85] Building single TensorFlow model file config:  model_name: use model_base_path: /models/use
2019-11-19 09:00:57.717961: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2019-11-19 09:00:57.718045: I tensorflow_serving/model_servers/server_core.cc:573]  (Re-)adding model: use
2019-11-19 09:00:57.829045: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: use version: 1}
2019-11-19 09:00:57.829156: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: use version: 1}
2019-11-19 09:00:57.829348: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: use version: 1}
2019-11-19 09:00:57.829472: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/use/1
2019-11-19 09:00:57.928433: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-11-19 09:00:58.010954: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-19 09:00:58.219108: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2019-11-19 09:01:01.635260: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:151] Running initialization op on SavedModel bundle at path: /models/use/1
2019-11-19 09:01:02.225268: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 4395799 microseconds.
2019-11-19 09:01:02.242062: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:105] No warmup data file found at /models/use/1/assets.extra/tf_serving_warmup_requests
2019-11-19 09:01:02.257728: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: use version: 1}
2019-11-19 09:01:02.263806: I tensorflow_serving/model_servers/server.cc:353] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
2019-11-19 09:01:02.267474: I tensorflow_serving/model_servers/server.cc:373] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
2019-11-19 09:02:00.488127: E external/org_tensorflow/tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] function_optimizer failed: Not found: Op type not registered 'SentencepieceOp' in binary running on 15ade9bf7009. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

Same output also with tensorflow/serving:nightly.

As @guillaumekln said, it looks like as simple update of the the supported TensorFlow Text ops will solve the issue (but I still haven't tested this).

[stefanondisponibile]

Are there any updates on this?

[broken]

I'm also reaching out to see if there is a way to have TF Text ops automatically included, rather than on a per op basis. The discussion currently revolves around testing - having TF Text ops tested inside a serving environment without having to rewrite new tests for all the ops.

[stefanondisponibile]

Thanks @broken, that sounds good, of course. I wonder if you knew if we're going to see the current TF Text Ops (i.e. including sentencepiece, etc) built into TensorFlow Serving any time soon with the next release? I am asking because I'm sure, like me, also for others this represent a main point for choosing a serving architecture or another at the moment.

[broken]

I'm working with TF Serving to add the rest of the TF Text ops currently. The original ones were added with little vision to the maintenance of adding more in the future. This time I'm wanting to move forward with an approach that makes regular updates smooth. I expect everything will be available in the release after we get these added, but I am not in position to speak to the model server release schedule.

[stefanondisponibile]

That's great! Thanks for the awesome work, keep it up! ⚡

[broken]

FYI; we compromised on an integration testing solution, but I'm going to be on vacation for a week, so I won't be able to make progress with it until then.

[stefanondisponibile]

Thank you @broken, I'll leave the issue open as a reference until it gets solved 👍 Have fun and relax in the meantime 🦸

[mfojtak]

Is this getting anywhere? The current version of tf.text deployed on serving is old. Just wanted to know if I can count on this to be implemented soon or switch to an alternative solution. Thank you.

[broken]

Yeah; I sent code for review over to the serving team before I went on vacation, but I believe most of them were already out.. Now that everybody is getting back from holiday, I'll push for the code to get in.

[hitchhicker]

We ran into a problem when trying to serve a model (universal-sentence-encoder-multilingual) that uses sentencepiece because of the missing operator SentencepieceOp. After several attempts, we have succeeded in compiling tensorflow_text 2.1.0 with the TF-serving 2.1 release and built it as a Docker image. It can be used as a temporary solution until the official release.
Here are the steps we did to make it work:

1. Create a Dockerfile as follows, it's based on Dockerfile.devel from the TF-serving project.

FROM ubuntu:18.04 as base_build

ARG TF_SERVING_VERSION_GIT_BRANCH=r2.1
ARG TF_SERVING_VERSION_GIT_COMMIT=head

RUN apt-get update && apt-get install -y --no-install-recommends \
        automake \
        build-essential \
        ca-certificates \
        curl \
        git \
        libcurl3-dev \
        libfreetype6-dev \
        libpng-dev \
        libtool \
        libzmq3-dev \
        mlocate \
        openjdk-8-jdk\
        openjdk-8-jre-headless \
        pkg-config \
        python-dev \
        software-properties-common \
        swig \
        unzip \
        wget \
        zip \
        zlib1g-dev \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN curl -fSsL -O https://bootstrap.pypa.io/get-pip.py && \
    python get-pip.py && \
    rm get-pip.py

RUN pip --no-cache-dir install \
    future>=0.17.1 \
    grpcio \
    h5py \
    keras_applications>=1.0.8 \
    keras_preprocessing>=1.1.0 \
    mock \
    numpy \
    requests

# Set up Bazel
ENV BAZEL_VERSION 0.24.1
WORKDIR /
RUN mkdir /bazel && \
    cd /bazel && \
    curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \
    curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE && \
    chmod +x bazel-*.sh && \
    ./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \
    cd / && \
    rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh

# Download TF Serving sources (optionally at specific commit).
WORKDIR /tensorflow-serving
RUN git clone --branch=${TF_SERVING_VERSION_GIT_BRANCH} https://github.com/tensorflow/serving . && \
    git remote add upstream https://github.com/tensorflow/serving.git && \
    if [ "${TF_SERVING_VERSION_GIT_COMMIT}" != "head" ]; then git checkout ${TF_SERVING_VERSION_GIT_COMMIT} ; fi

COPY workspace.bzl ./tensorflow_serving/workspace.bzl

COPY tftext.patch1 ./third_party/tf_text/tftext.patch1
COPY tftext.patch2 ./third_party/tf_text/tftext.patch2

RUN sed -i.bak '/@org_tensorflow_text\/\/tensorflow_text:wordpiece_tokenizer_cc/a\    "@org_tensorflow_text\/\/tensorflow_text:sentencepiece_tokenizer_cc",' \
       tensorflow_serving/model_servers/BUILD

FROM base_build as binary_build
# Build, and install TensorFlow Serving
ARG TF_SERVING_BUILD_OPTIONS="--config=nativeopt"
RUN echo "Building with build options: ${TF_SERVING_BUILD_OPTIONS}"
ARG TF_SERVING_BAZEL_OPTIONS=""
RUN echo "Building with Bazel options: ${TF_SERVING_BAZEL_OPTIONS}"

RUN bazel build --color=yes --curses=yes \
    ${TF_SERVING_BAZEL_OPTIONS} \
    --verbose_failures \
    --output_filter=DONT_MATCH_ANYTHING \
    ${TF_SERVING_BUILD_OPTIONS} \
    tensorflow_serving/model_servers:tensorflow_model_server && \
    cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server \
    /usr/local/bin/

CMD ["/bin/bash"]

Notice that we have edited tensorflow_serving/model_servers/BUILD to support sentencepiece tokenizer as it suggests in the TensorFlow's official document.

1. Create a workspace.bzl file to override the one under tensorflow_serving/, it defines all the external dependencies.

# TensorFlow Serving external dependencies that can be loaded in WORKSPACE
# files.

load("@org_tensorflow//third_party:repo.bzl", "tf_http_archive")
load("@org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

def tf_serving_workspace():
    """All TensorFlow Serving external dependencies."""

    tf_workspace(path_prefix = "", tf_repo_name = "org_tensorflow")

    # ===== gRPC dependencies =====
    native.bind(
        name = "libssl",
        actual = "@boringssl//:ssl",
    )

    # gRPC wants the existence of a cares dependence but its contents are not
    # actually important since we have set GRPC_ARES=0 in tools/bazel.rc
    native.bind(
        name = "cares",
        actual = "@grpc//third_party/nanopb:nanopb",
    )

    # ===== RapidJSON (rapidjson.org) dependencies =====
    http_archive(
        name = "com_github_tencent_rapidjson",
        urls = [
            "https://github.com/Tencent/rapidjson/archive/v1.1.0.zip",
        ],
        sha256 = "8e00c38829d6785a2dfb951bb87c6974fa07dfe488aa5b25deec4b8bc0f6a3ab",
        strip_prefix = "rapidjson-1.1.0",
        build_file = "@//third_party/rapidjson:BUILD",
    )

    # ===== libevent (libevent.org) dependencies =====
    http_archive(
        name = "com_github_libevent_libevent",
        urls = [
            "https://github.com/libevent/libevent/archive/release-2.1.8-stable.zip",
        ],
        sha256 = "70158101eab7ed44fd9cc34e7f247b3cae91a8e4490745d9d6eb7edc184e4d96",
        strip_prefix = "libevent-release-2.1.8-stable",
        build_file = "@//third_party/libevent:BUILD",
    )

    # ===== Override TF defined `com_google_absl` (we need a recent version).
    tf_http_archive(
        name = "com_google_absl",
        build_file = str(Label("@org_tensorflow//third_party:com_google_absl.BUILD")),
        sha256 = "b6aa25c8283cca9de282bb7f5880b04492af76213b2f48c135c4963c6333a21e",
        strip_prefix = "abseil-cpp-36d37ab992038f52276ca66b9da80c1cf0f57dc2",
        urls = [
            "http://mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/36d37ab992038f52276ca66b9da80c1cf0f57dc2.tar.gz",
            "https://github.com/abseil/abseil-cpp/archive/36d37ab992038f52276ca66b9da80c1cf0f57dc2.tar.gz",
        ],
    )

    # ===== Use tensorflow text 2.1.0
    http_archive(
        name = "org_tensorflow_text",
        sha256 = "6bfe4248fd1b9ede10e2d013a5ef4ff2fa1064687d1ed506b3c8cbe0a364b93b",
        strip_prefix = "text-2.1.0-rc0",
        urls = [
            "https://github.com/tensorflow/text/archive/v2.1.0-rc0.zip",
        ],
        patches = ["@//third_party/tf_text:tftext.patch1", "@//third_party/tf_text:tftext.patch2"],
        patch_args = ["-p1"]
    )

    # ===== sentencepiece dependencies =====
    http_archive(
        name = "com_google_sentencepiece",
        strip_prefix = "sentencepiece-1.0.0",
        sha256 = "c05901f30a1d0ed64cbcf40eba08e48894e1b0e985777217b7c9036cac631346",
        urls = [
            "https://github.com/google/sentencepiece/archive/1.0.0.zip"
        ],
    )

    # ===== glog dependencies =====
    http_archive(
        name = "com_google_glog",
        sha256 = "1ee310e5d0a19b9d584a855000434bb724aa744745d5b8ab1855c85bff8a8e21",
        strip_prefix = "glog-028d37889a1e80e8a07da1b8945ac706259e5fd8",
        urls = [
            "https://mirror.bazel.build/github.com/google/glog/archive/028d37889a1e80e8a07da1b8945ac706259e5fd8.tar.gz",
            "https://github.com/google/glog/archive/028d37889a1e80e8a07da1b8945ac706259e5fd8.tar.gz",
        ],
    )

Notice that we have used tensorflow text 2.1.0 instead of the old version and added two more dependencies (sentencepiece and glog).

1. Create two patch files: tftext.patch1 and tftext.patch2, they are used to replace @local_config_tf by @org_tensorflow in the tensorflow text 2.1.0.

tftext.patch1:

--- a/tensorflow_text/tftext.bzl    2020-01-03 10:40:52.570505318 +0000
+++ b/tensorflow_text/tftext.bzl    2020-01-03 10:48:05.610448912 +0000
@@ -44,8 +44,9 @@
         copts = [ "-pthread", ],
         alwayslink = 1,
         deps = cc_op_kernels + [
-            "@local_config_tf//:libtensorflow_framework",
-            "@local_config_tf//:tf_header_lib",
+       "@org_tensorflow//tensorflow/core:framework",
+            "@org_tensorflow//tensorflow/core:lib",
+            "@org_tensorflow//tensorflow/core:protos_all_cc",
         ],
     )

tftext.patch2:

--- a/tensorflow_text/core/kernels/BUILD    2020-01-03 10:40:52.562505384 +0000
+++ b/tensorflow_text/core/kernels/BUILD    2020-01-03 10:46:51.927143543 +0000
@@ -16,8 +16,9 @@
     "@com_google_absl//absl/strings",
     "@com_google_absl//absl/types:optional",
     "@com_google_absl//absl/types:span",
-    "@local_config_tf//:libtensorflow_framework",
-    "@local_config_tf//:tf_header_lib",
+    "@org_tensorflow//tensorflow/core:framework",
+    "@org_tensorflow//tensorflow/core:lib",
+    "@org_tensorflow//tensorflow/core:protos_all_cc",
 ]

 tf_kernel_library(

1. Run the following command to build the image

docker build -t tensorflow-serving-with-latest-tensorflow-text .

[m0baxter]

@hitchhicker is there a reason you dropped the stuff in Dockerfile.devel that appears below the comment # Build and install TensorFlow Serving API?

[hitchhicker]

@m0baxter, the reason is that we used only tensorflow_model_server in our case. If you need using pip package, you could keep the staff below the comment # Build and install TensorFlow Serving API.

[broken]

This took much longer to get approval and get in than I originally thought, but I just got in bd206e4, and now keeping tf.text updated should be much easier.

[ashmeet13]

@broken Thanks!
Testing tensorflow/serving:nightly to serve Multilingual-USE seems to work fine.
A small follow up question - Till when can this be expected to be built into the latest tag release?

[broken]

That's great to hear it works for you! I can't speak to serving's release timeline, but it should be in whatever is next. I'll hand this back to @nrobeR who may know.

[alpsholic]

can anyone share the .so file generated in the above compilation process? I converted multi-lang/1 in python from hub module to normal tf model to load in Java. But I get _TensorFlowException: Op type not registered 'SentencepieceEncodeSparse_ in binary... exception. I want to load the .so file onto tensorflow and load my converted model. Thanks a ton in advance.

[broken]

@alpsholic It looks like that model is using a TF op that was specifically released in the SP repo that was available before we got it integrated in tf.Text. You can either update the model to use the tf.Text SentencepieceTokenizer, or link in the version that model is using (https://github.com/google/sentencepiece / pip install tf_sentencepiece).

[ElliotVilhelm]

@broken

I ran into the op not found situation while using a model with multilingual embeddings: https://tfhub.dev/google/universal-sentence-encoder-multilingual/3

The model was built with:

tensorboard                        2.1.0
tensorflow                         2.1.0
tensorflow-estimator               2.1.0
tensorflow-hub                     0.7.0
tensorflow-text                    2.1.1

This occurred on our model server running tensorflow/serving:latest built today 02/14/2020.
Error output was as follows:

{ "error": "{{function_node __inference_signature_wrapper_213566}} {{function_node __inference_signature_wrapper_213566}} {{function_node __inference__wrapped_model_207781}} {{function_node __inference__wrapped_model_207781}} {{function_node __inference_restored_function_body_14431}} {{function_node __inference_restored_function_body_14431}} [_Derived_]{{function_node __inference___call___6416}} {{function_node __inference___call___6416}} Op type not registered \'SentencepieceOp\' in binary running on fraud-detection-model-serving-app-server-66c9df7788-g4vk7. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[StatefulPartitionedCall]]\n\t [[model/keras_layer/StatefulPartitionedCall]]\n\t [[StatefulPartitionedCall]]\n\t [[StatefulPartitionedCall]]" }

I tested tensorflow/serving:nightly and it resolved the issue. Looking forward to this change being released. Thanks!

[broken]

Me too. :) I believe tensorflow/serving times their releases around tensorflow releases. So I expect the next release will contain it.

[alpsholic]

@broken How can I update the model to use the tf.Text SentencepieceTokenizer? Even if I do, I may need TF 2.0 for serving? Since I dont have tf 2.0 jar for Java, I am afraid I cant pursue this approach. What works for me is if I can get _sentencepiece_processor_ops.so file, I can load onto tf in Java and extract the embeddings. Please throw some light on how can I get this done. My another question is- what was the last tf jar which had this setencepiece op embedded which does not need me loading this .so file?

[broken]

How can I update the model to use the tf.Text SentencepieceTokenizer?

This would require updating the source and including it in your model directly, rather than using hub.

Even if I do, I may need TF 2.0 for serving?

Do you say this because tf.Text 2.x is included as a dependency in model server? Actually, the code for tf Text's 1.x & 2.x branches are currently the same. If it's recompiling (like what is happening in serving), it shouldn't matter what version of TF your compiling with it.

My another question is- what was the last tf jar which had this setencepiece op embedded which does not need me loading this .so file?

Are you talking about TF or TF Serving? TF never had this included. For that matter, TF Serving never had this op included either, but the next version should have the tf.Text SP included.

What works for me is if I can get _sentencepiece_processor_ops.so file, I can load onto tf in Java and extract the embeddings. Please throw some light on how can I get this done.

Did you try downloading it from https://github.com/google/sentencepiece or using pip install tf_sentencepiece? That is where the missing op comes from.

Honestly, I don't know how or if you can get .so files working with model server.
The model server builds everything together, so I actually think the easiest thing would be to clone the repo, update the WORKSPACE file to include the correct version of SP that includes that TF code, update the BUILD file to include that SP op, and then build model server with everything included. Your changes will be similar to: https://github.com/tensorflow/serving/commit/142d0adb5e2975689d80d8fc608c9684e96de078

[alpsholic]

Thanks a lot for your reply. I will go through your reply throughly..but a quick reply on your following comment:
I dont have issues using the model in python. I install tf_sentencepiece as you said, it works without any issue. I read that guse hub model in python and save it as normal tf model. Then I load it in Java. That where i get this missing op exception.

"Did you try downloading it from https://github.com/google/sentencepiece or using pip install tf_sentencepiece? That is where the missing op comes from."

[broken]

I see. Unfortunately, I haven't loaded a model in Java, so any help I can provide there is limited. However, when you installed tf_sentencepiece, it should have installed the .so file in the process.

To find it, bring up a python shell and find where it is installed:

import tf_sentencepiece as sp
sp.__file__

Then you should be able to find it in that directory:
find <path_from_above> -name '*.so'

[alpsholic]

@broken I cant express enough how helpful your last suggest was. I was blocked all along and this just saved my day. Thank you so much. I use 1.14.0 so and load it in Java before loading the converted guse model. So all good. Btw, I am assuming this .so file is OS specific. If I deploy my code on centos, i would need different .SO. Right now, I am on Mac. Why I am saying this is- I loaded one so from their git repo, then java couldn't load so saying some bytes were not the right ones.

[broken]

@alpsholic Great! I'm happy that worked for you. And, you are correct. The .so file is compiled specifically for each OS platform (macos, win, linux). Pip install knows the platform your on, so it pulls the correct package from pypi and hides this from you.

[17patelumang]

@alpsholic can you please share you Java code for inference ?

[17patelumang]

@stefanondisponibile can you please share the code where you were successfully able to run I Wordpiece tokenization in different model for TF serving ?

[stefanondisponibile]

hey @17patelumang I've never tried running Wordpiece tokenization on an exported graph myself, but I can ensure models like the UniversalSentenceEncoder are currently running gracefully on tensorflow/serving:nightly.

[17patelumang]

@alpsholic any updates ?
@stefanondisponibile thank you for reply , also what you mean by "running Wordpiece tokenization on an exported graph" , does it mean that you never added Wordpiece tokenization op into metagraph ?

[alpsholic]

@17patelumang sorry for the delay. My code is scattered across. I will past the imp snippets here. I think you will get rough idea. I am sorry for jumbled code.

TensorFlow.loadLibrary("_sentencepiece_processor_ops.so.1.14.0");`
SavedModelBundle bundle = SavedModelBundle.load(model path, "serve").get();

//convert list of sentences into a single tensor

   byte[][] byteTensor = new byte[inputStringList.size()][maxLen];

       for (int i=0; i<inputStringList.size(); i++) {
           byteTensor[i] = inputStringList.get(i).getBytes(StandardCharsets.UTF_8);
       }
        Tensor.create(byteTensor);

       // get prediction from the model
        session
                .runner()
                .feed("input", inputTensor)
                .fetch("output")
                .run()
                .stream()
                .findFirst();
      //you can get session the following way
       session =  new SessionResource(true, bundle.session())

[17patelumang]

@alpsholic thank you

[anfernee-cyanoly]

@hitchhicker we're using SentencepieceEncodeSparse operator, your instruction is for operator SentencepieceOp, could you please share what need to done for SentencepieceEncodeSparse operator.

Thanks for your help!

[broken]

@anfernee-cyanoly SentencepieceEncodeSparse is not part of the tf text library, and not natively built in tf model server. You have two options:

1. Convert your code to use the SentencepieceTokenizer (SentencepieceOp) - api docs
2. Find tf serving documentation on building model server with custom ops, and supplying it with the code for SentencepieceEncodeSparse.

I'm going to close this issue since the original problem of keeping tensorflow_text updated within tf serving has been resolved.