Serving: how to load custom operation library

[codeVerySlow]

I export my model to saved model
I have some custom operation implemented by C++ and build to libcustomop.so
how to load my custom operation library when I use serving

[codeVerySlow]

in training and testing I use tf.load_op_library(path) to load my custom operation
how to load op library in serving

[kirilg]

You can add your op/kernel BUILD targets to the list of SUPPORTED_TENSORFLOW_OPS and recompile the ModelServer.

[codeVerySlow]

did we have some convenience way for load library
some way like tensorflow_model_server --port=9000 --model_name=mnist --model_base_path=/tmp/mnist_model/ --op_path=/some/path/to/custom/library

[kirilg]

No, I believe the ops will need to be linked into the binary at compile time.

[bknl]

With some surgery, this is possible without recompilation:

See the following patch in the "loadlibrary" branch. Currently I only support a single user_op library, ideally this should be a list.

https://github.com/bknl/serving/commit/72ccb799d3fd6e35b2acfcde72e33737fdb723a9

[sukritiramesh]

@codeVerySlow, looks like there are a couple of options listed here. Resolving for now, but please feel free to reopen if there are follow-up questions.

[bknl]

I am updating my serving source to 1.5 (and soon 1.6) and now have trouble with the loadable custom ops.
Aparently a "variant dtype op registry" was introduced, see https://github.com/tensorflow/models/issues/2355 and somehow parts of the initialization now somehow are called twice.

I can rebuild my custom ops in the "new" linking style with separately linking against "libtensor_framework" to make them work in TensorFlow 1.5, but when I try to load them with the patch in https://github.com/bknl/serving/commit/72ccb799d3fd6e35b2acfcde72e33737fdb723a9 , I now get te same double registration:

external/org_tensorflow/tensorflow/core/framework/variant_op_registry.cc:104] Check failed: existing == nullptr (0x1ac59f28 vs. nullptr)Unary VariantDecodeFn for type_name: tensorflow::Tensor already registered

EDIT:

Ok, after some searching, the issue is that Tensorflow Serving sort of has the libtensor_framework already in the server whereas getting the custom ops loaded into Python does not., i.e.:

python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))'

yields
-L/usr/lib/python2.7/site-packages/tensorflow -ltensorflow_framework

So in order to make a custom op loadable in Python the custom op library has to be linked with -ltensorflow_framework and for being loadable with the model_server, without it. But then maybe I haven't completely understood how this is supposed to work for C++.

[tobegit3hub]

The code from @bknl is nice and it should be great to add this feature in master.

[tobegit3hub]

If you are interested, we have the similar "TensorFlow Serving" which supports TensorFlow SavedModel and external custom op. It relies on the TensorFlow Python API to load custom op whiling starting the server. Refer to simple_tensorflow_serving for more details.

It would be great if we can make the contribution back to the TensorFlow community 😃

[praveeny1986]

@kirilg, Using tensorflow serving r1.6, I tried adding my custom op in tensorflow_serving/model_servers/BUILD
here

SUPPORTED_TENSORFLOW_OPS = [
    "@org_tensorflow//tensorflow/contrib:contrib_kernels",
    "@org_tensorflow//tensorflow/contrib:contrib_ops_op_lib",
    "//tensorflow_serving/myOp:myOp.so" #Added this line
]

And myOp/BUILD:

package(
    default_visibility = [
        "//tensorflow_serving:internal",
    ],
    features = ["-layering_check"],
)

cc_library(
    name = "myOp.so",
    visibility = ["//visibility:public"],
    srcs = [
            "myOp.cc",
            "myOp.h"
           ] ,
    copts = ["-std=c++11"],
    deps = ["@org_tensorflow//tensorflow/core:framework_headers_lib",
            "@org_tensorflow//tensorflow/core/util/ctc",
            "@org_tensorflow//third_party/eigen3",
    ],
)

But I get this error while launching tensorflow_model_server
Loading servable: {name: myModel version: 1} failed: Not found: Op type not registered 'MyOp' in binary running on 43e23d7f1ad2. Make sure the Op and Kernel are registered in the binary running in this process.

Is it the correct way of adding custom Op in tensorflow serving ?

[qiaohaijun]

@bknl I am in r1.6

[xuzhezhaozhao]

@praveeny1986 I add a line ' alwayslink = 1 ' in myOp/BUILD and it worked.

`package(
default_visibility = [
"//tensorflow_serving:internal",
],
features = ["-layering_check"],
)

cc_library(
name = "myOp.so",
visibility = ["//visibility:public"],
srcs = [
"myOp.cc",
"myOp.h"
] ,
copts = ["-std=c++11"],
deps = ["@org_tensorflow//tensorflow/core:framework_headers_lib",
"@org_tensorflow//tensorflow/core/util/ctc",
"@org_tensorflow//third_party/eigen3",
],
alwayslink = 1,
)`

[Remember2018]

Hi, @xuzhezhaozhao, @praveeny1986 , I follow your method to compile my custom op into the tensorflow_model_server, the code was compiled successfully. But when I run the file, it says:

  Segmentation fault (core dumped)

When I didn't use the bazel to build the custom op (i.e., just use the g++ -I... -L ... -o myop.so), the myop.so can be loaded into the python environment and can run OK.
I don't know where the problem is.

Could you please give some advice? Thanks.

[CLIsVeryOK]

@Remember2018 did you fix your problem? I also met the Segmentation fault problem when add myOp.so to serving. myOp.so can be loaded in python tensorflow environment

[highfly22]

Try this command to remove unneeded shared library dependency

patchelf --remove-needed libtensorflow_framework.so libhash_op.so