Serving: Op type not registered 'ClipByValue' in binary running on 229d61c80ffd

[cchung100m]

Dear all,
I am currently trying to serve keras model VGG19 in tensorflow-serving but I am not sure whether it is successful. I see see the error message in server log and client cannot communicate with server.
I had referred previous work #310 but still get failed
Please help me to clarify the root cause.
Thanks in advance.

system information:

ubuntu 16.04
Keras 2.1.3
tensorboard 1.8.0
tensorflow 1.8.0
tensorflow-gpu 1.4.0
tensorflow-serving-api-python3 1.7.0
tensorflow-tensorboard 0.4.0

Build label: 0.5.4
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Aug 25 10:00:00 2017 (1503655200)
Build timestamp: 1503655200
Build timestamp as int: 1503655200

code to load model and export model:

from __future__ import print_function
import os
import sys
import tensorflow as tf
from keras import backend as K
from keras.models import Sequential, Model
from keras.optimizers import SGD
from keras.applications.vgg19 import VGG19

tf.app.flags.DEFINE_integer('model_version', 1, 'version number of the model.')
tf.app.flags.DEFINE_string('work_dir', '/tmp', 'Working directory.')
FLAGS = tf.app.flags.FLAGS

# very important to do this as a first thing
K.set_learning_phase(0)

def build_model():
    model = VGG19(weights='/root/downloads/vgg19_weights_tf_dim_ordering_tf_kernels.h5', include_top=True)
    config = model.get_config()
    weights = model.get_weights()
    new_model = Model.from_config(config)
    new_model.set_weights(weights)
    return new_model

def save_model_to_serving(model):

    print(model.input, model.output)

    export_path_base = sys.argv[-1]
    export_path = os.path.join(tf.compat.as_bytes(export_path_base), tf.compat.as_bytes(str(FLAGS.model_version)))
    print('Exporting trained model to', export_path)
    print('Exporting trained model version: ', (str(FLAGS.model_version)))
    builder = tf.saved_model.builder.SavedModelBuilder(export_path)

    signature = tf.saved_model.signature_def_utils.predict_signature_def(
        inputs={'images': model.input}, outputs={'scores': model.output})

    legacy_init_op = tf.group(tf.tables_initializer(), name='legacy_init_op')
    with K.get_session() as sess:
        builder.add_meta_graph_and_variables(sess=K.get_session(), tags=[tf.saved_model.tag_constants.SERVING], signature_def_map={'predict_images': signature},legacy_init_op=legacy_init_op)
    builder.save()

    print('Done exporting!')

if __name__ == '__main__':

  if FLAGS.model_version <= 0:
    print('Please specify a positive value for version number.')
    sys.exit(-1)

  model = build_model()
  sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
  model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
  model.summary()
  save_model_to_serving(model)

commands of server side :

root@229d61c80ffd:~# export TF_CPP_MIN_VLOG_LEVEL=1
root@229d61c80ffd:~# tensorflow_model_server -v=1 --enable_batching --port=9000 --model_config_file=/root/model_config_v2.proto &> tensorflow-serving.log &

model_config_v2.proto:

model_config_list: {
  config: {
    name: "vgg19",
    base_path: "/tmp/keras_model/",
    model_platform: "tensorflow",
    model_version_policy: {
        all: {}
    }
  }
}

tensorflow-serving_server_log.txt

commands of client side :

root@229d61c80ffd:~# python3 keras_VGG_client_v2.py --server=localhost:9000
Using TensorFlow backend.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/grpc/beta/_client_adaptations.py", line 193, in _blocking_unary_unary
    credentials=_credentials(protocol_options))
  File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 500, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 434, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "keras_VGG_client_v2.py", line 38, in <module>
    result = stub.Predict(request, 10.0)  # 10 secs timeout
  File "/usr/local/lib/python3.5/dist-packages/grpc/beta/_client_adaptations.py", line 309, in __call__
    self._request_serializer, self._response_deserializer)
  File "/usr/local/lib/python3.5/dist-packages/grpc/beta/_client_adaptations.py", line 195, in _blocking_unary_unary
    raise _abortion_error(rpc_error_call)
grpc.framework.interfaces.face.face.AbortionError: AbortionError(code=StatusCode.UNAVAILABLE, details="Connect Failed")

keras_VGG_client_v2.py:

from __future__ import print_function
import sys
import numpy as np
from keras.preprocessing import image
from keras.applications.resnet50 import preprocess_input

from grpc.beta import implementations
import tensorflow as tf
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2

tf.app.flags.DEFINE_string('server', '', 'PredictionService host:port')
FLAGS = tf.app.flags.FLAGS

if __name__ == '__main__':

  img_path = '/root/downloads/bird01.jpg'
  img = image.load_img(img_path, target_size=(224, 224))
  x = image.img_to_array(img)
  x = np.expand_dims(x, axis=0)
  x = preprocess_input(x)

  host, port = FLAGS.server.split(':')
  channel = implementations.insecure_channel(host, int(port))
  stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)

  request = predict_pb2.PredictRequest()
  request.model_spec.name = 'vgg19'
  request.model_spec.version.value = 1
  request.model_spec.signature_name = 'predict_images'
  request.inputs['images'].CopyFrom(tf.contrib.util.make_tensor_proto(x, shape=[1, 3, 224, 224]))

  result = stub.Predict(request, 10.0)  # 10 secs timeout
  to_decode = np.expand_dims(result.outputs['outputs'].float_val, axis=0)
  decoded = decode_predictions(to_decode, 5)
  print(decoded)

[cchung100m]

Dear all,

The new TF version 1.8 cause this issue and it can be solved by downgrade TF version.
Thanks to @tspthomas @R-Miner #310

Package Version

---

Keras 2.1.3
tensorboard 1.7.0
tensorflow 1.7.0
tensorflow-gpu 1.4.0
tensorflow-serving-api-python3 1.7.0

[cchung100m]

Dear all,
I re-open the issue because it still exists in TF version 1.8

Have I written custom code................N/A
OS Platform and Distribution.............Ubuntu 16.04
TensorFlow installed from....................apt-get
TensorFlow version..................................1.8
Bazel version..............................................0.54
CUDA/cuDNN version.............................. N/A
GPU model and memory......................... N/A
Exact command to reproduce................above

error message:
2018-06-07 15:06:36.856731: E tensorflow_serving/util/retrier.cc:38] Loading servable: {name: vgg19 version: 3} failed: Not found: Op type not registered 'ClipByValue' in binary running on 229d61c80ffd. Make sure the Op and Kernel are registered in the binary running in this process.

similar problem:

https://github.com/tensorflow/tensorflow/issues/19822

[OmriShiv]

I think you're running into the same issue we did: you compiled a graph in TF 1.8 using a new OP that isn't available in Serving 1.7. You'll have to wait until Serving is updated to 1.8 or use TF 1.7 or below.

[OmriShiv]

Which, I just checked, is now available.