Serving: [Question] How do I use tensorflow_serving deployed on kubernetes?

[atwj]

Hey guys,

So i've followed the instructions (https://tensorflow.github.io/serving/serving_inception) to deploy tensorflow_serving on Kubernetes on google container service.

But how do I go about calling the service on the cluster?

# Copyright 2016 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

#!/usr/bin/env python2.7

"""Send JPEG image to tensorflow_model_server loaded with inception model.
"""

# This is a placeholder for a Google-internal import.

from grpc.beta import implementations
import tensorflow as tf

from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2


tf.app.flags.DEFINE_string('server', 'localhost:9000',
                           'PredictionService host:port')
tf.app.flags.DEFINE_string('image', '', 'path to image in JPEG format')
FLAGS = tf.app.flags.FLAGS


def main(_):
  host, port = FLAGS.server.split(':')
  channel = implementations.insecure_channel(host, int(port))
  stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
  # Send request
  with open(FLAGS.image, 'rb') as f:
    # See prediction_service.proto for gRPC request/response details.
    data = f.read()
    request = predict_pb2.PredictRequest()
    request.model_spec.name = 'inception'
    request.inputs['images'].CopyFrom(
        tf.contrib.util.make_tensor_proto(data, shape=[1]))
    result = stub.Predict(request, 10.0)  # 10 secs timeout
    print(result)


if __name__ == '__main__':
  tf.app.run()

The above code shows how that there's a service name to call from tensorflow_serving via rpc or something. But after much searching I don't know what the service name is.

I'm intending to expose tensorflow_serving as a REST API.

Would appreciate if you could point me in to the right resources! Thanks!

[ic]

[I am on the middle of a similar task]

The code your refer to communicates with the deployed server via gRPC. The tutorial runs server code that just manages that protocol. It does not understand HTTP.

There are two approaches from here:

• Write and run in Kubernetes' container an HTTP-ready server in C++, based on the existing gRPC code and some solid HTTP/S library (or perhaps just a TCP socket at first?).
• Use a gRPC library in an API server to expose a REST interface and forward to the tutorial's solution. For example, a Django-based REST API with the gRPC Python module, or a Rails-Based REST API with the gRPC Ruby gem, etc.

There may be a canned solution to start with out there, but I have not find any for now.

[atwj]

Hey @ic!

Thanks for the comments! I'll go check it out.
Currently I've to hacked out a simple solution by wrapping a Flask-Restful api around the python tensorflow example (label_image.py from this tutorial. It's probably not as fast, but it works for now!

[ic]

Super! Perhaps you can contribute your code to the project? There may be some people interested in a sample approach.

On my side, my partner opts for the second option: Work with gRPC and use a client-side library. So work boils down to proto definition (aside the model work!) for now.

If the issue is solved, how do you think about closing it?

[atwj]

Hey @ic Yeah sure! I'll clean up my example and post it here shortly. But yeah i'll close it first! (: