Keras-yolo3: Keras yolo3 model output layers objective in tensorflow serving

[spate141]

I'm trying to use yolo3 with tensorflow serving and I'm having hard time converting returned response values.

I exported Keras model for TF Serving and as I can see there are three outputs available in keras model as follow:

>> yolo_keras_model.output
[<tf.Tensor 'conv2d_59_1/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32>,
 <tf.Tensor 'conv2d_67_1/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32>,
 <tf.Tensor 'conv2d_75_1/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32>]

... I exported the model with following signature:

signature = predict_signature_def(
    inputs={'inputs': yolo_keras_model.input}, 
    outputs={
            'conv2d_59': yolo_keras_model.output[0],
            'conv2d_67': yolo_keras_model.output[1],
            'conv2d_75': yolo_keras_model.output[2]
    }

I'm getting following response in tensroflow serving client script:

response = stub.Predict(request, 60.0)
conv2d_59 = np.squeeze(tf.contrib.util.make_ndarray(response.outputs['conv2d_59']))
conv2d_67 = np.squeeze(tf.contrib.util.make_ndarray(response.outputs['conv2d_67']))
conv2d_75 = np.squeeze(tf.contrib.util.make_ndarray(response.outputs['conv2d_67']))

>> conv2d_59.shape, conv2d_67.shape, conv2d_75.shape
((15, 25, 255), (30, 50, 255), (60, 100, 255))

The confusion here is, how should I interpret each outputs? Are these values correlate with box, score, classes values?

[spate141]

Problem solved! I just followed the following yolo.py code structure and got boxes, scores and classes values!

[martinopresnik]

Hey, did you add this to model or did you processed output outside of tensorflow model?

[spate141]

I've exported keras version of yolo model to tensorflow graph and used it in tensorflow serving.

[ps48]

@spate141 I exported the model i a similar way but, I get this error when starting the model server:
Loading servable: {name: yolov3 version: 1} failed: Not found: Op type not registered 'NonMaxSuppressionV3' in binary
Did you see something similar?

[ps48]

@spate141 Thing is solved was some minor installation issue! 👍

[Jackwanguestc]

@spate141 Thing is solved was some minor installation issue! 👍

hello, I get same error. can you explain how to solve this error?

[thinhlx1993]

I've exported keras version of yolo model to tensorflow graph and used it in tensorflow serving.

I think we can directly get the boxes from Tensorflow Serving without run there code in yolo.py but I do not have any ideas to do that. Did you try to do this?

[npbit]

Stepwise to deploy it on TFS
Add this snippet below this line of yolo.py to save model in tfs servables:
`

    print("saving model********************")

    tf.saved_model.simple_save(
        self.sess,
        './test/1/',
        inputs={'input_image': self.yolo_model.input,'image_shape': self.input_image_shape},
        outputs={t.name:t for t in [self.boxes,self.scores,self.classes]})

    print('model saved *********************')

`

run TFS
and while making api call make sure
json payload should have this format:
data = json.dumps({"signature_name": "serving_default", "inputs":{"input_image":image_data.tolist(),"image_shape":image_tensor} } )

[kelvin-jose]

Problem solved! I just followed the following yolo.py code structure and got boxes, scores and classes values!

Can you please share the code?

[spate141]

@kelvin-jose I'm sorry but I can't share the code. Are you stuck at any part?

[kelvin-jose]

@kelvin-jose I'm sorry but I can't share the code. Are you stuck at any part?

Yes, I don't understand how boxes, scores and classes could be obtained out of the features. Can you please help?

[spate141]

yolo.py code mentioned above is all that you need to convert the tf-serving tensor to np array. You can find all the function and variable mentioned below in that code. Here is how you can convert the tensor values to numpy array.

class Sample:
    def __init__(self):
        # define tf placeholders for outputs
        self.input_image_shape = tf.placeholder(dtype=tf.int32, shape=(2,))
        self.out_first = tf.placeholder(dtype=tf.float32, shape=(None, None, None, 255))
        self.out_second = tf.placeholder(dtype=tf.float32, shape=(None, None, None, 255))
        self.out_third = tf.placeholder(dtype=tf.float32, shape=(None, None, None, 255))
        outs_placeholder = [self.out_first, self.out_second, self.out_third]

        # get tf placeholders for boxes, scores and classes
        self.my_boxes, self.my_scores, self.my_classes = yolo_eval(
            outs_placeholder, self.anchors, len(self.class_names),
            self.input_image_shape, score_threshold=self.score, iou_threshold=self.iou
        )

    def predict(self):
        boxes = tf.make_ndarray(response.outputs['conv2d_59'])
        scores = tf.make_ndarray(response.outputs['conv2d_67'])
        classes = tf.make_ndarray(response.outputs['conv2d_75'])

        # runs single operations and evaluates tensors in one 
        # "step" of TensorFlow computation to get boxes, scores
        # and classes values
        out_boxes, out_scores, out_classes = self.sess.run(
            [self.my_boxes, self.my_scores, self.my_classes],
            feed_dict={
                self.out_first  : boxes, 
                self.out_second : scores,
                self.out_third  : classes,
                self.input_image_shape : [image.size[1], image.size[0]]
                }
            )

[kelvin-jose]

no luck, I was trying something similar to this but I'm getting a huge reshape error.