Serving: How to call Tensorflow Serving from Golang?

[mauri870]

Hi, I'm trying to implement a tensorflow serving client specifically for the inception example.

What I have so far is:

imageBytes, err := ioutil.ReadFile('image.jpg')
ImageTensor, err := tf.NewTensor(string(imageBytes))
if err != nil {
    ...
}

This tensor has a DType 0x7(DataType_DT_STRING)
Then I use tensorString, ok := imageTensor.Value().(string) to get the tensor value as string since Value() return an interface.

And the grpc request trying to mimic the python and c++ examples:

request := &pb.PredictRequest{
    ModelSpec: &pb.ModelSpec{
        Name:          "inception",
        SignatureName: "predict_images",
        Version: &google_protobuf.Int64Value{
            Value: 1,
        },
    },
    Inputs: map[string]*tfframework.TensorProto{
        "images": &tfframework.TensorProto{
            Dtype: tfframework.DataType_DT_STRING,
            TensorShape: &tfframework.TensorShapeProto{
                Dim: []*tfframework.TensorShapeProto_Dim{
                    &tfframework.TensorShapeProto_Dim{
                        Size: int64(1),
                    },
                },
            },
            TensorContent: []byte(tensorString),
        },
    },
}

resp, err := h.GrpcClient.Predict(context.Background(), request)

When I run the code I get the following error back:

rpc error: code = InvalidArgument desc = tensor parsing error: images

I can't see any related logs in the tensorflow serving container.

Edit: My problem seems to be related with this piece of code. Any workround to get a valid proto tensor from Go?

[mauri870]

Got it 😓
I changed TensorContent: []byte(tensorString), to StringVal: [][]byte{[]byte(tensorString)}, and its working like a charm

Can I add a golang client to the examples folder?

[grafael]

For instance, @mauri870 could you share the client code in a gist ?

[mauri870]

@grafael Yes, sure. See this gist

You need to follow the tensorflow instructions for go and compile the proto files as described in the gist

[grafael]

Thanks @mauri870 !!

[jaxsongqd]

I have Tensor,DataType:0x1,shape:[]int64{28, 28, 1} , and how can i call grpc witch request, Inputs paraemeter how to write

request := &pb.PredictRequest{
    ModelSpec: &pb.ModelSpec{
      Name:          "mnist",
      SignatureName: "predict_images",
      Version: &google_protobuf.Int64Value{
        Value: int64(1),
      },  
    },  
    Inputs: map[string]*tf_core_framework.TensorProto{
      "images": &tf_core_framework.TensorProto{
        Dtype: tf_core_framework.DataType_DT_FLOAT,
        TensorShape: &tf_core_framework.TensorShapeProto{
          Dim: []*tf_core_framework.TensorShapeProto_Dim{
            &tf_core_framework.TensorShapeProto_Dim{
              Size:28 ,
            },  

            &tf_core_framework.TensorShapeProto_Dim{
              Size: 28,
            },  

            &tf_core_framework.TensorShapeProto_Dim{
              Size: 1,
            },  
          },  
        },  
        FloatVal: tensor.Value(),
      },  
    },  
}

tensor.Value() type is [][][]float32, how can i transfrom the [][][]float32 tensor value to []float32?

[jaxsongqd]

@mauri870

[orian]

Is it possible / how to generate protos using bazel?

I've published a repo with some dummy example:
https://github.com/datainq/go-mnist-client

I've just noticed that some packages are not named correctly. It seems like an overkill to fix it in my repo. Probably would a be a good idea to add support in Go TensorFlow

[mauri870]

Updated version here

:smile:

[frpunzalan]

@mauri870
Thanks for the tutorial.
I tried building the go inception-client in your tutorial but I ran into this error:
vendor/tensorflow_serving/apis/prediction_log.pb.go:9:8: cannot find package "tensorflow_serving/core" in any of:
/home/../vendor/tensorflow_serving/core (vendor tree)
/usr/local/go/src/tensorflow_serving/core (from $GOROOT)

Any idea what is causing this error?

[mauri870]

@frpunzalan What commands are you running to compile the proto files? Honestly I didn't test against the 1.8 version.

[frpunzalan]

@mauri870 I tried protocol buffers v3.5 and 3.6 and using the latest golang version (1.10). Can you tell me which versions are you using? thanks

[mauri870]

I was able to compile the protos using tensorflow and serving branch r1.7 and golang 1.10, protoc 3.5.1

[mauri870]

Unfortunately I can't build with latest r1.8 tf / serving branch.

But the following is working for the r1.7 branch:

PROTOC_OPTS='-I lib/tensorflow -I lib/serving --go_out=plugins=grpc:vendor'

eval "protoc $PROTOC_OPTS lib/serving/tensorflow_serving/apis/*.proto"
eval "protoc $PROTOC_OPTS lib/serving/tensorflow_serving/config/*.proto"
eval "protoc $PROTOC_OPTS lib/serving/tensorflow_serving/util/*.proto"
eval "protoc $PROTOC_OPTS lib/serving/tensorflow_serving/sources/storage_path/*.proto"
eval "protoc $PROTOC_OPTS lib/tensorflow/tensorflow/core/framework/*.proto"
eval "protoc $PROTOC_OPTS lib/tensorflow/tensorflow/core/example/*.proto"
eval "protoc $PROTOC_OPTS lib/tensorflow/tensorflow/core/lib/core/*.proto"
eval "protoc $PROTOC_OPTS lib/tensorflow/tensorflow/core/protobuf/{saver,meta_graph}.proto"

[frpunzalan]

great! got it working with tf /serving branch 1.7. Thanks
Hopefully they fix that in later versions soon.

[wt-huang]

This is likely a sync issue.

[azer]

Serving works, but I can't compilete tensorflow/core/framework. It fails with following error message;

2019/07/06 00:45:24 protoc-gen-go: error:inconsistent package import paths: "github.com/tensorflow/tensorflow/tensorflow/go/core/framework", "tensorflow/core/framework"
--go_out: protoc-gen-go: Plugin failed with status code 1.

[eulerwang]

@JaxSONG ,loop image into a slice , slice is []float32 type , size is 28281

[filipecosta90]

@azer if you use up to version v1.13.2 of tensorflow, you should be able to run:
protoc $PROTOC_OPTS tensorflow/tensorflow/core/framework/*.proto

[ledaiduongvnth123456]

@JaxSONG ,loop image into a slice , slice is []float32 type , size is 28_28_1

@eulerwang : How can i do with an color image ? I did that, but it seems incorrect:

inputTensorValues := make([]float32, 3*160*160)
    for i := 0; i < 160; i++ {
        for j := 0; j < 160; j++ {
            r, g, b, _ := p.At(i, j).RGBA()
            inputTensorValues[i*160*3 + j*3] = float32(r/255)
            inputTensorValues[i*160*3 + j*3 + 1] = float32(g/255)
            inputTensorValues[i*160*3 + j*3 + 2] = float32(b/255)
        }
    }

[AlexanderJLiu]

This could be help for anyone who want to generate tensorflow serving api for go step by step.