Serving: Tensorflow Object detection model export for serving.

[ashwini-git]

I have trained a Tensorflow Object detection model. and used export_inference_graph.py script to generate inference graph. following files were created -
""saved_model.pb""
""variables/"" (an empty folder)

Now I want to export model for serving. exporter.py script does not create "variables" which are required to host model on Tensorflow serving.

Please let me know changes might be required to export model for Serving in the exporter.py script.

Any possible help will be much appreciated.
Thanks

[farzaa]

Hi there!

You're almost there :). You actually have everything you need for serving - all you need is the saved_model.pb file. I can quickly show you my setup and hopefully you can use a similar setup!

I actually serve two models, at the same time - here is my file structure:

Note: in both these folders, variables is empty. This is not necessary at all.

And here is my Dockerfile.

FROM tensorflow/serving:1.13.0-gpu

WORKDIR /models
COPY all_saved_models .
COPY model_server.conf .
RUN mkdir -p league-ocr-detector/1
RUN mkdir -p fortnite-ocr-detector/1

RUN mv ./league_saved_model/league_saved_model.pb ./league_saved_model/saved_model.pb
RUN mv ./league_saved_model/saved_model.pb league-ocr-detector/1/
RUN mv ./league_saved_model/variables league-ocr-detector/1/

RUN mv ./fortnite_saved_model/fortnite_saved_model.pb ./fortnite_saved_model/saved_model.pb
RUN mv ./fortnite_saved_model/saved_model.pb fortnite-ocr-detector/1/
RUN mv ./fortnite_saved_model/variables fortnite-ocr-detector/1/

ENTRYPOINT ["tensorflow_model_server", "--model_config_file=/models/model_server.conf", "--rest_api_port=8501", "--port=8081"]

Notice how I set up the file structure as fortnite-ocr-detector/1/ and the working directory is models. The 1 is the version of the model.

Also, I use a ModelServerConfig which you can see that I copy in that looks like this:

model_config_list {
  config {
    name: 'fortnite-model'
    base_path: '/models/fortnite-ocr-detector'
    model_platform: "tensorflow",
  },
   config {
    name: 'league-model'
    base_path: '/models/league-ocr-detector'
    model_platform: "tensorflow",
  }
 }

And that's it! Let me know if you have issues making a REST request or using GRPC.

[annish07]

@farzaa Can you please tell me how to make a REST request for the same.

[ashwini-git]

@farzaa, Thank You for your response! it was very helpful.
Plz, let us know client script to make REST request.

[farzaa]

Hey there! Yeah, so a REST request is much simpler. Just be very careful with the naming of your models.

import requests
image = cv2.imread("image.jpg").astype(np.uint8)

def make_request():
    payload = {"instances": [image.tolist()]}
    res = requests.post("http://35.229.22.200:8501/v1/models/league-model:predict", json=payload)

In this case, my league-model is in the models folder. So, my exact file structure is: league-ocr-detector/1/saved_model.pb and my model.config file properly specifies the names of everything. The 1 is the version of the model and is the most confusing thing ever. This is also why we do v1 within the endpoint.

This fellow actually goes into it more in depth within his blogpost, so be sure to check it out!

https://medium.com/@pierrepaci/deploy-tensorflow-object-detection-model-in-less-than-5-minutes-604e6bb0bb04

[gowthamkpr]

Thank you @farzaa for your response. I'm closing this issue as it has been answered!