Serving: How to use Vocabulary(e.g. cnn-text-classification model) in Tensorflow-Serving? ... would be appreciated!...

[wenfeixiang1991]

Ultimate purpose:

I want to establish a web service that could classifiy the text by CNN model(Implementing a CNN for Text Classification in TensorFlow，which means the input must be text (type is string), and output must be string(e.g. some kinds of labels, not tensor or numpy array).

Question description:

We constructed cnn text classify model basing on paper
http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow.

In my case, I need to construct vocabulary using my train data(which is text) firstly.

vocab_processor = tensorflow.contrib.learn.preprocessing.VocabularyProcessor(max_length)
input_x = numpy.array(list(vocab_processor.fit_transform(my_train_data)))
# Write vocabulary
vocab_processor.save(os.path.join(out_dir, "vocab"))

Then I got a file named 'vocab' in my disk, and when I training my model, 'input_x' is feeded, let's see portion of my model:

input_x = tf.placeholder(tf.int32, [None, sequence_length], name="input_x")
input_y = tf.placeholder(tf.float32, [None, num_classes], name="input_y")
# Embedding layer
with tf.device('/cpu:0'), tf.name_scope("embedding"):
W = tf.Variable(
tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
name="W")
embedded_chars = tf.nn.embedding_lookup(W, input_x)
# Output layer
predictions = tf.nn.sigmoid(scores, name="predictions")

And when I predict, I need use 'label reverse dictionary 'from 'predictions' transferring to label string.
e.g.
{
"label_dic": {
"0": "bad",
"1": "middle",
"2": "good",
"3": "perfect"
},
"label_reverse_dic": {
"bad": 0,
"middle": 1,
"good": 2,
"perfect": 3,
}
}

Now I got two static file in my disk, and different model probably have different vocabulary or label_reverse_dic, and notice that Tensorflow-serving only recognize proto model input and outputs, in my model above which means your input must be tf.int32(......name="input_x") , so if you want use tensorflow-serving to prediction, you need transfer your text string to numpy by using tensorflow.contrib.learn.preprocessing.VocabularyProcessor.restore(vocab_path), which is exactly same vocabulary when you training model.

Now problem is coming:

1. Tensorflow-serving only recognize tensor input?
   e.g. input_x = tf.placeholder(tf.int32, [None, sequence_length], name="input_x")

2. How could I add additional operation in Tensorflow-serving such as transfer word to word index by using vocabulary and transfer tensor to label by using 'label_recerse_dic 'when prediction?

My idea for now

The most stupid solution in my head for now is using django/tornado + python_client + tensorflow-serving, deal with input text、vocabulary and business logic in python client, then grpc get return from tensorflow-serving, then broadcast in django/tornado..., but when I change tensorflow model, tensorflow-serving could recognize new vision, django/tornado can not, that is make no sense for using TF-serving.
I 've been reading plenty of documents and issues on git, but still confused with how exactly using SavedModelBundle assets_collection servable? it seems helps!

What I've done

1. I already run example in Tutorial success and I know how to export a proto-model that TF-serving recognize by using saved_model, also basic method in example such as builder.add_meta_graph_and_variables, ..signature.., assets_collection

2. I already known that I could export model with vocab and static file by using assets_collection, and I think I could get model structure like this:
   assets/ --------Subfolder called assets. Contains auxiliary files such as vocabularies, etc.
assets.extra/ --------Subfolder where higher-level libraries and users can add their own assets that co-exist with the model, but are not loaded by the graph.
variables/ --------variables are files that hold the serialized variables of the graphs.
variables.data-?????-of-?????
variables.index
saved_model.pb

But I still do not know how asset and asset.extra using in prediction. there is no example I can referring.

1. I look up this issues'how to serve a chatbot's model #264'
   https://github.com/tensorflow/serving/issues/264, and I found that you need to write c++ code when prediction?

2. I noticed that:

Creating a new kind of servable
This document explains how to extend TensorFlow Serving with a new kind of servable. The most > prominent servable type is SavedModelBundle, but it can be useful to define other kinds of servables, > to serve data that goes along with your model. Examples include: a vocabulary lookup table, feature > transformation logic. Any C++ class can be a servable, e.g. int, std::map or any class > defined in your binary -- let us call it YourServable.
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/custom_servable.md

Model: A machine-learned model is represented by one or more servables. Examples of servables > are:

TensorFlow session or wrappers around them, such as SavedModelBundle.
Other kinds of machine-learned models.
Vocabulary lookup tables.
Embedding lookup tables.
A composite model could be represented as multiple independent servables, or as a single > composite servable. A servable may also correspond to a fraction of a Model, for example with a large > lookup table sharded across many Manager instances. https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/serving_advanced.md

But how? It is really confusing me!

Final specific questions

add_meta_graph_and_variables(
sess,
tags,
signature_def_map=None,
assets_collection=None,
legacy_init_op=None,
clear_devices=False,
main_op=None
)

1. what is main_opfor?
2. what is legacy_init_opfor?
3. what does legacy_init_op = tf.group(tf.tables_initializer(), name='legacy_init_op') mean?
4. what does assign_filename_op = filename_tensor.assign(original_assets_filename) mean?

[wenfeixiang1991]

Is there anybody can help me in this？ Or any example about How to use look-up-table/word embedding/ by vocabulary in TF-Serving? Or make another servable through C++ code? I really appreciate it! :)

[ghost]

@wenfeixiang1991 Yes there is a way. We have a working model in our production systems. Here is an small example how you achieve it.

[wenfeixiang1991]

@kishorenayar I really appreciate your proposal, and I checked out the example you give me also other articles in _MEDIUMS_ you've written, then I realize that you are very impressive man in machine learning field. The articles are very concise and clear for me.
I have two prior questions according your paper, according:

The really cool thing from my perspective about the Estimators API is that using it is a very easy way > to create distributed TensorFlow models.
So, please, please, please, if you see a TensorFlow sample that doesn’t use the Estimators API, ignore it. It will be a lot of work to make it work on your production (read: large) datasets — there will be monitors, coordinators, parameter servers, and all kinds of systems programming craziness that you don’t want to have to dive into. Start with the Estimator API and use the Experiment class. (Disclaimer: my views, not that of my employer).

1. Question: does Estimators API can export a TF-serving model structure like this? Which is what _Tensorflow-Serving Tutorial_ told me.

More explanation:
As far as I know, there are two different export models method in tensorflow, first one is

saver = tf.train.Saver(tf.global_variables(), max_to_keep=FLAGS.num_checkpoints)
saver.save(sess, checkpoint_prefix, global_step=current_step)

second is

builder = tf.saved_model.builder.SavedModelBuilder(export_path)
builder.add_meta_graph_and_variables( sess, [tf.saved_model.tag_constants.SERVING], signature_def_map={ '...': ..., tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: classification_signature }, assets_collection=tf.get_collection(tf.GraphKeys.ASSET_FILEPATHS)) legacy_init_op=legacy_init_op))
builder.save()

Now I want to use Tensorflow-Serving, so the second one is what I need, it could export a protocol buffers model that TF-serving can recognize. The problem I had is I don't know how to make serving's iput/output to string, like using vocabulary make tf.placeholder data type to tf.string then calculation in tensorflow, after read your code I think I could fix it out using

input:
table = lookup.index_table_from_file(vocabulary_file=..., num_oov_buckets=1, default_value=-1)
'title': tf.placeholder(tf.string, [None])
titles = tf.squeeze(features['title'], [1])
words = tf.string_split(titles)
densewords = tf.sparse_tensor_to_dense(words, default_value=PADWORD)
numbers = table.lookup(densewords)

output:
'source': tf.gather(TARGETS, tf.argmax(logits, 1))

api like tf.sparse_tensor_to_dense(), tf.gather() is the key! It also approved your another point of view "one stumbling block was that many tutorials would use terms I wasn’t familiar with. In addition,
many of the tutorials use different levels of the TensorFlow API, or different high-level wrappers, adding to my confusion."

1. Question: according your articles and interview by Google, I think you are supporter on serving machine learning in Google Cloud Platform, special serving in production which is also I interested in. But what relationshipe of GCP and Tensorflow-Serving?
   After I learned TF-serving Tutorial I know that is C++ open source project for serving tensorflow model, It's a grpc serving high performance and I could do model version management and all kinds of config by custom servable which I don't understand yet. After configuring, bazel it and put service on docker???
   But in your case you didn't tell much about Tensorflow-serving, you put export model on Cloud ML and I think you might mentioned Restful not grpc, is there any config for GCP or just bazel Tensorflow-serving on GCP or totally different things? You are using GCP instead of TF-serving? if so, is there version control or model config like TF-serving in GCP?
   or I get it in wrong way?
   Should I use Cloud ML instead of bazel tensorflow-serving/docker ?

PS: I majored in mathematics while I was in College and graduate school, I could understand machine learning theory very well in math way, but I am roughly an engineering greener like how implement it and serving for production in computer science filed. I start coding one and a half year, and I study python/tensorflow over 4 months. I wish I describe my purpose as clearly as possible!
Looking forward to your reply! :)

[ghost]

First of all, I'm not the one who wrote those medium article.

To answer to your question yes estimator will generate pb file and asset folder which can be deployed in cloudml or docker, etc.

To answer your second question, you can deploy the model in docker or cloud ml.

So far I have used only cloud ml and not docker. Cloud ml helps you train model on large clusters. You can train model in few minutes or hours. It also provides to deploy the model in server so that others can query through rest API. You can also train in local machine, generate the model and deploy in cloud ml. One big advantage of cloud ml is it is integrated with cloud data flow which is a killer feature.

[wenfeixiang1991]

Copy that and thank u!

[wenfeixiang1991]

@kishorenayar I almost finished my tensorflow-serving exploration，I could simply using tf-serving in our production, thank you for your example and opinion, that is the key to my work.
There is a one more question that I tried and I really don't know how to deal with, it is about how to return multiple labels in multi-label ML-model using tf method?

[ghost]

You can try using tf.nn.sparse_softmax_cross_entropy_with_logits for multi
class.

For example of the usage see here -
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py#L522

On 12 Sep 2017 8:13 a.m., "Feixiang.Wen" [e-mail] wrote:

@kishorenayar https://github.com/kishorenayar I almost finished my
tensorflow-serving exploration，I could simply using tf-serving in our
production, thank you for your example and opinion, that is the key to my
work.
There is a one more question that I tried and I really don't know how to
deal with, it is about how to return multiple labels in multi-label
ML-model using tf method?
https://stackoverflow.com/questions/46090436/how-to-get-multi-label-output-after-using-tf-nn-sigmoid

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/serving/issues/582#issuecomment-328716538,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKOlE_VKW2aeb9Q1wtyw2NG1leH9RWpLks5she_ugaJpZM4PImRZ
.

[wenfeixiang1991]

@kishorenayar Thank you bro, but I pretty sure that is not my solution which is a multi-classes issue instead of multi-labels issue.
Let's suppose a scenario like this, the input is a paragraph which is a Personalized signature in social web, and we give this paragraph the labels such as 'sports man', 'confidence', 'pride', 'Introvert', 'quiet'..., and this is one input but multi-labels which is any combinations by those labels. I can do this using tf.nn.sigmoid() which I see this is a series of binary problem.
Practice proved it works but a little difficult transform the one-hot label when you prediction which is transfer tf.nn.sigmoid() result to string ...

[ghost]

Sorry, I don't have a exact solution with me. Please refer the following
links which may help you

https://medium.com/towards-data-science/multi-label-image-classification-with-inception-net-cbb2ee538e30
https://stackoverflow.com/questions/35400065/multilabel-text-classification-using-tensorflow
https://stackoverflow.com/questions/42207928/how-to-perform-a-multi-label-classification-with-tensorflow

On Mon, Sep 18, 2017 at 8:21 AM, Feixiang.Wen [e-mail]
wrote:

Thank you bro, but I pretty sure that is not my solution which is a
multi-classes problem instead of ,multi-labels.
Let's suppose a scenario like this, the input is a paragraph which is a
Personalized signature in social web, and we give this paragraph the labels
such as 'sports man', 'confidence', 'pride', 'Introvert', 'quiet'..., and
this is one input but multi-labels which is any combinations by those
labels. I can do this using tf.nn.sigmoid() which I see this is a series
of binary problem.
Practice prove it works but little difficult transform the one-hot label
when you prediction which is transfer tf.nn.sigmoid() result to string ...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/serving/issues/582#issuecomment-330117282,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKOlE2rPV7mG_tVLacpr2BZ8FLaC7A9lks5sjdqugaJpZM4PImRZ
.

[wenfeixiang1991]

@kishorenayar Thank you very much!

[JoelGooch]

Is it possible to convert the incoming Tensor string (placeholder) to vector space inside of the saved model export? I am struggling as you cannot perform string operations on tensors such as converting to lower case, removing whitespace, special characters, etc on a tensor object. Unless you start a session and .eval() but that is not the method when exporting saved model bundle? Any help please... Thanks

[ghost]

As of now, Tensorflow does not support that option directly. You have use
cloud dataflow or tf-transform. More on here
https://research.googleblog.com/2017/02/preprocessing-for-machine-learning-with.html

On Fri, Sep 22, 2017 at 8:22 PM, JoelGooch [e-mail] wrote:

Is it possible to convert the incoming Tensor string (placeholder) to
vector space inside of the saved model export? I am struggling as you
cannot perform string operations on tensors such as converting to lower
case, removing whitespace, special characters, etc on a tensor object.
Unless you start a session and .eval() but that is not the method when
exporting saved model bundle? Any help please... Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/serving/issues/582#issuecomment-331469459,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKOlE9IhAKBF6x8_aTRQMZ4BVUDOuQBHks5sk8msgaJpZM4PImRZ
.

[snakeztc]

@wenfeixiang1991 May I ask you if you have found a solution to load vocabulary file in Tensoflow Serving? @kishorenayar solution is pretty good, but that uses GCP. Thanks!

[vitalyli]

better way is to load into the graph itself

[ideabrdg]

Following up on snakeztc's question; "@wenfeixiang1991 May I ask you if you have found a solution to load vocabulary file in Tensoflow Serving? @kishorenayar solution is pretty good, but that uses GCP. Thanks!"

[vacing]

Following up on snakeztc's question; "@wenfeixiang1991 May I ask you if you have found a solution to load vocabulary file in Tensoflow Serving? @kishorenayar solution is pretty good, but that uses GCP. Thanks!"

I have the same question...

[ghost]

@wenfeixiang1991, Check it out here for multi-label classification on tensorflow serving

[sany13-94]

@wenfeixiang1991 HI can you share a tutorial on how to serve a cnn text classification with tensorflow serving

[ghost]

@sany13-94 Here is an example of cnn text classification

[sany13-94]

@KishoreKarunakaran Thank youu

[25julia]

@wenfeixiang1991 I am using the same model for a text classification problem, can you please tell me how did you define the input in prediction_signature, because the model reads raw data from a file and then transforms it in numpy array, meanwhile the input in the prediction signature is required to be tensor. I would really appreciate if you could give me some hints on how to do that, so I can later call from a client with a raw input sentence