Bert: run_classifier.py missing online prediction function

[brightmart]

Hi,

how can we use fine-tuning model for online prediction.
the input is a sentence, the output is a possibility distribution.

currently, as we can see, prediction is possible by using a file(like test.tsv), but it may not feasible to use file based style.

[jacobdevlin-google]

You'll have to write whatever "frontend" wrapper is appropriate to your use case. You should be able to use the existing code in run_classifier.py as a guide.

[brightmart]

run_classifier_predict_online.py.txt

able to deploy it for online prediction by using session and feed style settings.

shall i make it works it all tasks, then send a merging request?

[CoSeCant-csc]

@brightmart nice one.
i failed to use estimator.predict to do online prediction, cuz it seems to reload the graph every time once estimator.predict is called.

is that the reason that you choose the way of session and feed style to solve this issue, right

able to deploy it for online prediction by using session and feed style settings.

[brightmart]

the reason why I used session and feed style settings is that training and evaluation is designed based on files, so they load data from files then do training or evaluation. but input of online prediction is based on sequence/strings/tokens.

[mokundong]

@brightmart nice one.
i failed to use estimator.predict to do online prediction, cuz it seems to reload the graph every time once estimator.predict is called.

is that the reason that you choose the way of session and feed style to solve this issue, right

able to deploy it for online prediction by using session and feed style settings.

is there any other good solutions to avoid reload the graph every time once estimator.predict is called ?

[xikunlun001]

@brightmart
not working by the error on:
saver = tf.train.Saver()

[cdathuraliya]

Thanks a lot @brightmart for sharing your script! I cleaned it up a little bit and added notebook to demonstrate the usage. You can find it here.

[brightmart]

great

[JimAva]

Can you guys share your Flask app.py code? I'm having a hard time finding any samples for BERT serving through Flask. Thank you in advance.