Bert-as-service: How to get probability from a finetuned classify model

[Gpwner]

Prerequisites

Please fill in by replacing [ ] with [x].

• [x] Are you running the latest bert-as-service?
• [x] Did you follow the installation and the usage instructions in README.md?
• [x] Did you check the FAQ list in README.md?
• [x] Did you perform a cursory search on existing issues?

System information

Some of this information can be collected via this script.

• OS Platform and Distribution :Ubuntu 16.04
• TensorFlow installed from (source or binary):binary
• TensorFlow version:1.12.0
• Python version:Python 3.6.8
• bert-as-service version: 1.7.5
• GPU model and memory:Tesla P 100 12GB
• CPU model and memory:

---

Description

Please replace YOUR_SERVER_ARGS and YOUR_CLIENT_ARGS accordingly. You can also write your

I am trying 'Serving a fine-tuned BERT model ' of bert as service,I follow the guide.Everything run fine.
But I wonder since I am serving a classifier model,why the result is still be the embedding of sentence?
Actually ，I want to get the possibility of each class from the finetuned classifier model.

What should I do to get the possibility?Thanks in advance!

I'm using this command to start the server:

-model_dir=uncased_L-12_H-768_A-12/
-tuned_model_dir=/tmp/mrpc_output/
-ckpt_name=model.ckpt-343

and calling the server via:

from bert_serving.client import BertClient
bc = BertClient()  # ip address of the GPU machine
res=bc.encode(['First do it', 'then do it right', 'then do it better'])

Then this issue shows up:

...

[mt-07]

I'm trying to use bert-as-service exactly for the same purpose and stumbled upon the same issue. Guide on how to get class probabilities in sentence pair classification tasks (like MRPC or MNLI) would be very helpful!

[ffabulous]

+1 Exactly what I'm waiting for :grin:

[hanxiao]

related to #210 #216

okay I understand serving an end-to-end model seems like a common requirement. However, this isn't the motivation behind bert-as-service, of which I provide a scalable feature extraction service for downstream tasks.

Supporting end2end model isn't that difficult, a quick hack will do. The major work is to rethink a friendly-yet-compatible-enough API. I understand one may not care about API, but I do.

That said, serving end2end model isn't my priority for now. You are welcome to make your own by forking this repo or contribute to this repo by making a PR.

[ZRiddle]

My hack to do this was save out the weights after fine-tuning BERT:

weights = estimator.get_variable_value('output_weights')
bias = estimator.get_variable_value('output_bias')
np.savetxt('weights.csv', weights, delimiter=",")
np.savetxt('bias.csv', bias, delimiter=",")

Then to edit graph.py and hijack the CLS_TOKEN pooling strategy.

My edit to line 108 in this file (https://github.com/hanxiao/bert-as-service/blob/master/server/bert_serving/server/graph.py)

elif args.pooling_strategy == PoolingStrategy.FIRST_TOKEN or \
    args.pooling_strategy == PoolingStrategy.CLS_TOKEN:

    #pooled = tf.squeeze(encoder_layer[:, 0:1, :], axis=1)
    output_layer = model.get_pooled_output()
    hidden_size = output_layer.shape[-1].value
    output_weights = tf.convert_to_tensor(np.loadtxt('weights.csv', delimiter=','), dtype=tf.float32)
    output_bias = tf.convert_to_tensor(np.loadtxt('bias.csv', delimiter=','), dtype=tf.float32)

    logits = tf.matmul(output_layer, output_weights, transpose_b=True)
    logits = tf.nn.bias_add(logits, output_bias)
    pooled = tf.nn.softmax(logits, axis=-1)

(Yes, I know this is a crappy hack)

Better solution might be to add PREDICT or something to the PoolingStrategy ENUM class. Also this might be re-loading the files for every prediction which is unnecessary.

[leerumor]

@ZRiddle In fact there is no need to store the fine tuned weights, just add the variables and load the fine tuned model after 'run_classifier'.
I add a new pooling strategy, and the following codes in graph.py:

    if args.pooling_strategy == PoolingStrategy.CLASSIFICATION:
                hidden_size = 768
                output_weights = tf.get_variable(
                    "output_weights", [args.num_labels, hidden_size],
                    initializer=tf.truncated_normal_initializer(stddev=0.02))

                output_bias = tf.get_variable(
                    "output_bias", [args.num_labels], initializer=tf.zeros_initializer())

add the above variables before "tvars = tf.trainable_variables()"

then for pooling:

     elif args.pooling_strategy == PoolingStrategy.CLASSIFICATION:
                    pooled = tf.squeeze(encoder_layer[:, 0:1, :], axis=1)
                    logits = tf.matmul(pooled, output_weights, transpose_b=True)
                    logits = tf.nn.bias_add(logits, output_bias)
                    pooled = tf.nn.softmax(logits, axis=-1)

don't forget to add the following args:
num_labels

[pjmha]

@ZRiddle In fact there is no need to store the fine tuned weights, just add the variables and load the fine tuned model after 'run_classifier'.
I add a new pooling strategy, and the following codes in graph.py:

    if args.pooling_strategy == PoolingStrategy.CLASSIFICATION:
                hidden_size = 768
                output_weights = tf.get_variable(
                    "output_weights", [args.num_labels, hidden_size],
                    initializer=tf.truncated_normal_initializer(stddev=0.02))

                output_bias = tf.get_variable(
                    "output_bias", [args.num_labels], initializer=tf.zeros_initializer())

add the above variables before "tvars = tf.trainable_variables()"

then for pooling:

     elif args.pooling_strategy == PoolingStrategy.CLASSIFICATION:
                    pooled = tf.squeeze(encoder_layer[:, 0:1, :], axis=1)
                    logits = tf.matmul(pooled, output_weights, transpose_b=True)
                    logits = tf.nn.bias_add(logits, output_bias)
                    pooled = tf.nn.softmax(logits, axis=-1)

don't forget to add the following args:
num_labels
这句代码 : pooled = tf.squeeze(encoder_layer[:, 0:1, :], axis=1)
需要修改为 ：pooled = model.get_pooled_output()
你觉得呢？

[pjmha]

@ZRiddle In fact there is no need to store the fine tuned weights, just add the variables and load the fine tuned model after 'run_classifier'.
I add a new pooling strategy, and the following codes in graph.py:

    if args.pooling_strategy == PoolingStrategy.CLASSIFICATION:
                hidden_size = 768
                output_weights = tf.get_variable(
                    "output_weights", [args.num_labels, hidden_size],
                    initializer=tf.truncated_normal_initializer(stddev=0.02))

                output_bias = tf.get_variable(
                    "output_bias", [args.num_labels], initializer=tf.zeros_initializer())

add the above variables before "tvars = tf.trainable_variables()"

then for pooling:

     elif args.pooling_strategy == PoolingStrategy.CLASSIFICATION:
                    pooled = tf.squeeze(encoder_layer[:, 0:1, :], axis=1)
                    logits = tf.matmul(pooled, output_weights, transpose_b=True)
                    logits = tf.nn.bias_add(logits, output_bias)
                    pooled = tf.nn.softmax(logits, axis=-1)

don't forget to add the following args:
num_labels
这句代码 : pooled = tf.squeeze(encoder_layer[:, 0:1, :], axis=1)
需要修改为 ：pooled = model.get_pooled_output()
你觉得呢？

[blazejdolicki]

Do the predictions you obtain actually make sense after those changes? I tried a bunch of sentences after aforementioned modifications with my fine-tuned sentiment classification model and the predictions seem to be completely random. However, when I load the same model with bert library I get great results, so the model itself is fine for sure. I think maybe the random predictions are due to using placeholders as input_ds, input_mask etc. @hanxiao do you have any idea what could go wrong?

[lapolonio]

based on: https://github.com/google-research/bert/blob/master/run_classifier.py#L606
It should be something like:

logits = tf.matmul(model.pooled_output, output_weights, transpose_b=True)
logits = tf.nn.bias_add(logits, output_bias)
probabilities = tf.nn.softmax(logits, axis=-1)
log_probs = tf.nn.log_softmax(logits, axis=-1)