Hub: Load Bidirectional Encoder Representations from Transformers (BERT) in TF2.0

[mborsetti]

Would like to request the ability to load these v1 modules in TF2: https://tfhub.dev/s?q=bert

Thanks!

[vbardiovskyg]

Hi Mike,

FYI, it should be currently possible to load BERT modules in TF2 in a very bleeding edge setup.

With both TensorFlow and TensorFlow Hub built from head, one should be able to use the modules as:

module = hub.load("https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1")
token_signature = module.signatures["tokens"]
module_input = dict(
  input_ids=tf.constant(3, shape=[1, 4]),
  input_mask=tf.constant(1, shape=[1, 4]),
  segment_ids=tf.constant(4, shape=[1, 4]),
)
output = token_signature(**module_input)

[robianmcd]

@vbardiovskyg would this work with the tf-nightly-gpu-2.0-preview build or does tensorflow need to be built from source? Also are there any docs on building tensorflow-hub from source or is there a nightly build published? Thanks!

[vbardiovskyg]

Hi Rob, yes, the tf-nightly-gpu-2.0-preview should be fine.

Instructions for building TF-Hub at head are here: https://github.com/tensorflow/hub/blob/master/tensorflow_hub/pip_package/PIP.md

[robianmcd]

Thanks @vbardiovskyg. I got this working with tf-nightly-gpu-2.0-preview and a manually built .whl for tensorflow hub (https://www.dropbox.com/s/2oiib353k5jmnn6/tensorflow_hub-0.5.0.dev0-py2.py3-none-any.whl?dl=1 in case anyone else wants to try)

FYI I also had to add tags='train' e.g.

module = hub.load("https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1", tags='train')

[rmothukuru]

@mborsetti ,
Hope the instructions given by @vbardiovskyg and @robianmcd has addressed your request. Please confirm so that we can close this issue.

[vbardiovskyg]

Thanks for trying this out and confirming it works in the nightly.

[robianmcd]

I just noticed that the vocab_file seems to be missing when I load this from tf 2. e.g. this code doesn't work because vocab_file is "SAVEDMODEL-ASSET".

bert_module = hub.load("https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1", tags='train')

tokenization_info = bert_module.signatures['tokenization_info']()
vocab_file = tokenization_info['vocab_file'].numpy().decode("utf-8")
do_lower_case = tokenization_info['do_lower_case'].numpy()

tokenizer = FullTokenizer(vocab_file=vocab_file, do_lower_case=do_lower_case)

If I run similar code from TF 1 and TF Hub 0.4.0 then vocab_file is something like "C:\\Users\\ROBMCD~1\\AppData\\Local\\Temp\\tfhub_modules\\5a395eafef2a37bd9fc55d7f6ae676d2a134a838\\assets\\vocab.txt"

[mborsetti]

Hi @rmothukuru,

We found another solution for BERT, and am working on a different project
at the moment; I hope to be able to spend some time this weekend to test.

On Tue, 14 May 2019 at 21:24, rmothukuru [e-mail] wrote:

@mborsetti https://github.com/mborsetti ,
Hope the instructions given by @vbardiovskyg
https://github.com/vbardiovskyg and @robianmcd
https://github.com/robianmcd has addressed your request. Please confirm
so that we can close this issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/hub/issues/281?email_source=notifications&email_token=AGLWC7YIGNQRZT3KO4D3C3LPVOF77A5CNFSM4HII3F6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVNOZGI#issuecomment-492498073,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGLWC73JWGMEZLMVQTZV3ITPVOF77ANCNFSM4HII3F6A
.

[robianmcd]

BERT also doesn't seem to work hub.KerasLayer(). E.g. if I run

hub.KerasLayer("https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1", trainable=True)

I get the following error

ValueError: Importing a SavedModel with tf.saved_model.load requires a 'tags=' argument if there is more than one MetaGraph. Got 'tags=None', but there are 2 MetaGraphs in the SavedModel with tag sets [['train'], []]. Pass a 'tags=' argument to load this SavedModel.

I tried modifying KerasLayer to just hardcode tags=['train'] which seems to have fixed this error but I still haven't found any combination of parameters that doesn't result in an error being thrown.

It would be great to see an example of using BERT via KerasLayer in TF2 once this is ready.

[aishfenton]

I've also been trying the suggestions above, and so far haven't been able to find any combination that works for BERT. Best would be if it was available as a TF2 compatible KerasLayer.

[lileicv]

@vbardiovskyg Thanks for your solution. But I have a question. I want to finetune the weights in bert, how can I get the variables in bert? Hope for reply.

grads = tape.gradient(loss, .....)
optimizer.apply_gradients(zip(grads, ......), global_step=step_counter)

[lileicv]

The output is different for the same input.

The code:

import tensorflow as tf
import tensorflow_hub as hub 

tf.enable_eager_execution()

module = hub.load('/home/lilei/model/hub/bert_chinese', tags='train')

token_signature = module.signatures["tokens"]
module_input = dict(
  input_ids=tf.constant(3, shape=[5, 4]),
  input_mask=tf.constant(1, shape=[5, 4]),
  segment_ids=tf.constant(4, shape=[5, 4]),
)

for i in range(10):
    output = token_signature(**module_input)['pooled_output']
    print(output[0][0:10])

The output is:

tf.Tensor(
[ 0.84101045 -0.52301174  0.89123935  0.86203486 -0.81671154 -0.9023562
 -0.16460024  0.88539684 -0.7207676  -0.2098382 ], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.99301547  0.94316477  0.99185836 -0.44699657 -0.9081208   0.3868939
  0.5249809   0.65112615  0.24869189 -0.8822683 ], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.99922633  0.02912696  0.9999662   0.44633976  0.958979   -0.65724874
  0.47251755 -0.3443879   0.947718   -0.99842954], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.90354276  0.98951757  0.65481853  0.78175414  0.6429206  -0.32127693
  0.1595023  -0.63641524  0.67010474  0.15018927], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.9805742   0.9720993   0.6583968   0.54680395 -0.4091723  -0.9110228
 -0.70226157  0.27443266  0.0047663  -0.44588575], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.9950925   0.7994529   0.98469424  0.8409917   0.7910001   0.4103651
 -0.8078767  -0.6737721   0.88066256 -0.95471025], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.9271285   0.8283412   0.6830909   0.76988363  0.481798   -0.83320653
  0.7446666  -0.48387873  0.8283634  -0.8955145 ], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.9925193   0.97748035  0.72850645  0.9692629   0.8803055  -0.7052871
 -0.46985647 -0.8920912   0.9006626  -0.9724752 ], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.94277215  0.99123913 -0.5381528   0.9500218   0.84336615 -0.5192468
 -0.96091235 -0.94564384  0.89634675 -0.8403746 ], shape=(10,), dtype=float32)
tf.Tensor(
[ 0.54037327  0.7655801  -0.26436925  0.9818621   0.00504047 -0.19389834
 -0.6971407  -0.7678226  -0.06771909 -0.5422716 ], shape=(10,), dtype=float32)

[vbardiovskyg]

Hi @lileicv, sorry for the late response. You are using the training graph which contains non-zero dropout.

To load the inference version, use
module = hub.load('https://hub.tensorflow.google.cn/google/bert_chinese_L-12_H-768_A-12/1', tags=[])

Variables are loaded into module.variables.

[rjurney]

@vbardiovskyg @lileicv What are the arguments you pass in: input_ids, input_mask, segment_ids? Where do they come from and what do they signify? I'm not sure how to define these for my data.

[vbardiovskyg]

Hi Russel,

please take a look at https://github.com/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb.

In risking that I will mess this up, my understanding is:
input_ids - ids corresponding to tokens. These come out from the BERT tokenizer.
input_mask - just 0/1 values saying which inputs are just padding and which are real.
segment_ids - in the paper and original research this was used to separate input into 2 sentences (or more?). In the colab they just use one value 0 everywhere.

[rjurney]

@vbardiovskyg Thanks!

[vbardiovskyg]

Closing since the new set of TF2 compatible BERT models have been published:
https://tfhub.dev/s?publisher=tensorflow&q=bert

[iamxpy]

Hi Mike,

FYI, it should be currently possible to load BERT modules in TF2 in a very bleeding edge setup.

With both TensorFlow and TensorFlow Hub built from head, one should be able to use the modules as:

module = hub.load("https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1")
token_signature = module.signatures["tokens"]
module_input = dict(
  input_ids=tf.constant(3, shape=[1, 4]),
  input_mask=tf.constant(1, shape=[1, 4]),
  segment_ids=tf.constant(4, shape=[1, 4]),
)
output = token_signature(**module_input)

@vbardiovskyg Hi, I am trying the snippet, but I got the error KeyError: 'tokens', the full error message is:

Traceback (most recent call last):
  File "/home/xiepengyu/google_quest/scripts/bert_test.py", line 37, in <module>
    token_signature = module.signatures["tokens"]
  File "/home/xiepengyu/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow_core/python/saved_model/signature_serialization.py", line 195, in __getitem__
    return self._signatures[key]
KeyError: 'tokens'

My code is:

# The only difference is I am using the model `bert_en_uncased_L-12_H-768_A-12` (not `bert_uncased_L-12_H-768_A-12`)
# and try to load it from disk directly.
module = hub.load("/home/xiepengyu/google_quest/input/bert-base-from-tfhub/bert_en_uncased_L-12_H-768_A-12/") 
token_signature = module.signatures["tokens"]
module_input = dict(
  input_ids=tf.constant(3, shape=[1, 4]),
  input_mask=tf.constant(1, shape=[1, 4]),
  segment_ids=tf.constant(4, shape=[1, 4]),
)
output = token_signature(**module_input)
print(output)

The only difference is I am using the model bert_en_uncased_L-12_H-768_A-12 (not bert_uncased_L-12_H-768_A-12) and try to load it from disk directly.