Transformers: save as tensorflow saved model format and how to inference?

[cbqin]

❓ Questions & Help

Hi, l follow the script in readme, train a model and save as tensorflow saved_model format instead of h5 format.
When inferencing, I get some problem, I don't know how to feed the inputs to the model. Here is code.

import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule 
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

tf.saved_model.save(model,"/content/saved")

I change the last line code to get a tensorflow saved_model. I get a problem when inferencing.

loaded = tf.saved_model.load("/content/saved")
inference_func = loaded.signatures["serving_default"]
for inputs,_ in valid_dataset:
  print(inference_func(inputs))

Then I get:

TypeError                                 Traceback (most recent call last)
<ipython-input-27-7c90c411776e> in <module>()
      1 for inputs,_ in valid_dataset:
----> 2   print(inference_func(inputs))

1 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py in _call_impl(self, args, kwargs, cancellation_manager)
   1098            "of {}), got {}. When calling a concrete function, positional "
   1099            "arguments may not be bound to Tensors within nested structures."
-> 1100           ).format(self._num_positional_args, self._arg_keywords, args))
   1101     args = list(args)
   1102     for keyword in self._arg_keywords[len(args):]:

TypeError: Expected at most 0 positional arguments (and the rest keywords, of ['attention_mask', 'input_ids', 'token_type_ids']), got ({'input_ids': <tf.Tensor: id=130383, shape=(64, 128), dtype=int32, numpy=
array([[ 101, 1284, 5376, ...,    0,    0,    0],
       [ 101, 2061,  117, ...,    0,    0,    0],
       [ 101, 1130, 1103, ...,    0,    0,    0],
       ...,
       [ 101, 1109, 3302, ...,    0,    0,    0],
       [ 101, 1556, 1292, ...,    0,    0,    0],
       [ 101, 1109,  158, ...,    0,    0,    0]], dtype=int32)>, 'attention_mask': <tf.Tensor: id=130382, shape=(64, 128), dtype=int32, numpy=
array([[1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       ...,
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>, 'token_type_ids': <tf.Tensor: id=130384, shape=(64, 128), dtype=int32, numpy=
array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>},). When calling a concrete function, positional arguments may not be bound to Tensors within nested structures.

Has anyone encountered this problem before?

[mandubian]

It's a mix of 2 issues:

• you need to transform your input dict into function args
• you need to expand batch dimension in all tensors

Please try:

inference_func(**({k: tf.expand_dims(v, axis=0) for k, v in inputs.items()}))

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[xiaoyangnihao]

@cbqin @mandubian Hi, do you solve this problem, Can you explain about this, I met similar problem.
By the way,

loaded = tf.saved_model.load("/content/saved")
inference_func = loaded.signatures["serving_default"]  
# is this line necessary ??? why not  just use loaded(inputs) when inferencing
for inputs,_ in valid_dataset:
  print(inference_func(inputs))

[Shiro-LK]

@xiaoyangnihao I have some issue about incompatible shape too, have you solved the error ?