Transformers: Fine-tuning TF models on Colab TPU

[NaxAlpha]

❓ Questions & Help

Hi,

I am trying to fine-tune TF BERT on Imdb dataset on Colab TPU. Here is the full notebook:

https://colab.research.google.com/drive/16ZaJaXXd2R1gRHrmdWDkFh6U_EB0ln0z

Can anyone help me what I am doing wrong?
Thanks

[TheEdoardo93]

I've read your code, and I don't see anything strange in it (I hope). It seems to be an error due to training a (whatever) model on TPUs rather than Transformers.

Do you see this issue reported in TensorFlow's GitHub? It seems to be the same error, and someone gives indications about how to resolve.

Questions & Help

Hi,

I am trying to fine-tune TF BERT on Imdb dataset on Colab TPU. Here is the full notebook:

https://colab.research.google.com/drive/16ZaJaXXd2R1gRHrmdWDkFh6U_EB0ln0z

Can anyone help me what I am doing wrong?
Thanks

[NaxAlpha]

Thanks for the help. Actually my code was inspired from this colab notebook. This notebook works perfectly but there is one major difference might be causing the problem:

• I am force installing tensorflow 2.x for transformer notebook because transformer works only for TF>=2.0 but colab uses TF 1.15 on colab for TPUs

So I went to GCP and used a TPU for TF 2.x. The error changed to this:

But yes, you are right this issue might be related to TF in general.

[TheEdoardo93]

In the official docs of GCP here, they show the current list of supported TensorFlow and Cloud TPU versions; _only_ TensorFlow 1.13, 1.14 and 1.15 are supported (they don't mention TensorFlow 2.0).
I think it's why you have the problem you've highlighted.

Thanks for the help. Actually my code was inspired from this colab notebook. This notebook works perfectly but there is one major difference might be causing the problem:

• I am force installing tensorflow 2.x for transformer notebook because transformer works only for TF>=2.0 but colab uses TF 1.15 on colab for TPUs

So I went to GCP and used a TPU for TF 2.x. The error changed to this:

But yes, you are right this issue might be related to TF in general.

[bestpredicts]

tpu supports . tf2.1

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.