Hub: small_bert/bert_uncased_L-*_H-768_A-12/1 models produce different embeddings from released checkpoints

[chiragjn]

24 BERT checkpoints were released with Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. These are linked in the readme at https://github.com/google-research/bert

These 24 models are also now available on tfhub: https://tfhub.dev/s?q=small_bert. However, out of the 24, 6 of them that have 768 hidden size and 12 attention heads produce different results (embeddings) from their checkpoint versions while rest seem to be equivalent.

Testing code here: https://gist.github.com/chiragjn/8dfd08078403d005fbd80150c3c3999a

Perhaps there was a mistake in exporting to tfhub?

[arnoegw]

Thank you very much for your report. We were able to confirm it and are looking more into it.

[arnoegw]

Thank you again for your report. The full set of small_bert modules has been republished as version /2. They are all equivalent now with the previously published checkpoints. (Sizes L-*_H-768_A-12 have been updated; sizes L-*_H-128_A-2, L-*_H-256_A-4 and L-*_H-512_A-8 are unchanged from version /1 but get the same version number for convenience.)