Transformers: Faster Pegasus tokenizer tests

Created on 24 Sep 2020  路  4Comments  路  Source: huggingface/transformers

Current test_tokenization_pegasus.py takes more than a minute to run because it uses a full size tokenizer here

It should use "fixtures/test_sentencepiece.model" like tests/test_tokenization_t5.py.

Help wanted Tests
Source
picture sshleifer

All 4 comments

Any interest @stas00 ?

sshleifer picture sshleifer on 24 Sep 2020

Yes, please, you can assign this to me, but most likely will be able to start on it in a few weeks when I have free time.

stas00 picture stas00 on 24 Sep 2020

We can start this, but if we do we should wait for @thomwolf 's fast tokenizer PR to merge before we merge the fix.

sshleifer picture sshleifer on 7 Oct 2020
👍1

This is unblocked, thom merged!

sshleifer picture sshleifer on 8 Oct 2020
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hsajjad picture hsajjad  路  3Comments
lcswillems picture lcswillems  路  3Comments
quocnle picture quocnle  路  3Comments
0x01h picture 0x01h  路  3Comments
alphanlp picture alphanlp  路  3Comments