Transformers: T5 - Finetuning of an EncoderDecoder Model

[f-lng]

Hello,

I know that the T5 implementation is quite new, but is there already code to finetune and lateron decode from the T5 model?

As I understand most of your models are no EncoderDecoder models, so I guess that the default pipeline / code is not working for T5, is that right?

Could you point me to a script / command / piece of code for finetuning T5?

[TheEdoardo93]

As I know, there is no Python scripts for fine-tuning T5 model, at the moment.
Besides the source code you can see in this library, you can see the PR #1739 which implements T5 model.

Hello,

I know that the T5 implementation is quite new, but is there already code to finetune and lateron decode from the T5 model?

As I understand most of your models are no EncoderDecoder models, so I guess that the default pipeline / code is not working for T5, is that right?

Could you point me to a script / command / piece of code for finetuning T5?

[avostryakov]

The same question. #1739 was merged. First of all, In T5_INPUTS_DOCSTRING is said:

            To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
            (a) For sequence pairs:
                tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
            (b) For single sequences:
                tokens:         [CLS] the dog is hairy . [SEP]

At second, it looks like T5Model can work in encoder mode only. So, it's possible to treat it as usual LM:

        tokenizer = T5Tokenizer.from_pretrained('t5-small')
        model = T5Model.from_pretrained('t5-small')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids)
        last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

Maybe @thomwolf can clarify how better to fine-tune T5 for classification tasks

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.