Simpletransformers: Possibility to process long documents?

[acadTags]

I tried to set max_seq_length to over 512 and there is an error (see error info below). I think this is probably because the original BERT model is trained on 512 tokens?

Is there a possibility to process longer documents based on the current implementation or any simple adaptation (e.g. shared model weights for different chunks in a long document)?

I am thinking about adapting the code for long documents, if you have any suggestions on the implementation, please kindly let me know. Many thanks!

---

Error when using max_seq_length as 513 is provided below:

Traceback (most recent call last):
  File "xxx.py", line 98, in <module>
    model.train_model(train_df)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/simpletransformers/classification/multi_label_classification_model.py", line 127, in train_model
    args=args,
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py", line 262, in train_model
    **kwargs,
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py", line 352, in train
    outputs = model(**inputs)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/simpletransformers/custom_models/models.py", line 47, in forward
    head_mask=head_mask,
  File "xxx/envs/pt100/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/transformers/modeling_bert.py", line 799, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/transformers/modeling_bert.py", line 195, in forward
    embeddings = self.dropout(embeddings)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/torch/nn/modules/dropout.py", line 58, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "xxx/anaconda/envs/pt100/lib/python3.6/site-packages/torch/nn/functional.py", line 749, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: CUDA error: device-side assert triggered

Best wishes,
A

[kinoute]

Have a look at the sliding_window feature in the README.

[ThilinaRajapakse]

Yes, BERT won't let you go over 512 tokens. sliding_window is intended to help with this. It works by splitting longer documents into "windows" to keep everything under the length limit. It's not going to work well in all cases but you can try it out and see.

[kinoute]

@ThilinaRajapakse by the way, regarding this feature, is this normal that when evaluating our model using sliding_window, the validation does not appear to be splitted?

When I start the training, the splitting works:

Converting to features started. Cache is not used.
Sliding window enabled

15761 features created from 9290 samples.

But when it gets evaluated every 50 steps, I get:

Running loss: 0.380932
Converting to features started. Cache is not used.
Sliding window enabled
1033 features created from 1033 samples.

I checked, my validation dataset contains entries with more tokens than I set with max_seq_length.

[ThilinaRajapakse]

It's split with evaluation as well. The print statement is kinda misleading there. The reason is that the feature list is flattened (increasing the number of features compared to samples) before the print statement, but it is flattened after the print statement in evaluation.

I'm a little cautious about changing it as you get the same number of predictions as you have samples even if more features are used under the hood. Maybe it would be better to not print that line with evaluation? :thinking:

[kinoute]

I guess you can use this "issue" to start replacing all the print calls with a proper logger integration 😄

[Jupiter79]

First of all, @ThilinaRajapakse , great work for creating simpletransformers, it is, as the name says, really simple to use, and I enjoy it very much.
I am also working on similar problem, multi label classification of long documents (2000 to 7000 words), and I have tried couple of different models, Bert base, Bert large, XLnet base and XLnet large. Until now, I have achieved best results with XLnet large, with max sequence set to 1024 (I wanted to try even longer sequence, but I started running into OOM problem). So, maybe XLnet could be better option for classification of long documents.

[ThilinaRajapakse]

Thank you for your kind words and your input!

That is another option of course. But at some point, your text is still going to get truncated. sliding_window is one way to ensure that it doesn't happen. It's not going to be suitable for all situations, more's the pity!

[Jupiter79]

Thanks, I tired sliding_window, but I got warning that it does not work for Multi Label classification?
I will try to experiment with different approach today: I was thinking of using some algorithm for keywords extraction, and than instead of using whole document, I thought of using extracted keywords as input for Multi Label classification. I'm going to experiment a little bit, and I'll write impressions when I finish.

[ThilinaRajapakse]

Ah yes. It's not implemented for Multi Label atm. That one is a little trickier.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.