Transformers: 3 sentences as input for BertForSequenceClassification?

[mikelkl]

Hi there,

Thanks for releasing this awesome repo, it does lots people like me a great favor.

So far I've tried sentence-pair BertForSequenceClassification task, and it indeed work. I'd like to know if it is possible to use BertForSequenceClassification to model triple sentences classification problem and its input can be described as below:

**[CLS]A[SEP]B[SEP]C[SEP]**

Expecting for your reply!

Thanks & Regards

[artemisart]

Technically it is possible but BERT was not pretrained to handle multiple SEP tokens between sentences and does not have a third token_type, so I think it won't be easy to make it work. You may also want to use a new token for the second separation.

[mikelkl]

Technically it is possible but BERT was not pretrained to handle multiple SEP tokens between sentences and does not have a third token_type, so I think it won't be easy to make it work. You may also want to use a new token for the second separation.

Hi artemisart,

Thanks for your reply.

So, if someone wanna take multiple sentences as input of BertForSequenceClassification, let's say a whole passage, an alternative way is to concatenate them into a single "sentence" and then fit it in, right?

[artemisart]

I you don't have a separation (like question/answer) then yes you can just concatenate them (but you are still limited to 512 tokens).

[thomwolf]

@mikelkl I would also go with the solution and answer of @artemisart.

[alphanlp]

@artemisart hi, if i have a single sentence classification task, should the max length of sentence limited to half of 512, that is to say 256?

[artemisart]

No, it will be better if you use the full 512 tokens.

[thedrowsywinger]

wouldn't concatenating the whole passage into a single sentence mean losing context of each sentence? @artemisart

[artemisart]

No it shouldn't

[thedrowsywinger]

What if I want to check on a huge corpus, that even concatenating into one sentence exceeds the 512 token limit? @artemisart

[mikelkl]

@thedrowsywinger maybe u should try Transformer-XL