Transformers: Longformer run error

Created on 12 Sep 2020 · 3Comments · Source: huggingface/transformers

❓ Questions & Help

Details

When I train on a classification model by Longformer
def forward(self,input):
embding=input['enc']
att_mask=input['mask']
att_mask[:,[100,300,500,800,1200,]]=2
labels=input['targets']
print('jeff:',embding.device,att_mask.device,self.l1.device,embding.shape,att_mask.shape,self.maxlen)

    logit=self.l1(inputs_embeds=embding,attention_mask=att_mask)#[:2]
    return [logit,labels]

Meet error:
pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo, TensorInfo, TensorInfo, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [205,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed

max_num_extra_indices_per_batch = num_extra_indices_per_batch.max()
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/THCReduceAll.cuh:327

I checked the length of the attention_mask is same as the config.max_len which is like [bs,max_len]
Do anyone meet the same issue?

A link to original question on the forum/Stack Overflow:

Source