Flair: CUDA error: device-side assert triggered

Created on 15 Jan 2019 · 12Comments · Source: flairNLP/flair

when I using 'bert-base-chinses'

RuntimeError Traceback (most recent call last)
in
1 document_embeddings: DocumentLSTMEmbeddings = DocumentLSTMEmbeddings(embeddings,hidden_size=1024,reproject_words=True,reproject_words_dimension=512)

~/anaconda3/envs/flair/lib/python3.6/site-packages/flair/embeddings.py in __init__(self, embeddings, hidden_size, rnn_layers, reproject_words, reproject_words_dimension, bidirectional, dropout, word_dropout, locked_dropout)
1459
1460 if torch.cuda.is_available():
-> 1461 self.cuda()
1462
1463 @property

~/anaconda3/envs/flair/lib/python3.6/site-packages/torch/nn/modules/module.py in cuda(self, device)
258 Module: self
259 """
--> 260 return self._apply(lambda t: t.cuda(device))
261
262 def cpu(self):

~/anaconda3/envs/flair/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
185 def _apply(self, fn):
186 for module in self.children():
--> 187 module._apply(fn)
188
189 for param in self._parameters.values():

~/anaconda3/envs/flair/lib/python3.6/site-packages/torch/nn/modules/module.py in _apply(self, fn)
191 # Tensors stored in modules are graph leaves, and we don't
192 # want to create copy nodes, so we have to unpack the data.
--> 193 param.data = fn(param.data)
194 if param._grad is not None:
195 param._grad.data = fn(param._grad.data)

~/anaconda3/envs/flair/lib/python3.6/site-packages/torch/nn/modules/module.py in (t)
258 Module: self
259 """
--> 260 return self._apply(lambda t: t.cuda(device))
261
262 def cpu(self):

RuntimeError: CUDA error: device-side assert triggered

bug

Source

illcat

Most helpful comment

Hi Alan,
I am also facing same issue
it happens with custom data.
Please suggest

Thanks
Mahesh

search4mahesh on 11 Apr 2019

👍3

All 12 comments

I can confirm this error. The strange thing is, that this happens only for some datasets. E.g the Dutch CoNLL 2003 dataset (that I used here). I could reproduce this on two machines with latest PyTorch version (I even tried recent nightly version).

stefan-it on 15 Jan 2019

Here's a step-to-step guide to reproduce the error:

wget https://www.clips.uantwerpen.be/conll2002/ner/data/ned.train
wget https://www.clips.uantwerpen.be/conll2002/ner/data/ned.testa
wget https://www.clips.uantwerpen.be/conll2002/ner/data/ned.testb

# Recode it
recode l1..u8 ned.train
recode l1..u8 ned.testa
recode l1..u8 ned.testb

Training code:

import gensim
import re

from pathlib import Path

from flair.data import Sentence, TaggedCorpus, Token
from flair.data_fetcher import NLPTaskDataFetcher, NLPTask
from flair.embeddings import TokenEmbeddings, WordEmbeddings, StackedEmbeddings, CharLMEmbeddings, BertEmbeddings
from typing import List

columns = {0: 'text', 1: 'pos', 2: 'ner'}

corpus: TaggedCorpus = NLPTaskDataFetcher.load_column_corpus(Path("."), columns, train_file="ned.train",
                                                        dev_file="ned.testa",
                                                        test_file="ned.testb",
                                                        tag_to_biloes='ner')

tag_type = 'ner'

tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)

embedding_types: List[TokenEmbeddings] = [
    WordEmbeddings('nl'),
    BertEmbeddings('bert-base-multilingual-cased', layers='-12')
    #custom_embedding,
    #char_lm_forward,
    #char_lm_backward
]

embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)

from flair.models import SequenceTagger

tagger: SequenceTagger = SequenceTagger(hidden_size=512,
                                        embeddings=embeddings,
                                        tag_dictionary=tag_dictionary,
                                        tag_type=tag_type,
                                        use_crf=True)

from flair.trainers import ModelTrainer 
from flair.training_utils import EvaluationMetric

trainer: ModelTrainer = ModelTrainer(tagger, corpus)

trainer.train('resources/taggers/ner-dutch-bert-fasttext-layer_minus-12',
              EvaluationMetric.MICRO_F1_SCORE,
              learning_rate=0.1,
              mini_batch_size=1,
              max_epochs=500)

After a few steps the following error message is thrown:

2019-01-15 09:20:07,729 Reading data from .
2019-01-15 09:20:07,730 Train: ned.train
2019-01-15 09:20:07,730 Dev: ned.testa
2019-01-15 09:20:07,730 Test: ned.testb
2019-01-15 09:20:53,777 ----------------------------------------------------------------------------------------------------
2019-01-15 09:20:53,777 Evaluation method: MICRO_F1_SCORE
2019-01-15 09:20:53,779 ----------------------------------------------------------------------------------------------------
2019-01-15 09:20:53,892 epoch 1 - iter 0/15806 - loss 9.91631985
2019-01-15 09:22:54,614 epoch 1 - iter 1580/15806 - loss 5.74029187
2019-01-15 09:24:56,232 epoch 1 - iter 3160/15806 - loss 4.84571188
2019-01-15 09:26:56,197 epoch 1 - iter 4740/15806 - loss 4.37643248
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "train_layer_minus_12.py", line 49, in <module>
    max_epochs=500)
  File "/mnt/flair/flair/trainers/trainer.py", line 150, in train
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]    loss = self.model.forward_loss(batch)
: block: [10,0  File "/mnt/flair/flair/models/sequence_tagger_model.py", line 248, in forward_loss
,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]    features, lengths, tags = self.forward(sentences, sort=sort)
: block: [10  File "/mnt/flair/flair/models/sequence_tagger_model.py", line 301, in forward
,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
    self.embeddings.embed(sentences)
/pytorch/aten/src/THC/THCTensorIndex.cu  File "/mnt/flair/flair/embeddings.py", line 123, in embed
:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362    embedding.embed(sentences)
: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]  File "/mnt/flair/flair/embeddings.py", line 56, in embed
: block: [10,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize    self._add_embeddings_internal(sentences)
` failed.
  File "/mnt/flair/flair/embeddings.py", line 914, in _add_embeddings_internal
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0    layer_output = all_encoder_layers[int(layer_index)].detach().cpu()[sentence_index]
,0RuntimeError: CUDA error: device-side assert triggered
], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

Here's another traceback with version 0.4.0 of pytorch-pretrained-bert:

Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
2019-01-15 09:31:20,616 Reading data from .
2019-01-15 09:31:20,616 Train: ned.train
2019-01-15 09:31:20,616 Dev: ned.testa
2019-01-15 09:31:20,616 Test: ned.testb
2019-01-15 09:32:06,778 ----------------------------------------------------------------------------------------------------
2019-01-15 09:32:06,778 Evaluation method: MICRO_F1_SCORE
2019-01-15 09:32:06,780 ----------------------------------------------------------------------------------------------------
2019-01-15 09:32:06,955 epoch 1 - iter 0/15806 - loss 132.80726624
2019-01-15 09:34:06,655 epoch 1 - iter 1580/15806 - loss 6.06238581
2019-01-15 09:36:08,259 epoch 1 - iter 3160/15806 - loss 5.16468262
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [10,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "train_layer_minus_12.py", line 49, in <module>
    max_epochs=500)
  File "/mnt/flair/flair/trainers/trainer.py", line 150, in train
    loss = self.model.forward_loss(batch)
  File "/mnt/flair/flair/models/sequence_tagger_model.py", line 248, in forward_loss
    features, lengths, tags = self.forward(sentences, sort=sort)
  File "/mnt/flair/flair/models/sequence_tagger_model.py", line 301, in forward
    self.embeddings.embed(sentences)
  File "/mnt/flair/flair/embeddings.py", line 123, in embed
    embedding.embed(sentences)
  File "/mnt/flair/flair/embeddings.py", line 56, in embed
    self._add_embeddings_internal(sentences)
  File "/mnt/flair/flair/embeddings.py", line 914, in _add_embeddings_internal
    layer_output = all_encoder_layers[int(layer_index)].detach().cpu()[sentence_index]
RuntimeError: CUDA error: device-side assert triggered

Flair version: bd1bc6306284c1c71781b503c244f5d3c9e29526
Tested pytorch-pretrained-bert versions: 0.3.0 and 0.4.0.

stefan-it on 15 Jan 2019

This is it.

illcat on 15 Jan 2019

I can confirm this error. The strange thing is, that this happens only for some datasets. E.g the Dutch CoNLL 2003 dataset (that I used here). I could reproduce this on two machines with latest PyTorch version (I even tried recent nightly version).

I found out the reason is that the sentence is too long.

illcat on 16 Jan 2019

Aha interesting. Could you create a minimal example with a problem sentence to reproduce the error?

alanakbik on 16 Jan 2019

Many thanks for developing this wonderful library!
I recently encountered the same issue when I was trying to use BertEmbeddings. Are we going to have a fix in upcoming 0.4.1 release?

gccome on 29 Jan 2019

👍1

I hope so - we'll definitely investigate this before the next release so hopefully we can find a solution.

alanakbik on 29 Jan 2019

@alanakbik Can we put a max_sequence_length limit after tokenization so that BertEmbedding won't exceed its max length? I noticed this is the case in BioBERT implementation. https://github.com/dmis-lab/biobert/blob/master/run_ner.py#L239

gccome on 25 Feb 2019

Hi Alan,
I am also facing same issue
it happens with custom data.
Please suggest

Thanks
Mahesh

search4mahesh on 11 Apr 2019

👍3

C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [96,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [97,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [98,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [99,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [100,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [101,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [102,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [103,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [104,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [105,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [106,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [107,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [108,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [109,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [110,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [111,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [112,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [113,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [114,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [115,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [116,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [117,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [118,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [119,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [120,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [121,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [122,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [123,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [124,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [64,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [65,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [66,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [67,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [68,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [69,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [70,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [71,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [72,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [73,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [74,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [75,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [76,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [77,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [78,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [79,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [80,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [81,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [82,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [83,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [84,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [85,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [86,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [87,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [88,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [89,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [90,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [91,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [92,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [93,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [94,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [32,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [33,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [34,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [35,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [36,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [37,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [38,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [39,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [40,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [41,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [42,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [43,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [44,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [45,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [46,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [47,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [48,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [49,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [50,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [51,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [52,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [53,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [54,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [55,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [56,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [57,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [58,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [59,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [60,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [96,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [97,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [98,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [99,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [100,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [101,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [102,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [103,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [104,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [105,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [106,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [107,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [108,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [109,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [110,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [111,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [112,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [113,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [114,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [115,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [116,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [117,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [118,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [119,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [120,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [121,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [122,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [123,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [124,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [1,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [32,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [33,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [34,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [35,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [36,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [37,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [38,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [39,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [40,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [41,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [42,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [43,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [44,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [45,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [46,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [47,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [48,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [49,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [50,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [51,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [52,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [53,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [54,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [55,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [56,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [57,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [58,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [59,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [60,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [64,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [65,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [66,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [67,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [68,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [69,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [70,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [71,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [72,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [73,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [74,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [75,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [76,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [77,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [78,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [79,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [80,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [81,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [82,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [83,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [84,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [85,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [86,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [87,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [88,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [89,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [90,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [91,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [92,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [93,0,0] Assertion srcIndex < srcSelectDimSize failed.
C:/w/1/s/tmp_conda_3.6_041836/conda/conda-bld/pytorch_1556684464974/work/aten/src/THC/THCTensorIndex.cu:308: block: [0,0,0], thread: [94,0,0] Assertion srcIndex < srcSelectDimSize failed.

RuntimeError: CUDA error: device-side assert triggered

gauravkoradiya on 11 Jun 2019

@search4mahesh and @gauravkoradiya could you please have a look at #387?

You could try to filter out long sentences with: https://github.com/zalandoresearch/flair/issues/387#issuecomment-477049028 :)

stefan-it on 17 Jun 2019

👍1

I see a problem in https://github.com/flairNLP/flair/blob/master/flair/embeddings.py#L2678--L2687

        # first, find longest sentence in batch
        longest_sentence_in_batch: int = len(
            max(
                [
                    self.tokenizer.tokenize(sentence.to_tokenized_string())
                    for sentence in sentences
                ],
                key=len,
            )
        )

This is passed to

        # prepare id maps for BERT model
        features = self._convert_sentences_to_features(
            sentences, longest_sentence_in_batch
        )

which sets max_sequence_length in:

https://github.com/flairNLP/flair/blob/master/flair/embeddings.py#L2620-L2622
```python
_convert_sentences_to_features(
self, sentences, max_sequence_length: int
)
````

But this does not account for or check the max-sequence-length supported by the BERT model, which is accessible in either of the above functions through self.model.config.max_position_embeddings.