Text: field larger than field limit (131072)

[lipingbj]

When I use the data.TabularDataset ,there occurs an problem(field larger than field limit (131072)).

train_data = data.TabularDataset(path='train.txt.labeled2', format='tsv',
fields=[('sentence', TEXT),('label', LABEL)], csv_reader_params={"lineterminator": "\n"})

The lines of the train.txt.labeled2 is larger than 1000,0000. I don't know how to set the attribute of the csv.field_size_limit. And i found there is no place to set this parameter.

[zhangguanheng66]

@lipingbj can you upload your dataset so I can reproduce the error?

[lipingbj]

@lipingbj can you upload your dataset so I can reproduce the error?

@zhangguanheng66 I have uploaded the dataset. The address is (https://drive.google.com/file/d/11-dsF5wuaSB7FPyo56MJDw8Z9-zm9WGF/view?usp=sharing).

[zhangguanheng66]

@lipingbj can you also copy/paste the objects of TEXT and LABEL so I could reproduce the error.

[anonymone]

In windows, When I load a large dataset, the Python int too large to convert to C long error happened.
To solve the error, I changed https://github.com/pytorch/text/blob/9a20c79c13657543adc28ca8e8355daaf591eaf6/torchtext/utils.py#L129 into csv.field_size_limit(int(ctypes.c_ulong(-1).value // 2)) .

[zhangguanheng66]

Do you still see the error after checking out the master branch? @rouzbeh-afrasiabi

[zhangguanheng66]

Feel free to re-open the issue if you still have questions.