Text: UnicodeError while creating TabularDataset

[rahulbhalley]

I am following the tutorial which can be found here.
The code is written as following:

>>> import torch
>>> from torchtext import data, datasets
>>> from torch.autograd import Variable
>>> import torch.nn as nn
>>> import torch.nn.functional as F
>>> import sys
>>> 
>>> 
>>> text_field = data.Field(lower=True, tokenize='spacy',tensor_type=torch.LongTensor)
>>> label_field = data.Field(sequential=False)
>>> 
>>> text_field.preprocessing = lambda x:x
>>> 
>>> pr_data = data.TabularDataset(path='polarity.tsv',format='tsv',fields=[('text',text_field),('label',label_field)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/rahulbhalley/miniconda2/lib/python2.7/site-packages/torchtext/data/dataset.py", line 235, in __init__
    examples = [make_example(line, fields) for line in reader]
  File "/Users/rahulbhalley/miniconda2/lib/python2.7/site-packages/torchtext/utils.py", line 60, in unicode_csv_reader
    for row in csv_reader:
  File "/Users/rahulbhalley/miniconda2/lib/python2.7/site-packages/torchtext/utils.py", line 69, in utf_8_encoder
    for line in unicode_csv_data:
  File "/Users/rahulbhalley/miniconda2/lib/python2.7/codecs.py", line 314, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd1 in position 8: invalid continuation byte

[nzw0301]

I guess that your torchtext is older version than master branch. My code based on your snippet works with the latest torchtext. You can solve this isseu by upgrading your pytorch==0.4.1 and torchtext.

import torch
from torchtext import data, datasets
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import sys

text_field = data.Field(lower=True, tokenize='spacy')
label_field = data.Field(sequential=False)

text_field.preprocessing = lambda x:x

pr_data = data.TabularDataset(path='polarity.tsv', format='tsv', fields=[('text', text_field), ('label', label_field)])

[rahulbhalley]

@nzw0301 thanks for replying. Can you please verify if torchtext's latest version is 0.2.3?

[nzw0301]

Yes, but I mean that you use the master branch's code because the latest stable version is a little bit old...

To install the master branch code, you run commands below:

git clone [email protected]:pytorch/text.git
cd text
python setup.py install

BTW, the code works on my environment with torchtext=0.2.3 (, MacOSX Mojave, anaconda2-latest, and pytorch=0.4.1).

[rahulbhalley]

Now I am on torchtext 0.3.0 version but still the same error. 😞 But note that I am using PyTorch 1.0 nightly-build.

[nzw0301]

PyTorch 1.0 nightly-build is also fine on my env... Sorry, I have no idea to solve your issue, but your error may occur by outside of this repo.

[mttk]

Could you try downloading polarity.tsv again and running again with the fresh download (without opening the downloaded file in any editor)?

[rahulbhalley]

Thanks @mttk. I guess the problem was that I opened raw version of polarity.tsv in GitHub and then saved it.
Now I just ran

git clone https://github.com/DaehanKim/torchtext_tutorial.git

and loaded polarity.tsv & it fa*king worked! Thanks a lot!