Opennmt-py: Upgrade to torchtext 0.3

Created on 18 Jun 2018  路  27Comments  路  Source: OpenNMT/OpenNMT-py

pytorch: 0.4
torchtext: 0.3

Hello, I've got one error when doing the first preprocessing step from here.

Traceback (most recent call last):
  File "preprocess.py", line 204, in <module>
    main()
  File "preprocess.py", line 191, in main
    fields = onmt.io.get_fields(opt.data_type, src_nfeats, tgt_nfeats)
  File "/home/lr/yukun/OpenNMT-py/onmt/io/IO.py", line 44, in get_fields
    return TextDataset.get_fields(n_src_features, n_tgt_features)
  File "/home/lr/yukun/OpenNMT-py/onmt/io/TextDataset.py", line 229, in get_fields
    postprocessing=make_src, sequential=False)
TypeError: __init__() got an unexpected keyword argument 'tensor_type'

It seems that torchtext 0.3 has changed their interface parameter in torchtext.data.Field()

But in torchtext 0.2.3, it works well except following warning:

xxx/torchtext/data/field.py:321: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.

And it seems new version of torchtext has fixed this from here. I am new to pytorch and opennmt. I don't know whether this warning matter?

feature
Source
picture yukunfeng

Most helpful comment

Extracting features...

  • number of source features: 0.
  • number of target features: 0.
    Building Fields object...
    Traceback (most recent call last):
    File "preprocess.py", line 193, in
    main()
    File "preprocess.py", line 180, in main
    fields = onmt.io.get_fields(opt.data_type, src_nfeats, tgt_nfeats)
    File "/content/MultimodalNMT/onmt/io/IO.py", line 43, in get_fields
    return TextDataset.get_fields(n_src_features, n_tgt_features)
    File "/content/MultimodalNMT/onmt/io/TextDataset.py", line 218, in get_fields
    postprocessing=make_src, sequential=False)
    TypeError: init() got an unexpected keyword argument 'tensor_type'

same error, have you solved it?

Fixed!

pip uninstall torchtext
pip install torchtext==0.2.3
picture Eurus-Holmes on 12 Dec 2019
👍2

All 27 comments

For now we recommned torchtext 0.2.1 . We will try to support the new version.

srush picture srush on 18 Jun 2018

Hi guys,
I went into this issue too. There is not much to adjust but there are a few breaking changes that would require a pip install git+git://github.com/pytorch/text installation.
I will add these changes in the #762 since I did it already.
But in the end, yes for pytorch 0.4 full compatibilty, it requires torchtext 0.3

vince62s picture vince62s on 18 Jun 2018

Oh, Thanks both!

yukunfeng picture yukunfeng on 18 Jun 2018

So the solution here is to use torchtext 0.2.1?

ciphurus picture ciphurus on 25 Oct 2018

no you need >=0.3.0

vince62s picture vince62s on 25 Oct 2018

I get this error with torchtext = 0.3.1 and torch = 1.0.0.dev20181014. Am I doing something wrong?

Actually, I am sorry. I get this error with
LABEL = Field(sequential=False, use_vocab=False,tensor_type=torch.FloatTensor)
Different than the issue here. I am guessing tensor_type is not an argument that is accepted any more. What's the best way to change the code correctly? Thanks for the help and I know this is probably not related to the issue here

ciphurus picture ciphurus on 25 Oct 2018

yes it's dtype now but don't do this, open a new issue when you face this, otherwise it's a waste of time ofr you and us. thanks.

vince62s picture vince62s on 25 Oct 2018

@ciphurus Just use this format of dtype=torch.long or dtype=torch.float.

text = data.Field(dtype=torch.long)
label = data.LabelField(dtype=torch.float)
ritchieng picture ritchieng on 30 Oct 2018
👍1

I got an error in preprocessing step. plz help.
Extracting features...

  • number of source features: 0.
  • number of target features: 0.
    Building Fields object...
    Traceback (most recent call last):
    File "preprocess.py", line 193, in
    main()
    File "preprocess.py", line 180, in main
    fields = onmt.io.get_fields(opt.data_type, src_nfeats, tgt_nfeats)
    File "/content/OpenNMT-py/onmt/io/IO.py", line 43, in get_fields
    return TextDataset.get_fields(n_src_features, n_tgt_features)
    File "/content/OpenNMT-py/onmt/io/TextDataset.py", line 231, in get_fields
    postprocessing=make_src, sequential=False)
    TypeError: __init__() got an unexpected keyword argument 'tensor_type'
sahinurlaskar picture sahinurlaskar on 21 Jun 2019

/content/OpenNMT-py/onmt/io
You are on a very very old version of onmt-py

vince62s picture vince62s on 21 Jun 2019

ok now m facing this issue while preprocessing in my local machine.
_pickle.PicklingError: Can't pickle at 0x7f0d1f050e18>: attribute lookup Field. on torchtext.data.field failed

sahinurlaskar picture sahinurlaskar on 21 Jun 2019

solved my above issues by installing all requirements packages.

sahinurlaskar picture sahinurlaskar on 21 Jun 2019

Thanks for advice

sahinurlaskar picture sahinurlaskar on 21 Jun 2019

how to apply subword-nmt on tokenized file as mentioned in preprocessing step
https://github.com/iacercalixto/MultimodalNMT

sahinurlaskar picture sahinurlaskar on 21 Jun 2019

how can i fix this error?
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/torch/csrc/cuda/Module.cpp:33

sahinurlaskar picture sahinurlaskar on 22 Jun 2019

Extracting features...

  • number of source features: 0.
  • number of target features: 0.
    Building Fields object...
    Traceback (most recent call last):
    File "preprocess.py", line 193, in
    main()
    File "preprocess.py", line 180, in main
    fields = onmt.io.get_fields(opt.data_type, src_nfeats, tgt_nfeats)
    File "/content/MultimodalNMT/onmt/io/IO.py", line 43, in get_fields
    return TextDataset.get_fields(n_src_features, n_tgt_features)
    File "/content/MultimodalNMT/onmt/io/TextDataset.py", line 218, in get_fields
    postprocessing=make_src, sequential=False)
    TypeError: __init__() got an unexpected keyword argument 'tensor_type'
sahinurlaskar picture sahinurlaskar on 23 Jun 2019
👍1

Using cached https://files.pythonhosted.org/packages/ee/67/f403d4ae6e9cd74b546ee88cccdb29b8415a9c1b3d80aebeb20c9ea91d96/pytorch-1.0.2.tar.gz
Building wheels for collected packages: pytorch
Building wheel for pytorch (setup.py) ... error
ERROR: Failed building wheel for pytorch
Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
Running setup.py install for pytorch ... error
ERROR: Command "/usr/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-kuybhq1y/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-seag18hw/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-kuybhq1y/pytorch/

sahinurlaskar picture sahinurlaskar on 23 Jun 2019

Traceback (most recent call last):
File "preprocess.py", line 11, in
import onmt.io
File "/content/MultimodalNMT/onmt/__init__.py", line 1, in
import onmt.io
File "/content/MultimodalNMT/onmt/io/__init__.py", line 1, in
from onmt.io.IO import collect_feature_vocabs, make_features, \
File "/content/MultimodalNMT/onmt/io/IO.py", line 7, in
import torchtext.data
File "/usr/local/lib/python3.6/dist-packages/torchtext/__init__.py", line 1, in
from . import data
File "/usr/local/lib/python3.6/dist-packages/torchtext/data/__init__.py", line 4, in
from .field import RawField, Field, ReversibleField, SubwordField, NestedField, LabelField
File "/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py", line 61, in
class Field(RawField):
File "/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py", line 118, in Field
torch.float32: float,
AttributeError: module 'torch' has no attribute 'float32'

sahinurlaskar picture sahinurlaskar on 23 Jun 2019

Extracting features...

  • number of source features: 0.
  • number of target features: 0.
    Building Fields object...
    Building & saving training data...
  • saving train data shard to data/m30k.train.1.pt.
    Building & saving vocabulary...
  • reloading data/m30k.train.1.pt.
  • tgt vocab size: 4.
  • src vocab size: 2.
    Building & saving validation data...
  • saving valid data shard to data/m30k.valid.1.pt.
sahinurlaskar picture sahinurlaskar on 23 Jun 2019

Loading train dataset from data/m30k.train.1.pt, number of examples: 0
data_type: text
Traceback (most recent call last):
File "train_mm.py", line 448, in
main()
File "train_mm.py", line 428, in main
fields = load_fields(first_dataset, data_type, checkpoint)
File "train_mm.py", line 329, in load_fields
fields = dict([(k, f) for (k, f) in fields.items()
File "train_mm.py", line 330, in
if k in dataset.examples[0].__dict__])
IndexError: list index out of range

sahinurlaskar picture sahinurlaskar on 23 Jun 2019

Can any one help after after preprocessing step while applying training command in multimodal NMT step i found this error:
In processing step, after tokenization i applied BPE (https://nlp.h-its.org/bpemb/#download) then apply presprocssing command as given in Step1.

Loading train dataset from data.train.1.pt, number of examples: 0
data_type: text
Traceback (most recent call last):
File "train_mm.py", line 448, in
main()
File "train_mm.py", line 428, in main
fields = load_fields(first_dataset, data_type, checkpoint)
File "train_mm.py", line 329, in load_fields
fields = dict([(k, f) for (k, f) in fields.items()
File "train_mm.py", line 330, in
if k in dataset.examples[0].__dict__])
IndexError: list index out of range

sahinurlaskar picture sahinurlaskar on 25 Jun 2019

Traceback (most recent call last):
File "extract_image_features.py", line 11, in
from onmt.PretrainedCNNModels import PretrainedCNN
File "/content/MultimodalNMT/onmt/PretrainedCNNModels.py", line 8, in
import pretrainedmodels
File "/usr/local/lib/python3.6/dist-packages/pretrainedmodels/__init__.py", line 3, in
from . import models
File "/usr/local/lib/python3.6/dist-packages/pretrainedmodels/models/__init__.py", line 19, in
from .torchvision_models import alexnet
File "/usr/local/lib/python3.6/dist-packages/pretrainedmodels/models/torchvision_models.py", line 3, in
import torchvision.models as models
File "/usr/local/lib/python3.6/dist-packages/torchvision/__init__.py", line 1, in
from torchvision import models
File "/usr/local/lib/python3.6/dist-packages/torchvision/models/__init__.py", line 11, in
from . import detection
File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/__init__.py", line 1, in
from .faster_rcnn import *
File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/faster_rcnn.py", line 7, in
from torchvision.ops import misc as misc_nn_ops
File "/usr/local/lib/python3.6/dist-packages/torchvision/ops/__init__.py", line 1, in
from .boxes import nms, box_iou
File "/usr/local/lib/python3.6/dist-packages/torchvision/ops/boxes.py", line 2, in
from torchvision import _C
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

sahinurlaskar picture sahinurlaskar on 25 Jun 2019

How i solve above issue
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

sahinurlaskar picture sahinurlaskar on 25 Jun 2019

IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number
(while training process this error come)

sahinurlaskar picture sahinurlaskar on 26 Jun 2019

IndexError: index 28931 is out of bounds for axis 0 with size 28929

sahinurlaskar picture sahinurlaskar on 26 Jun 2019

Extracting features...

  • number of source features: 0.
  • number of target features: 0.
    Building Fields object...
    Traceback (most recent call last):
    File "preprocess.py", line 193, in
    main()
    File "preprocess.py", line 180, in main
    fields = onmt.io.get_fields(opt.data_type, src_nfeats, tgt_nfeats)
    File "/content/MultimodalNMT/onmt/io/IO.py", line 43, in get_fields
    return TextDataset.get_fields(n_src_features, n_tgt_features)
    File "/content/MultimodalNMT/onmt/io/TextDataset.py", line 218, in get_fields
    postprocessing=make_src, sequential=False)
    TypeError: init() got an unexpected keyword argument 'tensor_type'

same error, have you solved it?

Eurus-Holmes picture Eurus-Holmes on 12 Dec 2019

Extracting features...

  • number of source features: 0.
  • number of target features: 0.
    Building Fields object...
    Traceback (most recent call last):
    File "preprocess.py", line 193, in
    main()
    File "preprocess.py", line 180, in main
    fields = onmt.io.get_fields(opt.data_type, src_nfeats, tgt_nfeats)
    File "/content/MultimodalNMT/onmt/io/IO.py", line 43, in get_fields
    return TextDataset.get_fields(n_src_features, n_tgt_features)
    File "/content/MultimodalNMT/onmt/io/TextDataset.py", line 218, in get_fields
    postprocessing=make_src, sequential=False)
    TypeError: init() got an unexpected keyword argument 'tensor_type'

same error, have you solved it?

Fixed!

pip uninstall torchtext
pip install torchtext==0.2.3
Eurus-Holmes picture Eurus-Holmes on 12 Dec 2019
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lizamiller picture lizamiller  路  3Comments
howardyclo picture howardyclo  路  6Comments
nikhilweee picture nikhilweee  路  3Comments
liangbright picture liangbright  路  4Comments
mrpega picture mrpega  路  3Comments