Text: importing torchtext: module 'tokenize' has no attribute 'open'

Created on 9 Jul 2018 · 11Comments · Source: pytorch/text

When trying to import any module within torchtext (either interactively or in a script), I get the following stack trace:

>>> import torchtext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torchtext/__init__.py", line 1, in <module>
    from . import data
  File "/usr/local/lib/python3.6/dist-packages/torchtext/data/__init__.py", line 1, in <module>
    from .batch import Batch
  File "/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py", line 1, in <module>
    import torch
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 244, in <module>
    _C._initExtension(manager_path())
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 126, in <module>
    _lazy_call(_check_capability)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 124, in _lazy_call
    _queued_calls.append((callable, traceback.format_stack()))
  File "/usr/lib/python3.6/traceback.py", line 193, in format_stack
    return format_list(extract_stack(f, limit=limit))
  File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/usr/lib/python3.6/traceback.py", line 358, in extract
    f.line
  File "/usr/lib/python3.6/traceback.py", line 282, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/lib/python3.6/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib/python3.6/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib/python3.6/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
AttributeError: module 'tokenize' has no attribute 'open'

This happens with the following relevant software:

Ubuntu 18.04
Python 3.6.5
torch 0.4.0
torchtext 0.2.3

Source

JesseTG

Most helpful comment

...ah, I think I've figured it out. It turns out I was in a directory with a Python module in my project called "tokenize", and that module was being preferred over another one. I renamed it, and now this works fine.

JesseTG on 9 Jul 2018

👍6 ❤1

All 11 comments

Maybe you can try newer version of the torchtext. The torchtext GitHub master is version 3.0 and is compatible with torch 0.4.

AveryLiu on 9 Jul 2018

No, still getting this same error. In fact, it's not just torchtext. If I import torch, I get this:

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 244, in <module>
    _C._initExtension(manager_path())
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 126, in <module>
    _lazy_call(_check_capability)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 124, in _lazy_call
    _queued_calls.append((callable, traceback.format_stack()))
  File "/usr/lib/python3.6/traceback.py", line 193, in format_stack
    return format_list(extract_stack(f, limit=limit))
  File "/usr/lib/python3.6/traceback.py", line 207, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/usr/lib/python3.6/traceback.py", line 358, in extract
    f.line
  File "/usr/lib/python3.6/traceback.py", line 282, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/lib/python3.6/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/usr/lib/python3.6/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/usr/lib/python3.6/linecache.py", line 136, in updatecache
    with tokenize.open(fullname) as fp:
AttributeError: module 'tokenize' has no attribute 'open'
>>>

And then, if I try again immediately thereafter:

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/__init__.py", line 80, in <module>
    __all__ += [name for name in dir(_C)
NameError: name '_C' is not defined
>>>

EDIT: Now it sometimes works, sometimes not. I don't know what exactly "sometimes" means yet.

JesseTG on 9 Jul 2018

👍6 ❤1

...ah, I think I've figured it out. It turns out I was in a directory with a Python module in my project called "tokenize", and that module was being preferred over another one. I renamed it, and now this works fine.

Thanks! i can't believe I had the exact same bug as yours.

violetguos on 9 Jan 2020

@violetguos Could you send a code snippet to reproduce the error? Thx.

zhangguanheng66 on 9 Jan 2020

I simply renamed my tokenize.py to tokenize_script.py. If you want to reproduce the error, just create a tokenize.py file.

violetguos on 9 Jan 2020

This is part of a much bigger project running on torch 0.4.1. Sorry I cannot share my source code.

violetguos on 9 Jan 2020

Fine. You mean you got the error while importing torchtext? That's pretty strange. Do you use the binary (pip/conda) or the source code from master branch.

zhangguanheng66 on 9 Jan 2020

haha, exactly the same bug here

greboide on 5 Oct 2020

haha, exactly the same bug here

Can you share a code snippet to reproduce the error?

zhangguanheng66 on 5 Oct 2020

I encountered this problem too. tokenize.py is a file of the same name of python library file. When you debug, python debugger mistask your file as a standard library file.Therefore you can change your file name from tokenize.py to tokenize_.py. After that everything goes ok.