Jedi: 0.17.1 seems to be completely broken

[HarryR]

ptpython derps out when trying to auto-complete (e.g. whenever you type a key) with the same error message as below, this essentially breaks jedi completely by one of the main cool projects which uses it. This only happens with Jedi 0.17.1

This does not happen with Jedi 0.17.0

With 0.17.0, ptpython works fine with Python 3.6.9, upgrading to 0.17.1 breaks ptpython and most other things which use jedi by import jedi.

The ptpython exception origin is:

Unhandled exception in event loop:
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/ptpython/python_input.py", line 828, in run
    document, self.get_locals(), self.get_globals()
  File "/home/user/.local/lib/python3.6/site-packages/ptpython/utils.py", line 46, in get_jedi_script_from_document
    import jedi  # We keep this import in-line, to improve start-up time.

The exception is caused by something in Jedi... even though traverse_parents exists in the jedi/common.py file... maybe some recursive import loop?

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jedi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.6/site-packages/jedi/__init__.py", line 32, in <module>
    from jedi.api import Script, Interpreter, set_debug_function, \
  File "/home/user/.local/lib/python3.6/site-packages/jedi/api/__init__.py", line 24, in <module>
    from jedi.api import classes
  File "/home/user/.local/lib/python3.6/site-packages/jedi/api/classes.py", line 24, in <module>
    from jedi.inference.utils import unite
  File "/home/user/.local/lib/python3.6/site-packages/jedi/inference/__init__.py", line 70, in <module>
    from jedi.inference import imports
  File "/home/user/.local/lib/python3.6/site-packages/jedi/inference/imports.py", line 24, in <module>
    from jedi.inference import sys_path
  File "/home/user/.local/lib/python3.6/site-packages/jedi/inference/sys_path.py", line 8, in <module>
    from jedi.common import traverse_parents
ImportError: cannot import name 'traverse_parents'
>>> 

---

Example proof that reverting back to Jedi 0.17.0 fixes it:

$ pip3 install --user jedi==0.17.0
Collecting jedi==0.17.0
  Using cached https://files.pythonhosted.org/packages/f3/3f/67f027e18c60a800875df1a0894a2436ce9053637fa39725766e937c0a71/jedi-0.17.0-py2.py3-none-any.whl
Collecting parso>=0.7.0 (from jedi==0.17.0)
  Using cached https://files.pythonhosted.org/packages/b5/61/998cce9e7476de000d031874df26a18f67cb73448164fc44a98f0c55920b/parso-0.7.0-py2.py3-none-any.whl
Installing collected packages: parso, jedi
Successfully installed jedi-0.17.1 parso-0.7.0

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jedi
>>> 

$ pip3 install --user jedi==0.17.1
Collecting jedi==0.17.1
  Using cached https://files.pythonhosted.org/packages/07/83/7e711550fcb2722f1ca9c8564d5bb23f625ae67d99f4a360b428c0f3e932/jedi-0.17.1-py2.py3-none-any.whl
Collecting parso<0.8.0,>=0.7.0 (from jedi==0.17.1)
  Using cached https://files.pythonhosted.org/packages/b5/61/998cce9e7476de000d031874df26a18f67cb73448164fc44a98f0c55920b/parso-0.7.0-py2.py3-none-any.whl
Installing collected packages: parso, jedi
Successfully installed jedi-0.17.1 parso-0.7.0

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jedi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.6/site-packages/jedi/__init__.py", line 32, in <module>
    from jedi.api import Script, Interpreter, set_debug_function, \
  File "/home/user/.local/lib/python3.6/site-packages/jedi/api/__init__.py", line 24, in <module>
    from jedi.api import classes
  File "/home/user/.local/lib/python3.6/site-packages/jedi/api/classes.py", line 24, in <module>
    from jedi.inference.utils import unite
  File "/home/user/.local/lib/python3.6/site-packages/jedi/inference/__init__.py", line 70, in <module>
    from jedi.inference import imports
  File "/home/user/.local/lib/python3.6/site-packages/jedi/inference/imports.py", line 24, in <module>
    from jedi.inference import sys_path
  File "/home/user/.local/lib/python3.6/site-packages/jedi/inference/sys_path.py", line 8, in <module>
    from jedi.common import traverse_parents
ImportError: cannot import name 'traverse_parents'
>>> 

[davidhalter]

I think there's something wrong with your installation. This is probably a bug in some packaging solution. traverse_parents used to be in common/__init__.py. Now it's in common.py. There's probably still a folder jedi.common with files in there that shouldn't be there.

In my build folder, I don't see a traverse_parents.

[davidhalter]

I'm closing, because the tests are passing on 0.17.1 (in the CI) and I really think that you have a broken installation of Jedi. I can also not reproduce this locally.

BTW: If Jedi 0.17.1 was completely broken, I would probably have heard about it in first few hours after release. There are >100k downloads a day.

[PeterJCLaw]

I hit something like this in development, though I was doing some weird permissions things at the time such that pip wasn't able to remove the directory. Agree this sounds like a mangled install issue.

[dcervenkov]

Same thing here @PeterJCLaw. Fixed it by uninstalling Jedi (using pip) and manually removing the following directories, that were not removed by pip.

~/.config/lib/python3.6/site-packages/jedi
~/.config/lib/python3.6/site-packages/jedi-0.16.0.dist-info

Then reinstalling via pip.

Curiously, Jedi was installed by coc-python. I wonder if they mess up the permissions somehow.

[danieldjewell]

Just to add: I had the exact same issue and it's caused by a stray common directory sitting in the jedi/ directory. However, this is after a pip install -I jedi ... and after a pip install --force-reinstall jedi

Interestingly, doing a pip uninstall jedi results in:

 Found existing installation: jedi 0.17.2
Uninstalling jedi-0.17.2:
  Would remove:
    /usr/local/lib/python3.8/dist-packages/jedi-0.17.2.dist-info/*
    /usr/local/lib/python3.8/dist-packages/jedi/*
  Would not remove (might be manually added):
    /usr/local/lib/python3.8/dist-packages/jedi/common/__init__.py
    /usr/local/lib/python3.8/dist-packages/jedi/common/utils.py
    /usr/local/lib/python3.8/dist-packages/jedi/common/value.py

Somewhere along the line, pip didn't remove the other files (presumably during an upgrade) and now since they're not part of 0.17.2, won't remove them. The solution is clear now, but maybe a limitation of pip to consider in the future? (i.e. Pip isn't 100% foolproof when trying to fix installs with -I or --force-reinstall in cases where files were deleted from a package....)

[davidhalter]

Good point @danieldjewell. If this is happening with latest pip, I think it's probably a pip bug and should be filed there. You can maybe reproduce it by installing an older version (might need to use it once to create the pyc files).

[HarryR]

Good point @danieldjewell. If this is happening with latest pip, I think it's probably a pip bug and should be filed there. You can maybe reproduce it by installing an older version (might need to use it once to create the pyc files).

I absolutely agree that you should lodge an upstream issue with pip, however it's unrealistic to expect everybody in the world to fix this manually and/or wait for pip to fix something that just happens to break your package just so they can upgrade to the latest version of pip.

Is there any compromise, like maybe adding a dummy file for 'backwards compatibility'?

[davidhalter]

I doubt that this is happening for everyone. Jedi has ~6m downloads a month. Since only three people have complained so far (and there are no likes on this issue), I would think that this happens only under certain conditions.

I'm also not going to create an issue upstream, I have never personally seen this issue.

[HarryR]

Fair enough, it's a weird edge case where pip shits the bed because some package was re-organised and directory moved and now import doesn't work because of Python import logic. Well anyway, 99.9% of those are likely CI builds, that you can filter out.

Either way, problem is solved here, comments above show how to solve problem etc.

[davidhalter]

99.9% is still 6k a month. ;)

My problem with these kind of issues is that I would rather spend my time playing beach volleyball, my guitar, spending time with my wife or actually coding something interesting.

Chasing a weird bug in pip in my unpaid spare time is really not one of the things I would love to do :).

[HarryR]

Respect.

Sorry for taking your time (even though, even by this message, i'm still consuming it... fuck)

[davidhalter]

No worries, I like working on this issue tracker. It's just that I don't like chasing bugs in other software that I have no idea about :)

[dougli1sqrd]

Same thing here @PeterJCLaw. Fixed it by uninstalling Jedi (using pip) and manually removing the following directories, that were not removed by pip.

~/.config/lib/python3.6/site-packages/jedi
~/.config/lib/python3.6/site-packages/jedi-0.16.0.dist-info

Then reinstalling via pip.

Curiously, Jedi was installed by coc-python. I wonder if they mess up the permissions somehow.

I just had this problem that is described above as well. I'm on linux mint/ubuntu. python 3.6.9, pip3 9.0.1. This showed up when trying to install the python-language-server and it updated from my old install of jedi. The above comment finally worked.