jedi-vim is not aware of pyenv path

[frank777777777]

Issue

I am using pyenv to switch between difference python version. The autocomplete and goto function only uses modules from system python. For example, while I am in a pyenv virtual environment, I installed beautifulsoup module, jedi-vim does not find my module in my virtual environment, it searches for module in system site-packages instead.
As you can see my verbose JediDebugInfo, my python2 environment is linking to pyenv, while python3 environment is linking to system python.
In vim, !python --version and !pip --version works, its just jedi seems not finding my pyenv.

Steps to reproduce

Output of “:verbose JediDebugInfo”

##### jedi-vim version

 - jedi-vim git version: 0.9.0-57-g132d850-dirty
 - jedi git submodule status:  a79a1fbef57564d031c59f47def20a6d9adf3439 pythonx/jedi (v0.12.1)
 - parso git submodule status:  e6bc924fbabbfa3090044e475c91a7655b9c6e9b pythonx/parso (v0.3.1)

##### Global Python

Using Python version 3 to access Jedi.
 - global sys.version: `3.5.2 (default, Nov 23 2017, 16:37:01), [GCC 5.4.0 20160609]`
 - global site module: `/usr/lib/python3.5/site.py`

##### Jedi

 - path: `/home/yili/.vim/bundle/jedi-vim/pythonx/jedi/jedi/__init__.py`
 - version: 0.12.1

##### Jedi environment: <Environment: 3.6.5 in /usr>

 - executable: /usr/bin/python3.6
 - sys_path:
    - `/usr/lib/python36.zip`
    - `/usr/lib/python3.6`
    - `/usr/lib/python3.6/lib-dynload`
    - `/usr/local/lib/python3.6/dist-packages`
    - `/usr/lib/python3/dist-packages`

##### Known environments

 - <Environment: 3.6.5 in /usr> (/usr/bin/python3.6)
 - <Environment: 3.5.2 in /usr> (/usr/bin/python3.5)
 - <Environment: 2.7.15 in /home/yili/.pyenv> (/home/yili/.pyenv/shims/python2.7)

##### Settings

g:jedi#documentation_command = 'J' (default: 'K')
g:jedi#usages_command = 'n' (default: 'n')
g:jedi#completions_enabled = 0 (default: 1)

omnifunc=python3complete#Complete
Last set from /usr/share/vim/vim81/ftplugin/python.vim
completeopt=preview,menuone
Last set from ~/.vim/bundle/YouCompleteMe/autoload/youcompleteme.vim

#### :version

VIM - Vi IMproved 8.1 (2018 May 18, compiled Jul 6 2018 08:05:07)
Included patches: 1-155
Modified by [email protected]
Compiled by [email protected]
Huge version without GUI. Features included (+) or not (-):
+acl +cursorshape +job -mouse_sysmouse +scrollbind +vertsplit
+arabic +dialog_con +jumplist +mouse_urxvt +signs +virtualedit
+autocmd +diff +keymap +mouse_xterm +smartindent +visual
-autoservername +digraphs +lambda +multi_byte +startuptime +visualextra
-balloon_eval -dnd +langmap +multi_lang +statusline +viminfo
+balloon_eval_term -ebcdic +libcall -mzscheme -sun_workshop +vreplace
-browse +emacs_tags +linebreak +netbeans_intg +syntax +wildignore
++builtin_terms +eval +lispindent +num64 +tag_binary +wildmenu
+byte_offset +ex_extra +listcmds +packages +tag_old_static +windows
+channel +extra_search +localmap +path_extra -tag_any_white +writebackup
+cindent +farsi -lua -perl -tcl -X11
-clientserver +file_in_path +menu +persistent_undo +termguicolors -xfontset
-clipboard +find_in_path +mksession +postscript +terminal -xim
+cmdline_compl +float +modify_fname +printer +terminfo -xpm
+cmdline_hist +folding +mouse +profile +termresponse -xsmp
+cmdline_info -footer -mouseshape -python +textobjects -xterm_clipboard
+comments +fork() +mouse_dec +python3 +timers -xterm_save
+conceal +gettext +mouse_gpm +quickfix +title
+cryptv -hangul_input -mouse_jsbterm +reltime -toolbar
+cscope +iconv +mouse_netterm +rightleft +user_commands
+cursorbind +insert_expand +mouse_sgr -ruby +vartabs
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
defaults file: "$VIMRUNTIME/defaults.vim"
fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -Wdate-time -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=f
ormat-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim -lm -ltinfo -l
nsl -lselinux -lacl -lattr -lgpm -ldl -L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -lpython3.5m -lpthread -l
dl -lutil -lm

[blueyed]

Try let g:jedi#force_py_version = '2.7.15'.

Also see https://github.com/davidhalter/jedi-vim/pull/836.

[frank777777777]

Try let g:jedi#force_py_version = '2.7.15'.

I am trying to use python 3 not python2...
I just realized that, if I install python3.6 in pyenv, jedi will refer the python executable to pyenv. If I use python 3.7 in pyenv, jedi will ignore pyenv and use system python executable instead. It seems to me that jedi has hardcoded that it only look for python 3.6 and 3.5, because python 3.5 executable still link to system python. Here is the verbose JediDebugInfo after I install python3.6 in pyenv:

##### Jedi environment: <Environment: 3.6.0 in /home/yili/.pyenv>

 - executable: /home/yili/.pyenv/shims/python3.6
 - sys_path:
    - `/home/yili/.pyenv/versions/3.6.0/lib/python36.zip`
    - `/home/yili/.pyenv/versions/3.6.0/lib/python3.6`
    - `/home/yili/.pyenv/versions/3.6.0/lib/python3.6/lib-dynload`
    - `/home/yili/.pyenv/versions/3.6.0/lib/python3.6/site-packages`

##### Known environments

 - <Environment: 3.6.0 in /home/yili/.pyenv> (/home/yili/.pyenv/shims/python3.6)
 - <Environment: 3.5.2 in /usr> (/usr/bin/python3.5)
 - <Environment: 2.7.15 in /home/yili/.pyenv> (/home/yili/.pyenv/shims/python2.7)

The executable refers to pyenv python, works fine. The sys_path refers to pyenv site-package, works fine. Omni-completion and goto works perfect for modules install in pyenv site package.

However, when I create a virtualenv in pyenv like pyenv virtualenv 3.6.0 scrapper, pyenv local scrapper, the sys_path refer back to system site package. Here is the :verbose JediDebugInfo:

##### Jedi environment: <Environment: 3.6.5 in /home/yili/.pyenv>

 - executable: /home/yili/.pyenv/shims/python3.6
 - sys_path:
    - `/usr/lib/python36.zip`
    - `/usr/lib/python3.6`
    - `/usr/lib/python3.6/lib-dynload`
    - `/usr/local/lib/python3.6/dist-packages`
    - `/usr/lib/python3/dist-packages`

##### Known environments

 - <Environment: 3.6.5 in /home/yili/.pyenv> (/home/yili/.pyenv/shims/python3.6)
 - <Environment: 3.5.2 in /usr> (/usr/bin/python3.5)
 - <Environment: 2.7.15 in /home/yili/.pyenv> (/home/yili/.pyenv/shims/python2.7)

The executable stills uses pyenv, but the sys_path link back to system site-package, omni-completion and goto does not work anymore.

I am running out ideas.

[blueyed]

Thanks for the input.

Please consider trying #836 and see if that makes it better already - otherwise check the source to see what actually happens.
Currently venv support is very basic, and I do not remember the state of 836 really, but it should allow to switch and tab-complete them actually.

[blueyed]

Re "/home/daniel/.pyenv/shims/scrapper": it is using "Environment: 3.6.5 in /home/yili/.pyenv" there then.
IIRC jedi looks not for "python" first, but with versions appended.. at least it used to do.
Digging into the source (based on #836) is the best you can do currently I guess - if you want to help fix this.
Otherwise try to adjust g:jedi#force_py_version to make it pick up the version you want, but that is also only rudimentary on master.

[frank777777777]

Isn't g:jedi#force_py_version just to specify the python executable version only? I mean it is already picking up the correct python version in pyenv if I install 3.6.0.
It is the sys-path that is not incorrect. I saw in #836 you mentioned use let g:jedi#environment_paths = [$HOME . '/.pyenv/versions]', is it implemented yet?

[blueyed]

It does a bit more on master already:
https://github.com/davidhalter/jedi-vim/blob/132d8506ab40d1977310eb91bfb86c0d983e2812/pythonx/jedi_vim.py#L166-L185

you mentioned use let g:jedi#environment_paths = [$HOME . '/.pyenv/versions]', is it implemented yet?

I think so:

https://github.com/davidhalter/jedi-vim/blob/c06c8ef66ec264b343d85234870f2f9db8c521a5/pythonx/jedi_vim.py#L218-L223

[frank777777777]

wait, is this still a working progress and not committed to master yet? Because what I see in jedi-vim/pythonx/jedi_vim.py, it is still:

196 def get_known_environments():
197     """Get known Jedi environments."""
198     envs = list(jedi.api.environment.find_virtualenvs())
199     envs.extend(jedi.api.environment.find_system_environments())
200     return envs

[blueyed]

Yes, it is WIP - an open PR.
To get it working for you you could also hardcode something there, of course.

[blueyed]

https://github.com/davidhalter/jedi/pull/1199
https://github.com/davidhalter/jedi-vim/pull/872

[statham-stone]

change all "safe=True" to "safe=False" in "~/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py" solved the problem for me.

[cbows]

I am experiencing a similar issue.

I have system python version 3.6.8 installed and use pyenv to manage multiple python installations. When I create a virtual environment with python -m venv .venv jedi-vim is not able to provide completions. This only happens if I set the python version to anything other than the system version (e.g. via pyenv local 3.7.4). I am using ALE to provide completions.

:verbose JediDebugInfo

#### Jedi-vim debug information

##### jedi-vim version

 - jedi-vim git version: 0.10.0-3-g4f2499e
 - jedi git submodule status:  005f69390c4b1a3b864e7373cfcbdaf95e65411d pythonx/jedi (v0.15.1)
 - parso git submodule status:  c0ace63a6905500e18d8ac06af1a0a9057e63498 pythonx/parso (v0.5.1)

##### Global Python

Using Python version 3 to access Jedi.
 - global sys.version: `3.6.8 (default, Aug 20 2019, 17:12:48), [GCC 8.3.0]`
 - global site module: `/usr/lib/python3.6/site.py`

##### Jedi

 - path: `~/.local/share/nvim/plugged/jedi-vim/pythonx/jedi/jedi/__init__.py`
 - version: 0.15.1

##### Jedi environment: <Environment: 3.7.4 in /tmp/myapp/.venv>

 - executable: /tmp/myapp/.venv/bin/python
 - sys_path:
    - `~/.pyenv/versions/3.7.4/lib/python37.zip`
    - `~/.pyenv/versions/3.7.4/lib/python3.7`
    - `~/.pyenv/versions/3.7.4/lib/python3.7/lib-dynload`
    - `/tmp/myapp/.venv/lib/python3.7/site-packages`

##### Known environments

 - <Environment: 3.7.4 in /tmp/myapp/.venv> (/tmp/myapp/.venv/bin/python)
 - <Environment: 3.7.4 in ~/.pyenv/versions/3.7.4> (~/.pyenv/versions/3.7.4/bin/python3.7)
 - <Environment: 3.6.8 in /usr> (/usr/bin/python3.6)
 - <Environment: 2.7.15 in /usr> (/usr/bin/python2.7)

##### Settings

g:jedi#completions_enabled = 0 (default: 1)

  omnifunc=python3complete#Complete
        Zuletzt gesetzt in /tmp/.mount_nvimt3iaAF/usr/share/nvim/runtime/ftplugin/python.vim line 41
  completeopt=noinsert,menuone,noselect
        Zuletzt gesetzt in ~/dotfiles/nvim/.config/nvim/init.vim line 178

#### :version

NVIM v0.4.0-2014-g554566cce
Build type: RelWithDebInfo
LuaJIT 2.0.5
Features: +acl +iconv +tui

Although it recognizes the correct environment (which is in /tmp/myapp/.venv) and the correct executable (which is python3.7 in ~/.pyenv/versions) I can not autocomplete packages I install in the virtual environment. Weirdly I can only autocomplete packages installed with system python.

[blueyed]

@cbows
With Neovim check :checkhealth provider to see which Python it uses.

[cbows]

health#provider#check
========================================================================
## Clipboard (optional)
  - OK: Clipboard tool found: xclip

## Python 2 provider (optional)
  - INFO: pyenv: Path: ~/.pyenv/libexec/pyenv
  - INFO: pyenv: $PYENV_ROOT is not set. Infer from `pyenv root`.
  - INFO: pyenv: Root: ~/.pyenv
  - INFO: Using: g:python_host_prog = "/usr/bin/python2"
  - WARNING: $VIRTUAL_ENV exists but appears to be inactive. This could lead to unexpected results.
    - ADVICE:
      - If you are using Zsh, see: http://vi.stackexchange.com/a/7654
  - INFO: Executable: /usr/bin/python2
  - INFO: Python version: 2.7.15
  - INFO: pynvim version: 0.3.2

## Python 3 provider (optional)
  - INFO: pyenv: Path: ~/.pyenv/libexec/pyenv
  - INFO: pyenv: $PYENV_ROOT is not set. Infer from `pyenv root`.
  - INFO: pyenv: Root: ~/.pyenv
  - INFO: Using: g:python3_host_prog = "/usr/bin/python3"
  - WARNING: $VIRTUAL_ENV exists but appears to be inactive. This could lead to unexpected results.
    - ADVICE:
      - If you are using Zsh, see: http://vi.stackexchange.com/a/7654
  - INFO: Executable: /usr/bin/python3
  - INFO: Python version: 3.6.8
  - INFO: pynvim version: 0.3.2

I don't understand the warning WARNING: $VIRTUAL_ENV exists but appears to be inactive. This could lead to unexpected results.. I have activated the environment with source .venv/bin/activate as usual. I am using zsh but the advice didn't help.

If I unset g:python3_host_prog and let nvim figure out the executable, it finds the correct version from pyenv root, but it doesn't solve the problem (also I would have to install pynvim for every version I have installed).

[blueyed]

WARNING: $VIRTUAL_ENV exists but appears to be inactive. This could lead to unexpected results

This means it is not using the virtualenv's python (which might be fine of course).

As for jedi-vim: there's a host Python (the one Neovim uses), and it will use virtualenvs then.
Jedi/jedi-vim uses /tmp/myapp/.venv/bin/python - what completions are you missing from there?

[blueyed]

_maybe_ you are running into an issue where the host (3.6) cannot handle syntax for the venv (3.7)?

[cbows]

• I use a fresh Python executable (no site packages) installed with pyenv (not system python) to create a virtual environment (e.g. /tmp/myapp/.venv).

• I install packages in this environment.

• I open a fresh python file and type import ... (no special syntax) but my packages are not listed in the completions.

The only packages I do see, are the ones I have installed with my system python, which is a different version of python and was not used to create the virtual environment and should therefore be out of scope.

[cbows]

FYI: Completion works great in ipython with the same version of jedi and the exact same environment setup.

[blueyed]

@cbows
And JediDebugInfo says that it uses your env then?

From your initial output (where it looks like it), but please make sure that's still the case:

Jedi environment:

[blueyed]

@cbows
For what it's worth, this would have been better off in a new issue.

[cbows]

The JediDebugInfo of my initial post applies to the very scenario I just depicted.

I will file a new issue with a detailed description tomorrow.

[cbows]

@blueyed
I did some further investigation and the problem does not seem to be caused by jedi-vim.
With the following vimrc, completions work fine.

call plug#begin('~/.local/share/nvim/plugged')
Plug 'davidhalter/jedi-vim'
call plug#end()

set nocompatible

let script_dir = fnamemodify(expand('<sfile>'), ':h')
let &runtimepath .= ','.script_dir.','.script_dir.'/after'

syntax on
filetype plugin indent on

So the issue does not exist as far as jedi-vim is concerned.
I previously said

I am using ALE to provide completions.

This is of course nonsense. I mixed up names, sorry. What I actually meant was: I am using ncm2/ncm2-jedi to provide completions. And the problem only arises if I add this plugin to the vimrc. I filed an issue on their tracker and hope I will get some answer. If you have any further ideas, they are greatly appreciated. Otherwise, I thank you for your help so far.

[blueyed]

@cbows
Ok, cool. For reference: https://github.com/ncm2/ncm2-jedi/issues/23