Jedi-vim: sys.executable might be Vim itself with embedded Python (UnpicklingError)

[tonitch]

(sorry for miss spellings, I'm french and I'll try my best to explain my issue)

Issue

When I try to or when jedi-vim want to help me, a new window of gvim pop-up with __main__.py oppened from jedi's file instead of the list of propositions and sometimes, when I go back to the previews window, gvim stop working...

Steps to reproduce

here is my _vimrc ::

source $VIMRUNTIME/vimrc_example.vim

let python_highlight_all=1
syntax on

"Enable folding + space to fold
set foldmethod=indent
set foldlevel=99
nnoremap <space> za

set diffexpr=MyDiff()
function MyDiff()
  let opt = '-a --binary '
  if &diffopt =~ 'icase' | let opt = opt . '-i ' | endif
  if &diffopt =~ 'iwhite' | let opt = opt . '-b ' | endif
  let arg1 = v:fname_in
  if arg1 =~ ' ' | let arg1 = '"' . arg1 . '"' | endif
  let arg1 = substitute(arg1, '!', '\!', 'g')
  let arg2 = v:fname_new
  if arg2 =~ ' ' | let arg2 = '"' . arg2 . '"' | endif
  let arg2 = substitute(arg2, '!', '\!', 'g')
  let arg3 = v:fname_out
  if arg3 =~ ' ' | let arg3 = '"' . arg3 . '"' | endif
  let arg3 = substitute(arg3, '!', '\!', 'g')
  if $VIMRUNTIME =~ ' '
    if &sh =~ '\<cmd'
      if empty(&shellxquote)
        let l:shxq_sav = ''
        set shellxquote&
      endif
      let cmd = '"' . $VIMRUNTIME . '\diff"'
    else
      let cmd = substitute($VIMRUNTIME, ' ', '" ', '') . '\diff"'
    endif
  else
    let cmd = $VIMRUNTIME . '\diff'
  endif
  let cmd = substitute(cmd, '!', '\!', 'g')
  silent execute '!' . cmd . ' ' . opt . arg1 . ' ' . arg2 . ' > ' . arg3
  if exists('l:shxq_sav')
    let &shellxquote=l:shxq_sav
  endif
endfunction

set nu
set guifont=Courier_new:h11
set encoding=utf-8
set nocompatible
color desert

filetype off
set rtp+=%USERPROFILE%/vim/bundle/Vundle.vim/
call vundle#begin('%USERPROFILE%/vim/bundle/')

Plugin 'vundleVim/Vundle.vim'
Plugin 'scrooloose/nerdtree'
Plugin 'itchyny/lightline.vim'
Plugin 'mattn/emmet-vim'
Plugin 'Xuyuanp/nerdtree-git-plugin'
Plugin 'altercation/vim-colors-solarized'
Plugin 'jiangmiao/auto-pairs'
Plugin 'kien/rainbow_parentheses.vim'
Plugin 'vim-syntastic/syntastic'
Plugin 'nvie/vim-flake8'
Plugin 'tpope/vim-dispatch'
Plugin 'https://github.com/StanAngeloff/php.vim.git'
Plugin 'joonty/vim-phpqa'
Plugin 'davidhalter/jedi-vim'

call vundle#end()
filetype plugin indent on

"if has('gui_running')
"   set background=dark
"   colorscheme solarized
"else
"   colorscheme zenburn
"endif

filetype plugin on
set omnifunc=syntaxcomplete#Complete

Output of “:verbose JediDebugInfo”

when I do this command, the same windows apear with

import sys
import os


def _get_paths():
    # Get the path to jedi.
    _d = os.path.dirname
    _jedi_path = _d(_d(_d(_d(_d(__file__)))))
    _parso_path = sys.argv[1]
    # The paths are the directory that jedi and parso lie in.
    return {'jedi': _jedi_path, 'parso': _parso_path}


# Remove the first entry, because it's simply a directory entry that equals
# this directory.
del sys.path[0]

if sys.version_info > (3, 4):
    from importlib.machinery import PathFinder

    class _ExactImporter(object):
        def __init__(self, path_dct):
            self._path_dct = path_dct

        def find_module(self, fullname, path=None):
            if path is None and fullname in self._path_dct:
                p = self._path_dct[fullname]
                loader = PathFinder.find_module(fullname, path=[p])
                return loader
            return None

    # Try to import jedi/parso.
    sys.meta_path.insert(0, _ExactImporter(_get_paths()))
    from jedi.evaluate.compiled import subprocess  # NOQA
    sys.meta_path.pop(0)
else:
    import imp

    def load(name):
        paths = list(_get_paths().values())
        fp, pathname, description = imp.find_module(name, paths)
        return imp.load_module(name, fp, pathname, description)

    load('parso')
    load('jedi')
    from jedi.evaluate.compiled import subprocess  # NOQA

# Retrieve the pickle protocol.
pickle_protocol = int(sys.argv[2])
# And finally start the client.
subprocess.Listener(pickle_protocol).listen()

[davidhalter]

I have no idea what this could be. lol. This sounds like such a weird issue. Anyone else?

[lboulard]

+1

Reverting to 0.9.0 from master fixed the issue while editing Python 3 code.
Happen also with function parameters completion.

This is GVim on Windows (all 64bit, GVim and Python) and a new Gvim instance opens on jedi\jedi\evaluate\compiled\subprocess\__main__.py from submodule jedi.

[davidhalter]

@micbou Do you have any idea what this could be? I think this might be related to how Jedi starts subprocesses, but I have no idea about these Windows issues. Contrary to YCM Jedi is run directly in VIM's embedded Python.

[lboulard]

I can no longer reproduce issue editing same python source file after a JediClearCache and a fresh new GVim instance.

[micbou]

I can reproduce the issue by running the :JediDebugInfo command. This happens because sys.executable inside Vim returns the path to the Vim executable on Windows so, instead of starting a Python environment, Jedi is starting a new instance of Vim on that __main__.py file. A solution would be to overwrite sys.executable with the right path on Windows:

sys.executable = os.path.join(sys.prefix, 'python.exe')

[davidhalter]

We already do this for prefix anyway, because it gets messed up by virtualenvs, so this would be a good solution I think.

[yexianingyue]

hello .how did you solve it ? I had the same problem.

[AndreyZhdanov]

put
py3 import os; sys.executable=os.path.join(sys.prefix, 'python.exe')

into .vimrc or _vimrc file

[davidhalter]

@AndreyZhdanov Could you try to put that to the beginning of pythonx/jedi_vim.py? And see if it works?

[AndreyZhdanov]

@davidhalter Yes. I put
sys.executable=os.path.join(sys.prefix, 'python.exe')

after import os – line 9 and it works fine for me.

But I still prefer to have this piece in _vimrc beer in mind update of the script. And it looks like vim bug not issue with jedi-vim.

What is your opinion ?

[blueyed]

1. Is this broken/weird already with :py import sys; print(sys.executable) and :py3 import sys; print(sys.executable)?

2. Does it only happen with gvim, or also vim on the terminal/cmd.exe (if you have that)?

3. how is Python built into Vim? Dynamically? (look for "py" in :version)

Reverting to 0.9.0 from master fixed the issue while editing Python 3 code.

It would be useful to have this git-bisected to the commi where it started to happen.

[AndreyZhdanov]

1. I don't understand 1st question, the script is loading during startup, command in 1st point could be executed after. When I put these commands in _vimrc everything is fine.
2. With gvim only.
3. Dynamically.

[blueyed]

@AndreyZhdanov
Re 1.: I've meant to execute the command in a running gvim instance

[tngreene]

Can confirm: Reverting to 0.9.0 fixed the problem

[blueyed]

@tngreene
Can you bisect this to a particular commit?

[tngreene]

4c430ed536b6484f7c63929a4426d64f981d994e is the first bad commit
commit 4c430ed536b6484f7c63929a4426d64f981d994e
Author: Daniel Hahler <[email protected]>
Date:   Sat Jul 28 01:46:12 2018 +0200

    Improve JediDebugInfo for envs (#858)

    Improve JediDebugInfo for envs

    This is taken out of https://github.com/davidhalter/jedi-vim/pull/836.

Fortunately, this does not scream implausible to me.

[blueyed]

@tngreene
Does it happen (only) with :JediDebugInfo for you?
I assume 4c430ed536b6484f7c63929a4426d64f981d994e triggers it for this case, since it uses jedi.api.environment.

Can you post the output from running the Vim commands from the first point in https://github.com/davidhalter/jedi-vim/issues/870#issuecomment-429607475, please?

[micbou]

@blueyed Did you miss https://github.com/davidhalter/jedi-vim/issues/870#issuecomment-413937942? This comment explains the issue and gives a solution so it should only be a matter of implementing it.

[tngreene]

@tngreene
Does it happen (only) with :JediDebugInfo for you?
I assume 4c430ed triggers it for this case, since it uses jedi.api.environment.

Can you post the output from running the Vim commands from the first point in #870 (comment), please?

Running with :py breaks, because I don't have Python 2 installed. with :py3 it prints C:\Vim\gvim.exe

[blueyed]

@micbou
I wanted to understand the issue first.

It is clear now: since Python is linked into GVim gvim.exe appears to be the executable.
Sounds a bit like a GVim bug / build issue though: on Linux it returns a python binary with gvim and vim (dynamically linked also). Please consider reporting it with Vim also.

As for fixing this in jedi/jedi-vim: please submit a PR.

[micbou]

Sounds a bit like a GVim bug / build issue though: on Linux it returns a python binary with gvim and vim (dynamically linked also). Please consider reporting it with Vim also.

This has already been discussed on the Vim mailing list and it's not an issue with Vim but a peculiarity of Python on Windows: when a program is embedding Python, sys.executable points to the executable of that program instead of the interpreter.

As for fixing this in jedi/jedi-vim: please submit a PR.

I would if I were a user of jedi-vim but I am not.

[tngreene]

put
py3 import os; sys.executable=os.path.join(sys.prefix, 'python.exe')

into .vimrc or _vimrc file

You missed importing sys

py3 import os; import sys; sys.executable=os.path.join(sys.prefix, 'python.exe')

This made it work for me. Thank you! Looking forward to an official fix, however! That way no one else gets confused.

[lboulard]

Hi,

Finally, I have again issue of launching gvim.exe instead of python.exe.
I have been able to bisect commit of issue in https://github.com/davidhalter/jedi/commit/f9cbc65f2d114d6252a1b42f9deb958d54ab7b95 (from pythonx/jedi submodule).

Reverting this commit from version v0.13.1 of jedi allow JediDebugInfo to work correctly.

Hope this help.

[blueyed]

@lboulard
Thanks for finding this.

@davidhalter
You said:

We already do this for prefix anyway, because it gets messed up by virtualenvs, so this would be a good solution I think.

Where is that?

I think a fix needs to be applied to https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/api/environment.py#L133 (if it should be done for Jedi, which makes sense, since Python might be embedded also in non-Vim apps).

[j5shi]

Any update about the fix?

[gerroon]

I have the same issue.

[davidhalter]

Where is that?

@blueyed https://github.com/davidhalter/jedi-vim/blob/master/pythonx/jedi_vim.py#L31

I think a fix needs to be applied to https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/api/environment.py#L133 (if it should be done for Jedi, which makes sense, since Python might be embedded also in non-Vim apps).

I tend to agree. But what would be a good fix? I'm not really sure how to do get the executable path of Python if sys.executable was overridden.

[ganwell]

sys.executable is not overridden, vim loads python37.dll into vim, so the path to gvim.exe is the correct value for sys.executable. I wonder why it is different on unix.

[davidhalter]

so the path to gvim.exe is the correct value for sys.executable

I can see that and I'm not even arguing against it. The interesting thing would then be: How do we reliably get the Python executable path?

[lboulard]

May be this can help.

Reference: https://www.python.org/dev/peps/pep-0514/

Python executable binaries are always defined from Windows registry: python_executables.bat

@SETLOCAL ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION
@FOR /F "usebackq" %%v in (`py.exe -c "import sys;print(sys.winver)"`) DO @SET "VERSION=%%v"
@ECHO Current version: Python %VERSION%
@SET "HKEY=HKLM\Software\Python\PythonCore\%VERSION%"
@FOR /F "usebackq skip=2 tokens=1,2*" %%A IN (`REG QUERY %HKEY% /v Version 2^>nul`) DO @(
  ECHO %%A: "%%C"
)
@FOR /F "usebackq skip=2 tokens=1,2*" %%A IN (`REG QUERY %HKEY%\InstallPath /v *ExecutablePath 2^>nul`) DO @(
  IF /I "%%A" == "ExecutablePath" ECHO Console: "%%C"
  IF /I "%%A" == "WindowedExecutablePath" ECHO Windows: "%%C"
)

Output:

Current version: Python 3.6
Version: "3.6.7"
Console: "C:\Program Files\Python36\python.exe"
Windows: "C:\Program Files\Python36\pythonw.exe"

Note that WindowedExecutablePath key may be missing for console only installation.

Any _virtualenv_ is made from system installation.
In the past, I also known that you could install Python as user only. Not sure if user installation is still supported by Core team. In case of user installation, looks into HKCU and not HKLM registry base.

Windows _Vim_/_GVim_ are directly referencing a DLL name not the python installation. This is not correct as you shall look into registry for correct path (due to security issue of resolving library with PATH variable on Windows). But this is how Vim works. Probably to match Unix like behavior for vim library loader.

Edit: I only works in 64bit Python/Vim on 64bit Windows due to terrible mess of mixing 32/64 bits.

[ganwell]

I can see that and I'm not even arguing against it. The interesting thing would then be: How do we reliably get the Python executable path?

In theory multiprocessing is facing the same problem, I run some tests with gvim.exe and if it works in multiprocessing, I check that code.

[ganwell]

I can see that and I'm not even arguing against it. The interesting thing would then be: How do we reliably get the Python executable path?

Mutliprocessing suffers from the same problem. It executes gvim.exe. When I am using gvim.exe (sometimes, I have to fix stuff on windows}, I don't use virtualenvs, because they don't work properly with gvim.exe. They also don't work with msys2-vim, because msys2-vim is linked against msys2-python which has a whole lot of other problems. So the situation is very bad on windows. Something like this would probably be best (pseudo code):

```python
if on_windows and sys_executable_is_gvim_exe:
standard_interpreter = pjoin(sys.prefix, "python.exe")
venv_interpreter = pjoin(sys.prefix, "Scripts", "python.exe")
if exists(standard_interpreter):
return standard_interpreter
else if exists(venv_interpreter):
return venv_interpreter
else:
unix_code() # Also for the msys(2)-vim version
````
This would be correct too if someone actually uses virtualenv with gvim.exe (using a plugin changing sys.prefix, I guess). I would do a PR if we'd agree to a solution.

[ganwell]

Sidenote on gvim.exe + virutalenvs:

gvim.exe actually loads from the dll in the virtualenv, but probably only the python.exe is a wrapper that sets sys.prefix. So as far as gvim.exe is concerned we are not in a virtualenv.

````cmd
C:\Users\ganwell\Repositories>python -m venv test

C:\Users\ganwell\Repositories>test\Scripts\activate

(test) C:\Users\ganwell\Repositories>gvim

(test) C:\Users\ganwell\Repositories>rmdir /s test
test, Are you sure (Y/N)? y
test\Scripts\python3.dll - Access is denied.
test\Scripts\python37.dll - Access is denied.
````

It seems like virtualenv is a ugly hack, as everything in software is... sigh. python37.dll should consider the virtualenv environment variable and set sys.prefix accordingly.

[davidhalter]

@ganwell Thanks for the research. It looks really dire :/

Maybe the best approach is just to try to find the path if not sys.executable.startswith("python.exe"). We probably also need to use sys.base_prefix, if that's available, but that's a whole other issue...

[ganwell]

Maybe the best approach is just to try to find the path if not sys.executable.startswith("python.exe"). We probably also need to use sys.base_prefix, if that's available, but that's a whole other issue...

If we invert the logic we should probably try to be complete in the list of accepted interpreters:

1. if on_windows and not is_known_interpreter (see below)
2. Then check standard_interpreter and venv_interpreter
3. If nothing is found we still return sys.executable (maybe its an interpreter that can handle .py files, but we don't know it yet)

$> ipython
In [1]: import os.path as path

In [2]: import sys

In [3]: path.split(sys.executable)[-1] in ["python.exe", "python3.exe", "pypy.exe", "pypy3.exe"]
Out[3]: True
````

This should support msys2 and normal vim.exe/gvim.exe and it should fall-back to the most likely scenario: an unknown interpreter in sys.executable

There is already some logic in `jedi/api/environment.py`. I understand that this is general code not specific for jedi-vim and I try to respect/merge with existing behavior.

Why do you reset sys.prefix to sys.base_prefix in jedi_vim.py? Wouldn't that break stuff, for example finding the right interpreter? My venv_interpreter check gets useless then (not that it is working on windows, but still). I see that sys.path is more important than sys.prefix, but... ähm.

$> ag base_prefix
pythonx/jedi_vim.py
31: sys.prefix = sys.base_prefix
```

I guess we can discuss the rest on the PR.

[davidhalter]

I guess we can discuss the rest on the PR.

Awesome. I think this should all be part of Jedi. So please just go on and add it to jedi/api/environment.py.

Why do you reset sys.prefix to sys.base_prefix in jedi_vim.py? Wouldn't that break stuff, for example finding the right interpreter?

I reset that path, because it pointed to a virtualenv that was not properly loaded. AFAIK the Python executable would still point to /usr/bin/python when these two variables would point to some random VIRTUALENV, if it was activated. I think it might break some things, but generally I consider this an issue on VIM's side of how they load Python.

I'm not exactly sure anymore to which errors this led, but it was pretty significant, IIRC.

[odormond]

FYI, the problem exists also on macOS Mojave with vim and python 3.7 from latest homebrew.

[blueyed]

@odormond
Using gvim?
Do you see https://github.com/davidhalter/jedi-vim/issues/889?

[odormond]

Hi @blueyed,

I'm using vim directly within tmux and I do see #889. Explicitly setting sys.executable to the real python executable solves #889 too.

[ganwell]

@davidhalter Something cam up and I can't do the PR soonish. Don't wait for me.

For all those needing a quick solution: replace all instances of sys.executable in jedi/api/environment.py with the right interpreter.

[odormond]

I found https://bugs.python.org/issue34725 while reading the python changelog. To me, it is pointing at the root cause of the problem. A workaround is mentioned but in my understanding it would require a change in vim. However they aren’t that much happy with it and another solution might finally replace it.

[blueyed]

In #895 it was indicated that this might be caused by using SameEnvironment (https://github.com/davidhalter/jedi/commit/f9cbc65f2d114d6252a1b42f9deb958d54ab7b95).

@davidhalter
Regarding the hack in https://github.com/davidhalter/jedi-vim/blob/7f4f2db260e3e693c8b9fc91498f9299c99148c2/pythonx/jedi_vim.py#L31 - that also affects / gets used e.g. with Neovim, where Python is not linked, but an external prog (used through a virtualenv).

Does reverting this fixes for people affected by this?
(sorry, I still have no way to reproduce this - it would be really good if somebody could find a fix for this)

[blueyed]

Discussed for Vim at https://github.com/vim/vim/issues/3707. Interesting comment there how YCM works around it: https://github.com/vim/vim/issues/3707#issuecomment-450140767.

[blayz3r]

Discussed for Vim at vim/vim#3707. Interesting comment there how YCM works around it: vim/vim#3707 (comment).

@blueyed @davidhalter Is a YCM like workaround in the works?

[blueyed]

@blayz3r
Not from me, and also not from @davidhalter I guess.
I do not have a Windows/MacOS machine to test this myself.

[dusans]

I have tested the YCM workaround on

• Windows 7 64 bit
• Vim 8.1.700
• latest jedi-vim

py import os; sys.executable=os.path.join(sys.exec_prefix, 'python.exe')

It works :+1:

[blueyed]

Cool. I assumed so.. :)

What is missing now is somebody to actually step up and get this fixed (maybe in Jedi itself, since it affects Jedi being integrated elsewhere also, not just inside of Vim).
I guess this could be done as a fallback when getting the version information fails using the current method, but maybe also based on the system and Python version being used (it affects MacOS and Windows, and there is an issue for Python to get it fixed there eventually).

[blayz3r]

Cool. I assumed so.. :)

What is missing now is somebody to actually step up and get this fixed (maybe in Jedi itself, since it affects Jedi being integrated elsewhere also, not just inside of Vim).
I guess this could be done as a fallback when getting the version information fails using the current method, but maybe also based on the system and Python version being used (it affects MacOS and Windows, and there is an issue for Python to get it fixed there eventually).

@blueyed @davidhalter Can we have the above in Jedi-vim as a temp solution or something we can add to the vimrc or some documentation. I don't think vim, Jedi itself or python will fix this anytime soon, the bug has been around for more than half a year and doesn't seem high on anyone's priority list.

[blueyed]

Sure.

doesn't seem high on anyone's priority list

Not on mine at least.
I would really appreciate if one of the many affected people would step and fix this.
As for a temp workaround, I think it has been mentioned that you can either fix it in jedi, jedi-vim or your vimrc already, by setting sys.executable.
I do not think such a temporary workaround should be added to docs.

[blayz3r]

@blueyed lol. Would if I could. Just to confirm is this the same issue referenced here : https://github.com/davidhalter/jedi/issues/1072

[blueyed]

Yes.

[Integralist]

For anyone interested I run locally...

pip install jedi
python -c 'import sys; print(sys.executable)'  # <py_bin>

Then inside of vim I open a Python script and then execute the following command:

:python3 import os; sys.executable='<py_bin>'; print(sys.executable)

[davidhalter]

For all those people having this issue: Could you please execute this command and show me the output?

:python3 import sys; print(sys.implementation, sys.flags, dir(sys), sys.abiflags, sys.implementation, sys.thread_info)

I need to know if there's a way to identify these embedded Python versions.

[scrosland]

Mine says (line breaks added for readability):

namespace(_multiarch='darwin', cache_tag='cpython-37', hexversion=50791152, name='cpython', 
version=sys.version_info(major=3, minor=7, micro=2, releaselevel='final', serial=0)) 
sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, 
no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, 
quiet=0, hash_randomization=1, isolated=0, dev_mode=False, utf8_mode=0) 
['__breakpointhook__', '__displayhook__', '__doc__', '__excepthook__', 
'__interactivehook__', '__loader__', '__name__', '__package__', '__spec__', '__stderr__', 
'__stdin__', '__stdout__', '_clear_type_cache', '_current_frames', '_debugmallocstats', 
'_framework', '_getframe', '_git', '_home', '_xoptions', 'abiflags', 'api_version', 'argv', 
'base_exec_prefix', 'base_prefix', 'breakpointhook', 'builtin_module_names', 
'byteorder', 'call_tracing', 'callstats', 'copyright', 'displayhook', 'dont_write_bytecode', 
'exc_info', 'excepthook', 'exec_prefix', 'executable', 'exit', 'flags', 'float_info', 
'float_repr_style', 'get_asyncgen_hooks', 'get_coroutine_origin_tracking_depth', 
'get_coroutine_wrapper', 'getallocatedblocks', 'getcheckinterval', 'getdefaultencoding', 
'getdlopenflags', 'getfilesystemencodeerrors', 'getfilesystemencoding', 'getprofile', 
'getrecursionlimit','getrefcount', 'getsizeof', 'getswitchinterval', 'gettrace', 'hash_info', 
'hexversion', 'implementation', 'int_info', 'intern', 'is_finalizing', 'maxsize', 'maxunicode', 
'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 
'set_asyncgen_hooks', 'set_coroutine_origin_tracking_depth', 'set_coroutine_wrapper', 
'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit', 'setswitchinterval', 
'settrace', 'stderr', 'stdin', 'stdout', 'thread_info', 'version', 'version_info', 
'warnoptions'] m namespace(_multiarch='darwin', cache_tag='cpython-37', 
hexversion=50791152, name='cpython', version=sys.version_info(major=3, minor=7, 
micro=2, releaselevel='final', serial=0)) sys.thread_info(name='pthread', 
lock='mutex+cond', version=None)

That's vim from homebrew on macOS:

user:~$ which vim
/usr/local/bin/vim
user:~$ vim --version
VIM - Vi IMproved 8.1 (2018 May 18, compiled Jan 31 2019 16:43:52)
macOS version
Included patches: 1-850
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl               +extra_search      +mouse_netterm     +tag_old_static
+arabic            +farsi             +mouse_sgr         -tag_any_white
+autocmd           +file_in_path      -mouse_sysmouse    -tcl
+autochdir         +find_in_path      +mouse_urxvt       +termguicolors
-autoservername    +float             +mouse_xterm       +terminal
-balloon_eval      +folding           +multi_byte        +terminfo
+balloon_eval_term -footer            +multi_lang        +termresponse
-browse            +fork()            -mzscheme          +textobjects
++builtin_terms    +gettext           +netbeans_intg     +textprop
+byte_offset       -hangul_input      +num64             +timers
+channel           +iconv             +packages          +title
+cindent           +insert_expand     +path_extra        -toolbar
-clientserver      +job               +perl              +user_commands
+clipboard         +jumplist          +persistent_undo   +vartabs
+cmdline_compl     +keymap            +postscript        +vertsplit
+cmdline_hist      +lambda            +printer           +virtualedit
+cmdline_info      +langmap           +profile           +visual
+comments          +libcall           -python            +visualextra
+conceal           +linebreak         +python3           +viminfo
+cryptv            +lispindent        +quickfix          +vreplace
+cscope            +listcmds          +reltime           +wildignore
+cursorbind        +localmap          +rightleft         +wildmenu
+cursorshape       +lua               +ruby              +windows
+dialog_con        +menu              +scrollbind        +writebackup
+diff              +mksession         +signs             -X11
+digraphs          +modify_fname      +smartindent       -xfontset
-dnd               +mouse             +startuptime       -xim
-ebcdic            -mouseshape        +statusline        -xpm
+emacs_tags        +mouse_dec         -sun_workshop      -xsmp
+eval              -mouse_gpm         +syntax            -xterm_clipboard
+ex_extra          -mouse_jsbterm     +tag_binary        -xterm_save
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X -DMACOS_X_DARWIN  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang   -L. -fstack-protector-strong -L/usr/local/lib -L/usr/local/opt/libyaml/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib  -L/usr/local/lib -o vim        -lncurses -liconv -lintl -framework AppKit  -L/usr/local/opt/lua/lib -llua5.3 -mmacosx-version-min=10.14 -fstack-protector-strong -L/usr/local/lib  -L/usr/local/Cellar/perl/5.28.1/lib/perl5/5.28.1/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc  -L/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin -lpython3.7m -framework CoreFoundation  -lruby.2.6
user:~$

[davidhalter]

@scrosland Thanks a lot! I forgot, can you please also add:

:python3 print(sys.executable)
:python3 print(sys.prefix)
:python3 print(sys.base_prefix)
:python3 print(sys.exec_prefix)
:python3 print(sys.base_exec_prefix)

Thank you!

[davidhalter]

I just realized that there's an even better way, if possible just drop all the output of :python3 print(sysconfig.get_config_vars()). I'm pretty sure that I have all the information I need with that.

PS: I'd also be really interested to have this output from windows people.

[scrosland]

OK, to get a clean version of that I did:

user:~$ cat /tmp/s.vim
python3 <<EOF
import sysconfig
with open("/tmp/s.txt", "w") as fp:
    print(sysconfig.get_config_vars(), file=fp)
EOF
quit
user:~$ vim -u NONE -S /tmp/s.vim
user:~$ 

Which gave me this file: s.txt

[tomhull]

Here is a windows output:

{'prefix': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32', 'exec_prefix': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32', 'py_version': '3.6.8', 'py_version_short': '3.6', 'py_version_nodot': '36', 'installed_base': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32', 'base': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32', 'installed_platbase': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32', 'platbase': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32', 'projectbase': 'C:\\Program Files (x86)\\Vim\\vim81', 'abiflags': '', 'LIBDEST': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32\\Lib', 'BINLIBDEST': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32\\Lib', 'INCLUDEPY': 'C:\\Users\\betascoo8\\AppData\\Local\\Programs\\Python\\Python36-32\\Include', 'EXT_SUFFIX': '.pyd', 'EXE': '.exe', 'VERSION': '36', 'BINDIR': 'C:\\Program Files (x86)\\Vim\\vim81', 'SO': '.pyd', 'userbase': 'C:\\Users\\betascoo8\\AppData\\Roaming\\Python', 'srcdir': 'C:\\Program Files (x86)\\Vim\\vim81'}

version:

VIM - Vi IMproved 8.1 (2018 May 18, compiled May 18 2018 18:36:07)
MS-Windows 32-bit GUI version with OLE support
Included patches: 1
Compiled by mool@tororo
Huge version with GUI.  Features included (+) or not (-):
+acl                +byte_offset        +comments           +digraphs           +farsi              +iconv/dyn          +linebreak          +mouse              +packages           +python3/dyn        +startuptime        -termguicolors      +user_commands      +wildignore
+arabic             +channel            +conceal            +directx            +file_in_path       +insert_expand      +lispindent         +mouseshape         +path_extra         +quickfix           +statusline         +terminal           +vertsplit          +wildmenu
+autocmd            +cindent            +cryptv             -dnd                +find_in_path       +job                +listcmds           +multi_byte_ime/dyn +perl/dyn           +reltime            -sun_workshop       -tgetent            +virtualedit        +windows
+autoservername     +clientserver       +cscope             -ebcdic             +float              +jumplist           +localmap           +multi_lang         +persistent_undo    +rightleft          +syntax             -termresponse       +visual             +writebackup
+balloon_eval       +clipboard          +cursorbind         +emacs_tags         +folding            +keymap             +lua/dyn            +mzscheme/dyn       -postscript         +ruby/dyn           +tag_binary         +textobjects        +visualextra        -xfontset
-balloon_eval_term  +cmdline_compl      +cursorshape        +eval               -footer             +lambda             +menu               +netbeans_intg      +printer            +scrollbind         +tag_old_static     +timers             +viminfo            -xim
+browse             +cmdline_hist       +dialog_con_gui     +ex_extra           +gettext/dyn        +langmap            +mksession          +num64              +profile            +signs              -tag_any_white      +title              +vreplace           +xpm_w32
++builtin_terms     +cmdline_info       +diff               +extra_search       -hangul_input       +libcall            +modify_fname       +ole                +python/dyn         +smartindent        +tcl/dyn            +toolbar            -vtp                -xterm_save
   system vimrc file: "$VIM\vimrc"
     user vimrc file: "$HOME\_vimrc"
 2nd user vimrc file: "$HOME\vimfiles\vimrc"
 3rd user vimrc file: "$VIM\_vimrc"
      user exrc file: "$HOME\_exrc"
  2nd user exrc file: "$VIM\_exrc"
  system gvimrc file: "$VIM\gvimrc"
    user gvimrc file: "$HOME\_gvimrc"
2nd user gvimrc file: "$HOME\vimfiles\gvimrc"
3rd user gvimrc file: "$VIM\_gvimrc"
       defaults file: "$VIMRUNTIME\defaults.vim"
    system menu file: "$VIMRUNTIME\menu.vim"
Compilation: cl -c /W3 /nologo  -I. -Iproto -DHAVE_PATHDEF -DWIN32  -DFEAT_CSCOPE -DFEAT_TERMINAL -DFEAT_NETBEANS_INTG -DFEAT_JOB_CHANNEL   -DFEAT_XPM_W32   -DWINVER=0x0501 -D_WIN32_WINNT=0x0501 /MP -DHAVE_STDINT_H /Ox /GL -DNDEBUG /arch:IA32 /Zl /MT -DFEAT_OLE -DFEAT_MBYTE_IME -DDYNAMIC_IME -DFEAT_GUI_W32 -DFEAT_DIRECTX -DDYNAMIC_DIRECTX -DFEAT_DIRECTX_COLOR_EMOJI -DDYNAMIC_ICONV -DDYNAMIC_GETTEXT -DFEAT_TCL -DDYNAMIC_TCL -DDYNAMIC_TCL_DLL=\"tcl86t.dll\" -DDYNAMIC_TCL_VER=\"8.6\" -DFEAT_LUA -DDYNAMIC_LUA -DDYNAMIC_LUA_DLL=\"lua53.dll\" -DFEAT_PYTHON -DDYNAMIC_PYTHON -DDYNAMIC_PYTHON_DLL=\"python27.dll\" -DFEAT_PYTHON3 -DDYNAMIC_PYTHON3 -DDYNAMIC_PYTHON3_DLL=\"python36.dll\" -DFEAT_MZSCHEME -I "E:\Racket\include" -DMZ_PRECISE_GC -DDYNAMIC_MZSCHEME -DDYNAMIC_MZSCH_DLL=\"libracket3m_a36fs8.dll\" -DDYNAMIC_MZGC_DLL=\"libracket3m_a36fs8.dll\" -DFEAT_PERL -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DDYNAMIC_PERL -DDYNAMIC_PERL_DLL=\"perl524.dll\" -DFEAT_RUBY -DDYNAMIC_RUBY -DDYNAMIC_RUBY_VER=24 -DDYNAMIC_RUBY_DLL=\"msvcrt-ruby240.dll\" -DFEAT_HUGE /Fd.\ObjGXOULYHTRZi386/ /Zi
Linking: link  /nologo /subsystem:windows,5.01 /opt:ref /LTCG:STATUS oldnames.lib kernel32.lib advapi32.lib shell32.lib gdi32.lib  comdlg32.lib ole32.lib uuid.lib /machine:i386 gdi32.lib version.lib   winspool.lib comctl32.lib advapi32.lib shell32.lib  /machine:i386  libcmt.lib oleaut32.lib user32.lib  /nodefaultlib:lua53.lib  /STACK:8388608  /nodefaultlib:python27.lib /nodefaultlib:python36.lib   "E:\ActiveTcl\lib\tclstub86.lib" WSock32.lib xpm\x86\lib-vc14\libXpm.lib /PDB:gvim.pdb -debug-- INSERT --
0

[davidhalter]

Ok. Thanks a lot! That will help. Can you also send my sys.executable, sys.exec_base_prefix and sys.base_prefix? Sorry for bugging you again, but this feels liket he last missin piece. I wasn't initially sure which information was important (and sysconfig doesn't contain everything.

And also: I suppose you both have the this issue, right?

[davidhalter]

I quickly checked out the source code of Python. Indeed it's done differently for the different OS's. Mac is almost the same as Linux, but if no slash is found in the argv[0], something additional is done, it will use _NSGetExecutablePath, which probably returns VIM.

For Microsoft it's entirely different - it's even in separate files: A function called GetModuleFileNameW, whch is also just getting the VIM binary.

So it looks like this is a general issue with sys.executable might be a general one for Windows an Mac and definitely needs to be fixed in Jedi. I quickly checked the source code of multiprocessing where there seem to be some sys.executable references. They leave it to the user to change the executable, but at least offer a function: BaseContext.set_executable.

[scrosland]

And also: I suppose you both have the this issue, right?

Yes, I do, which I workaround in vimrc but assigning a sensible value to sys.executable.

[tomhull]

Ok. Thanks a lot! That will help. Can you also send my sys.executable, sys.exec_base_prefix and sys.base_prefix? Sorry for bugging you again, but this feels liket he last missin piece. I wasn't initially sure which information was important (and sysconfig doesn't contain everything.

No worries,

sys.executable: C:\Program Files (x86)\Vim\vim81\gvim.exe
sys.base_prefix: C:\Users\betascoo8\AppData\Local\Programs\Python\Python36-32
sys doesn't have an exec_base_prefix but it does have
sys.exec_prefix: C:\Users\betascoo8\AppData\Local\Programs\Python\Python36-32

And also: I suppose you both have the this issue, right?

Yep, but I've also never managed to get the .vimrc workaround to work and not open a second gvim window.

[tomhull]

Actually, workaround works for me, but only if I spell "executable" correctly.

[davidhalter]

@tomhull Is your Python 3.6 in C:\Users\betascoo8\AppData\Local\Programs\Python\Python36-32\Scripts\python.exe?

@scrosland Is your Python executable in /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/bin/python?

[tomhull]

@davidhalter nope, it's above the Scripts folder:

C:\Users\betascoo8\AppData\Local\Programs\Python\Python36-32\python.exe

[tngreene]

Glad to see this is getting attention again. I've tried to follow along with the conversation, I'm on Windows. What commands should I be running to give you what information?

[davidhalter]

@tngreene I guess most things have been answered, because of great gentlemen above. I'm interested what your sys.prefix is and where your Python is located at.

[davidhalter]

I hope the commit above fixes this issue for you guys, can you please try? Thanks! We will then create a new jedi release.

[tomhull]

Unfortunately I'm still getting the __main__.py window opening.

I disabled my .vimrc workaround
I uninstalled then reinstalled jedi using pip install -e git://github.com/davidhalter/jedi.git#egg=jedi

running
:py3 print( import jedi; jedi.get_default_environment() )
returns:
<SameEnvironment: 3.6.8 in C:\Users\betascoo8\AppData\Local\Programs\Python\Python36-32>

but
:py3 print(sys.executable)
returns
C:\Program Files (x86)\Vim\vim81\gvim.exe

[blueyed]

@tomhull

but
:py3 print(sys.executable)
returns
C:\Program Files (x86)\Vim\vim81\gvim.exe

This is expected I guess - it does not change sys.executable.

What does :py3 print( import jedi; jedi.get_default_environment()._start_executable ) give you?

[tomhull]

:py3 import jedi; print( jedi.get_default_environment()._start_executable )
->
C:\Program Files (x86)\Vim\vim81\gvim.exe

[blueyed]

@tomhull
Please check how / why the code from https://github.com/davidhalter/jedi/commit/b4a4dacebdcec303d68155c33f9cdf8746a5809d#diff-c204f420c3d90e6c005f6354c65dc638R187 does not work for you then.

[scrosland]

@scrosland Is your Python executable in /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/bin/python?

No, it's /usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7/bin/python3.7 which is also accessible via symlinks as /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.7/bin/python3 or simply /usr/local/bin/python3.

[tngreene]

@tngreene I guess most things have been answered, because of great gentlemen above. I'm interested what your sys.prefix is and where your Python is located at.

From running python on the command line and from running :python3

import sys; print(sys.prefix)

prints C:\Users\Ted\Miniconda3

where python says C:\Users\Ted\Miniconda3\python.exe

With py3 import os; import sys; sys.executable=os.path.join(sys.prefix, 'python.exe') in my vimrc (see above):

Jedi-vim debug information

jedi-vim version

• jedi-vim git version: 0.9.0-68-gea5e87a
• jedi git submodule status: bd1010bbd2693f189ff780eb21fc4294071cb280 pythonx/jedi (v0.13.1)
• parso git submodule status: f1ee7614c921e538aa093cd96c78e9aaced7a8ca pythonx/parso (v0.3.3)

Global Python

Using Python version 3 to access Jedi.

• global sys.version: 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
• global site module: C:\Users\Ted\Miniconda3\Lib\site.py

Jedi

• path: C:\Users\Ted\.vim\bundle\jedi-vim\pythonx\jedi\jedi\__init__.py
• version: 0.13.1

Jedi environment:

• executable: C:\Users\Ted\Miniconda3\python.exe
• sys_path:
  
  
  
  • C:\Users\Ted\Miniconda3\envs\blender_279_353\Lib\site-packages
  
  
  • C:\users\ted\XPlane2Blender\io_xplane2blender
  
  
  • C:\Users\Ted\XPlane2Blender
  
  
  • C:\Users\Ted\Miniconda3\python36.zip
  
  
  • C:\Users\Ted\Miniconda3\DLLs
  
  
  • C:\Users\Ted\Miniconda3\lib
  
  
  • C:\Users\Ted\Miniconda3
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\cythonize-0.0.0-py3.6-win-amd64.egg
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\win32
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\win32\lib
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\Pythonwin
  
  

Known environments

• (C:\Users\Ted\Miniconda3\python.exe)

Settings

g:jedi#popup_select_first = 0 (default: 1)
g:jedi#completions_command = '<C-n>' (default: '<C-Space>')
g:jedi#rename_command = '<leader>rn' (default: '<leader>r')
g:jedi#auto_vim_configuration = 0 (default: 1)
g:jedi#usages_command = '<C-G>' (default: '<leader>n')
g:jedi#goto_command = 'gd' (default: '<leader>d')


  omnifunc=
  completeopt=menu,preview,menuone,noinsert
    Last set from ~\.vim\.vimrc

Without it opens C:\Users\Ted\.vim\bundle\jedi-vim\pythonx\jedi\jedi\evaluate\compiled\subprocess\__main__.py. and gives this error.

ERROR: failed to get sys path from environment: jedi.api.environment.InvalidPythonEnvironment: Could not get version information for 'C:\\Vim\\gvim.exe': UnpicklingError("invalid load key, '3'.",)
  Traceback (most recent call last):
    File "C:\Users\Ted\.vim\bundle\jedi-vim\pythonx\jedi_vim_debug.py", line 71, in display_debug_info
      sys_path = environment.get_sys_path()
    File "C:\Users\Ted\.vim\bundle\jedi-vim\pythonx\jedi\jedi\cache.py", line 143, in wrapper
      result = method(self, *args, **kwargs)
    File "C:\Users\Ted\.vim\bundle\jedi-vim\pythonx\jedi\jedi\api\environment.py", line 128, in get_sys_path
      return self._get_subprocess().get_sys_path()
    File "C:\Users\Ted\.vim\bundle\jedi-vim\pythonx\jedi\jedi\api\environment.py", line 79, in _get_subprocess
      exc))

Checking out the new commit, I don't get the error and the plugin seems to work! Yay! Thank you!

For completeness, here is the final output from JediDebugInfo:

Jedi-vim debug information

jedi-vim version

• jedi-vim git version: 0.9.0-68-gea5e87a-dirty
• jedi git submodule status: +b4a4dacebdcec303d68155c33f9cdf8746a5809d pythonx/jedi (v0.13.2-5-gb4a4dace)
• parso git submodule status: f1ee7614c921e538aa093cd96c78e9aaced7a8ca pythonx/parso (v0.3.3)

Global Python

Using Python version 3 to access Jedi.

• global sys.version: 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)]
• global site module: C:\Users\Ted\Miniconda3\Lib\site.py

Jedi

• path: C:\Users\Ted\.vim\bundle\jedi-vim\pythonx\jedi\jedi\__init__.py
• version: 0.13.2

Jedi environment:

• executable: C:\Users\Ted\Miniconda3\python.exe
• sys_path:
  
  
  
  • C:\Users\Ted\Miniconda3\envs\blender_279_353\Lib\site-packages
  
  
  • C:\users\ted\XPlane2Blender\io_xplane2blender
  
  
  • C:\Users\Ted\XPlane2Blender
  
  
  • C:\Users\Ted\Miniconda3\python36.zip
  
  
  • C:\Users\Ted\Miniconda3\DLLs
  
  
  • C:\Users\Ted\Miniconda3\lib
  
  
  • C:\Users\Ted\Miniconda3
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\cythonize-0.0.0-py3.6-win-amd64.egg
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\win32
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\win32\lib
  
  
  • C:\Users\Ted\Miniconda3\lib\site-packages\Pythonwin
  
  

Known environments

• (C:\Users\Ted\Miniconda3\python.exe)

Settings

g:jedi#popup_select_first = 0 (default: 1)
g:jedi#completions_command = '<C-n>' (default: '<C-Space>')
g:jedi#rename_command = '<leader>rn' (default: '<leader>r')
g:jedi#auto_vim_configuration = 0 (default: 1)
g:jedi#usages_command = '<C-G>' (default: '<leader>n')
g:jedi#goto_command = 'gd' (default: '<leader>d')


  omnifunc=jedi#completions
    Last set from ~\.vim\bundle\jedi-vim\autoload\jedi.vim
  completeopt=menu,preview,menuone,noinsert
    Last set from ~\.vim\bundle\jedi-vim\autoload\jedi.vim

[davidhalter]

@tngreene Could you fetch latest Jedi (not jedi-vim) and try with that if it still happens? I feel like I fixed it.

@scrosland Thanks for that information! I have used your information in the latest commits to improve the environment selection. Could you please test the latest Jedi commits and see if it still happens?

[tngreene]

@davidhalter 8d313e0 Works!

[davidhalter]

@tomhull Can you please try again and upgrade Jedi master? I improved a few things, but if it still doesn't work something looks strange, because it's working for @tngreene on Windows.

[tomhull]

I've had another go with a different win10 machine and had the same issue. I'll have another look and see what is different about my configuration.

[tomhull]

@davidhalter, yes sorry, it works.
I didn't realise that installing jedi-vim with vim-plug would put a copy of jedi in the .vim\bundle\jedi-vim\pythonx folder, and then that version, not the fixed version installed with pip was the one being used.

[davidhalter]

No problem. :)

[davidhalter]

Ok. I just released new Jedi and upgraded the submodule here as well. Let me know if there are further problems.

@blueyed Sorry I wanted to create a PR but was too fast with my fingers. I wanted to do a PR and self merge, because it was just a dependency upgrade. I upgraded parso/jedi. Please look at it and let me know if there are issues with it.

[blueyed]

@davidhalter
LGTM. But s/parso/jedi in https://github.com/davidhalter/jedi/issues/1063#issuecomment-466799438

[scrosland]

@scrosland Thanks for that information! I have used your information in the latest commits to improve the environment selection. Could you please test the latest Jedi commits and see if it still happens?

I updated vim-jedi (and jedi) and it works correctly now without my previous vimrc workaround. Thanks!