Jedi-vim: UnpicklingError when jedi-vim tries to get vim version info

[hoylemd]

Issue

When I try to run a jedi command (e.g. <leader>d), I just get a stack trace. The most relevant line seems to be:

jedi.api.environment.InvalidPythonEnvironment: Could not get version information for '/usr/local/bin/vim': UnpicklingError("invalid load key, '3'.")

Full stack trace:

Traceback (most recent call last):
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py", line 74, in _get_subprocess
    info = self._subprocess._send(None, _get_info)
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/evaluate/compiled/subprocess/__init__.py", line 242, in _send
    is_exception, traceback, result = pickle_load(self._process.stdout)
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/_compatibility.py", line 464, in pickle_load
    return pickle.load(file, encoding='bytes')
_pickle.UnpicklingError: invalid load key, '3'.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi_vim.py", line 140, in wrapper
    return func(*args, **kwargs)
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi_vim.py", line 220, in get_script
    environment=get_environment(),
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/__init__.py", line 108, in __init__
    project, environment=environment, script_path=self.path
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/evaluate/__init__.py", line 94, in __init__
    self.compiled_subprocess = environment.get_evaluator_subprocess(self)
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py", line 113, in get_evaluator_subprocess
    return EvaluatorSubprocess(evaluator, self._get_subprocess())
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py", line 79, in _get_subprocess
    exc))
jedi.api.environment.InvalidPythonEnvironment: Could not get version information for '/usr/local/bin/vim': UnpicklingError("invalid load key, '3'.")
Traceback (most recent call last):
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi_vim.py", line 140, in wrapper
    return func(*args, **kwargs)
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi_vim.py", line 302, in goto
    definitions = script.goto_assignments(follow_imports=True)
AttributeError: 'NoneType' object has no attribute 'goto_assignments'

Steps to reproduce

Load a python script
Move cursor over an identifier
do \d

set nocompatible

let script_dir = fnamemodify(expand('<sfile>'), ':h')
let &runtimepath .= ','.script_dir.','.script_dir.'/after'

" Put your config changes here.
" let g:jedi#show_call_signatures=1

syntax on
filetype plugin indent on

minimal.vimrc:

set nocompatible

let script_dir = fnamemodify(expand('<sfile>'), ':h')
let &runtimepath .= ','.script_dir.','.script_dir.'/after'

set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

Plugin 'gmarik/Vundle.vim'                "required
Plugin 'davidhalter/jedi-vim'

call vundle#end()            " required

" Put your config changes here.
let g:jedi#show_call_signatures=1

syntax on
filetype plugin indent on

Jedi-vim debug information

jedi-vim version

• jedi-vim git version: 0.9.0-61-g2f65c63-dirty
• jedi git submodule status: bd1010bbd2693f189ff780eb21fc4294071cb280 pythonx/jedi (v0.13.1)
• parso git submodule status: 930ec08ab0cd77aae07e24aa78f9c5119be216bb pythonx/parso (v0.3.1-7-g930ec08)

Global Python

Using Python version 3 to access Jedi.

• global sys.version: 3.7.1 (default, Nov 6 2018, 18:49:54), [Clang 9.0.0 (clang-900.0.39.2)]
• global site module: /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site.py

Jedi

• path: /Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/__init__.py
• version: 0.13.1

Jedi environment:

• executable: /usr/local/bin/vim
  ERROR: failed to get sys path from environment: jedi.api.environment.InvalidPythonEnvironment: Could not get version information for '/usr/local/bin/vim': UnpicklingError("invalid load key, '3'.")
  Traceback (most recent call last):
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi_vim_debug.py", line 71, in display_debug_info
  sys_path = environment.get_sys_path()
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/cache.py", line 143, in wrapper
  result = method(self, args, *kwargs)
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py", line 128, in get_sys_path
  return self._get_subprocess().get_sys_path()
  File "/Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py", line 79, in _get_subprocess
  exc))

Settings

  omnifunc=jedi#completions
    Last set from ~/.vim/bundle/jedi-vim/autoload/jedi.vim line 612
  completeopt=menuone,longest,preview
    Last set from ~/.vim/bundle/jedi-vim/plugin/jedi.vim line 31

:version

VIM - Vi IMproved 8.1 (2018 May 18, compiled Oct 29 2018 06:57:16)
macOS version
Included patches: 1-500
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl               ++builtin_terms    +cmdline_info      +diff              +farsi             -hangul_input      +libcall           +modify_fname      -mouse_sysmouse    +packages          +python3           +startuptime       +termguicolors     +user_commands     +wildignore        -xsmp
+arabic            +byte_offset       +comments          +digraphs          +file_in_path      +iconv             +linebreak         +mouse             +mouse_urxvt       +path_extra        +quickfix          +statusline        +terminal          +vartabs           +wildmenu          -xterm_clipboard
+autocmd           +channel           +conceal           -dnd               +find_in_path      +insert_expand     +lispindent        -mouseshape        +mouse_xterm       +perl              +reltime           -sun_workshop      +terminfo          +vertsplit         +windows           -xterm_save
+autochdir         +cindent           +cryptv            -ebcdic            +float             +job               +listcmds          +mouse_dec         +multi_byte        +persistent_undo   +rightleft         +syntax            +termresponse      +virtualedit       +writebackup       
-autoservername    -clientserver      +cscope            +emacs_tags        +folding           +jumplist          +localmap          -mouse_gpm         +multi_lang        +postscript        +ruby              +tag_binary        +textobjects       +visual            -X11               
-balloon_eval      +clipboard         +cursorbind        +eval              -footer            +keymap            -lua               -mouse_jsbterm     -mzscheme          +printer           +scrollbind        +tag_old_static    +timers            +visualextra       -xfontset          
+balloon_eval_term +cmdline_compl     +cursorshape       +ex_extra          +fork()            +lambda            +menu              +mouse_netterm     +netbeans_intg     +profile           +signs             -tag_any_white     +title             +viminfo           -xim               
-browse            +cmdline_hist      +dialog_con        +extra_search      -gettext           +langmap           +mksession         +mouse_sgr         +num64             -python            +smartindent       -tcl               -toolbar           +vreplace          -xpm               
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X -DMACOS_X_DARWIN  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       
Linking: clang   -L. -fstack-protector -L/usr/local/lib -L/usr/local/opt/libyaml/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib  -L/usr/local/lib -o vim        -lncurses -liconv -framework AppKit   -mmacosx-version-min=10.12 -fstack-protector-strong -L/usr/local/lib  -L/usr/local/Cellar/perl/5.28.0/lib/perl5/5.28.0/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc  -L/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin -lpython3.7m -framework CoreFoundation  -lruby.2.5.3 -lobjc    

:messages

Messages maintainer: Bram Moolenaar <[email protected]>
"impact_service/impacts/models.py" 15L, 327C

:scriptnames

  1: ~/platform-impact-service/minimal.vimrc
  2: ~/.vim/bundle/Vundle.vim/autoload/vundle.vim
  3: ~/.vim/bundle/Vundle.vim/autoload/vundle/config.vim
  4: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/syntax/syntax.vim
  5: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/syntax/synload.vim
  6: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/syntax/syncolor.vim
  7: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/filetype.vim
  8: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/ftplugin.vim
  9: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/indent.vim
 10: ~/.vim/bundle/jedi-vim/plugin/jedi.vim
 11: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/getscriptPlugin.vim
 12: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/gzip.vim
 13: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/logiPat.vim
 14: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/manpager.vim
 15: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/matchparen.vim
 16: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/netrwPlugin.vim
 17: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/rrhelper.vim
 18: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/spellfile.vim
 19: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/tarPlugin.vim
 20: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/tohtml.vim
 21: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/vimballPlugin.vim
 22: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/plugin/zipPlugin.vim
 23: ~/unix-setup/vim/syntax/python.vim
 24: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/syntax/python.vim
 25: ~/.vim/bundle/jedi-vim/after/syntax/python.vim
 26: ~/.vim/bundle/jedi-vim/autoload/jedi.vim
 27: ~/.vim/bundle/jedi-vim/ftplugin/python/jedi.vim
 28: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/ftplugin/python.vim
 29: ~/unix-setup/vim/after/ftplugin/python.vim
 30: ~/.vim/bundle/jedi-vim/after/ftplugin/python/jedi.vim
 31: ~/unix-setup/vim/indent/python.vim
 32: /usr/local/Cellar/vim/8.1.0500/share/vim/vim81/indent/python.vim

```

[blueyed]

Looks like it might be trying to load a cache from an incompatible version?

1. Have you tried to see which file is used in /Users/michael.hoyle/.vim/bundle/jedi-vim/pythonx/jedi/jedi/_compatibility.py ?

2. I would suggest moving away Jedi's cache to see if it fixes it, and then try to figure out why it is trying to use incompatible versions.

Anyway, more an issue with Jedi and not jedi-vim.

[hoylemd]

Heh, I thought it was more of a jedi issue, but I wasn't quite sure so I posted here.

Where can I find out how to move away jedi's cache? I don't really know what that means.

[blueyed]

Then try the first suggestion first.. :)
You might find out more from pdb'ing there.

[bradwood]

I get exactly the same error;;; Here is my Debug snippet:

##### Jedi environment: <SameEnvironment: 3.7.1 in /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7>

 - executable: /usr/local/bin/vim
ERROR: failed to get sys path from environment: jedi.api.environment.InvalidPythonEnvironment: Could not get version information for '/usr/local/bin/vim': UnpicklingError("invalid load key, '3'.")
  Traceback (most recent call last):
    File "/Users/brad/.vim/bundle/jedi-vim/pythonx/jedi_vim_debug.py", line 71, in display_debug_info
      sys_path = environment.get_sys_path()
    File "/Users/brad/.vim/bundle/jedi-vim/pythonx/jedi/jedi/cache.py", line 143, in wrapper
      result = method(self, *args, **kwargs)
    File "/Users/brad/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py", line 128, in get_sys_path
      return self._get_subprocess().get_sys_path()
    File "/Users/brad/.vim/bundle/jedi-vim/pythonx/jedi/jedi/api/environment.py", line 79, in _get_subprocess
      exc))

##### Settings

It works fine inside a virtualenv, btw --- it just outside the virtual env that this happens... obviously an environment problem -- but what? any suggestions on where to look.

I'm using pyenv for my system python version management when not inside a virtualenv.

vim version:

VIM - Vi IMproved 8.1 (2018 May 18, compiled Nov 24 2018 19:11:56)
macOS version
Included patches: 1-500
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl               +extra_search      +mouse_netterm     +tag_old_static
+arabic            +farsi             +mouse_sgr         -tag_any_white
+autocmd           +file_in_path      -mouse_sysmouse    -tcl
+autochdir         +find_in_path      +mouse_urxvt       +termguicolors
-autoservername    +float             +mouse_xterm       +terminal
-balloon_eval      +folding           +multi_byte        +terminfo
+balloon_eval_term -footer            +multi_lang        +termresponse
-browse            +fork()            -mzscheme          +textobjects
++builtin_terms    +gettext           +netbeans_intg     +timers
+byte_offset       -hangul_input      +num64             +title
+channel           +iconv             +packages          -toolbar
+cindent           +insert_expand     +path_extra        +user_commands
-clientserver      +job               +perl              +vartabs
+clipboard         +jumplist          +persistent_undo   +vertsplit
+cmdline_compl     +keymap            +postscript        +virtualedit
+cmdline_hist      +lambda            +printer           +visual
+cmdline_info      +langmap           +profile           +visualextra
+comments          +libcall           -python            +viminfo
+conceal           +linebreak         +python3           +vreplace
+cryptv            +lispindent        +quickfix          +wildignore
+cscope            +listcmds          +reltime           +wildmenu
+cursorbind        +localmap          +rightleft         +windows
+cursorshape       +lua               +ruby              +writebackup
+dialog_con        +menu              +scrollbind        -X11
+diff              +mksession         +signs             -xfontset
+digraphs          +modify_fname      +smartindent       -xim
-dnd               +mouse             +startuptime       -xpm
-ebcdic            -mouseshape        +statusline        -xsmp
+emacs_tags        +mouse_dec         -sun_workshop      -xterm_clipboard
+eval              -mouse_gpm         +syntax            -xterm_save
+ex_extra          -mouse_jsbterm     +tag_binary        
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X -DMACOS_X_DARWIN  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       
Linking: clang   -L. -fstack-protector -L/usr/local/lib -L/usr/local/opt/libyaml/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib  -L/usr/local/lib -o vim        -lncurses -liconv -lintl -framework AppKit  -L/usr/local/opt/lua/lib -llua5.3 -mmacosx-version-min=10.14 -fstack-protector-strong -L/usr/local/lib  -L/usr/local/Cellar/perl/5.28.0/lib/perl5/5.28.0/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc  -L/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin -lpython3.7m -framework CoreFoundation  -lruby.2.5.3 -lobjc    

[blueyed]

Please try debugging this yourself.

1. see above
2. try setting it to 2 explicitly in https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/evaluate/compiled/subprocess/__main__.py#L53
3. check why it (appears to) use(s) 3 here https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/_compatibility.py#L490-L499

[blueyed]

@bradwood
What is :py3 import sys; print(sys.version_info) in Vim?
It appears that your Vim only has Python 3.
What does "inside a virtualenv" mean? Vim's Python should be the same there, so what is different inside/outside a Virtualenv? (i.e. python -V)

[dyno]

# x.py
import jedi_vim
environment = jedi_vim.get_environment(use_cache=False)
sys_path = environment.get_sys_path()

:py3f % will produce the problem on my system. exactly as @bradwood posted.

py3 print(sys.version)
3.7.1 (default, Nov  6 2018, 18:46:03) 
[Clang 10.0.0 (clang-1000.11.45.5)]

:version
VIM - Vi IMproved 8.1 (2018 May 18, compiled Dec 16 2018 05:28:49)
macOS version
Included patches: 1-600
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
   +python3 -python                           
...
  -L/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin -lpython3.7m -framework CoreFoundation  -lruby.2.5.3 -lobjc    

[dyno]

(Pdb) environment._subprocess
<CompiledSubprocess _executable='/usr/local/bin/vim', _pickle_protocol=2, is_crashed=False, pid=73486>

(Pdb) self
<SameEnvironment: 3.7.1 in /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7>

(Pdb) self.get_sys_path
<bound method Environment.get_sys_path of <SameEnvironment: 3.7.1 in /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7>>
(Pdb) self.get_sys_path()
*** _pickle.UnpicklingError: invalid load key, '3'.
(Pdb) import pickle
(Pdb) pickle.HIGHEST_PROTOCOL
4
(Pdb) pickle.DEFAULT_PROTOCOL
3

https://github.com/davidhalter/jedi/blob/master/jedi/evaluate/compiled/subprocess/__init__.py#L142
interesting comments

    # Start with 2, gets set after _get_info.
    _pickle_protocol = 2

[blueyed]

What is self._subprocess._pickle_protocol?
From this I would assume it should uses 2 here:
https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/evaluate/compiled/subprocess/__init__.py#L229

What is used / comes here?
https://github.com/davidhalter/jedi/blob/3bdb941daa2d87947fa265ebcf7d94f68efbc42d/jedi/evaluate/compiled/subprocess/__main__.py#L55

(I suggest writing debug info to a temporary file, and tail -fing it)

[blueyed]

pickle.load states that the protocol should be detected, so this indicates that the error is from the py2 process.

[blueyed]

@dyno
Oh, no py2 involved for you (directly)?!
I've tried your example script, and it works fine for me, also using 3.7.1.

[blueyed]

I also works for me using :let jedi#use_environment = 'python2'.
It correctly picks 2 then in subprocess/__main__.py.
And it uses 4 wih py3.

[blueyed]

Just saw that everybody appears to use macOS version via Homebrew (8.1.500 or 8.1.600), which might indicate something.

[dyno]

Vim: neovim | vim8
OS: Linux | Mac
Python: 3.6 | 3.7

neovim works all time.
it definitely has something to do with vim8 on Mac with python3.7 compiled in.
i did not test vim with 3.7 on linux as it is linked with 3.6.

$ otool -L /usr/local/Cellar/vim/8.1.0600/bin/vim | grep -i python
    /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python (compatibility version 3.7.0, current version 3.7.0)

[blueyed]

Closing as duplicate of https://github.com/davidhalter/jedi-vim/issues/870.

As mentioned in https://github.com/davidhalter/jedi-vim/issues/870#issuecomment-445987031.

[blueyed]

Not really the same, but certainly related - re-opening for now.

[dyno]

i think this is the problem.

for neovim or vim on linux,

environment._subprocess()
<CompiledSubprocess _executable='/usr/local/opt/python/bin/python3.7', _pickle_protocol=2, is_crashed=False, pid=17229>

for vim8:

<CompiledSubprocess _executable='/usr/local/bin/vim', _pickle_protocol=2, is_crashed=False, pid=73486>

the executable is identified as vim and it does not understand any thing you send it.

this can be easily identified,
:py3 print(sys.executable)

[dyno]

https://docs.python.org/3/library/sys.html#sys.executable

sys.executable
A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.

i think it is bug for brew installed vim.

[dyno]

a workaround is in your vimrc, change the executable
:py3 sys.executable='/usr/local/bin/python3'

[blueyed]

Ok, closing as duplicate then.
https://github.com/davidhalter/jedi-vim/issues/870

[dyno]

tried compile from vim source https://github.com/vim/vim/archive/v8.1.0600.tar.gz, and the executable is set to my compiled vim. it looks like a vim problem not brew.

[blueyed]

Might have started to happen only recently then.
Please consider reporting it to Vim - it is not a problem for me on Arch Linux. I am using Neovim by default, but have tested it with Vim also.
Might be helpful if you could git-bisect it in Vim - I recommend to use up ccache then.. ;)