Youcompleteme: Python Virtualenv detection seems to fail

[dbrgn]

Issue Prelude

• [x] I have read and understood YCM's CONTRIBUTING document.
• [x] I have read and understood YCM's CODE_OF_CONDUCT document.
• [x] I have read and understood YCM's README, especially the
  Frequently Asked Questions section.
• [x] I have searched YCM's issue tracker to find issues similar to the one I'm
  about to report and couldn't find an answer to my problem.
• [x] If filing a bug report, I have included the output of vim --version.
• [x] If filing a bug report, I have included the output of :YcmDebugInfo.
• [x] If filing a bug report, I have included the output of
  :YcmToggleLogs stderr.
• [x] If filing a bug report, I have included which OS (including specific OS
  version) I am using.
• [x] If filing a bug report, I have included a minimal test case that reproduces
  my issue.
• [x] I understand this is an open-source project staffed by volunteers and
  that any help I receive is a selfless, heartfelt _gift_ of their free time. I
  know I am not entitled to anything and will be polite and courteous.
• [x] I understand my issue may be closed if it becomes obvious I didn't
  actually perform all of these steps.
  

  Issue Details

GoTo seems to work with the Python stdlib, but in a Virtualenv it seems to fail.

First of all, my versions:

• Vim 7.4 (details)
• YCM 21a0019
• ArchLinux

Steps to reproduce:

$ virtualenv -p /usr/bin/python3.5 VIRTUAL
$ . VIRTUAL/bin/activate
$ pip install django==1.9.4
$ echo "from django.views.generic import TemplateView" > test.py

Now open test.py and run :YcmCompleter GoTo when hovering over the TemplateView word.

Everything works when I run Jedi directly:

$ PYTHONPATH=~/.vim/bundle/YouCompleteMe/third_party/jedi/ python
Python 3.5.1 (default, Mar  3 2016, 09:29:07) 
[GCC 5.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jedi
>>> jedi.set_debug_function()
>>> s = jedi.Script('from django.views.generic import TemplateView', 1, 42, '')
speed: init 2.4557113647460938e-05
>>> s.goto_definitions()
speed: parsed 9.574442625045776
dbg: start: TemplateView in <fast.Module: @1-1>
dbg: follow_module None django
dbg: search_module django 
dbg: follow_module /tmp/VIRTUAL/lib/python3.5/site-packages/django views
dbg: follow_module /tmp/VIRTUAL/lib/python3.5/site-packages/django/views generic
dbg: follow_module /tmp/VIRTUAL/lib/python3.5/site-packages/django/views/generic TemplateView
dbg: _follow_path: TemplateView in scope <fast.Module: generic@1-22>
dbg: finder.filter_name "TemplateView" in (<fast.Module: generic@1-22>-<fast.Module: generic@1-22>): [<Name: TemplateView@1,52>]@None
dbg: finder._names_to_types: [<Name: TemplateView@1,52>], old: [<Import: from django.views.generic.base import TemplateView @1,0>]
dbg: follow_module None django
dbg: search_module django /tmp/VIRTUAL/lib/python3.5/site-packages/django/views/generic
dbg: follow_module /tmp/VIRTUAL/lib/python3.5/site-packages/django views
dbg: follow_module /tmp/VIRTUAL/lib/python3.5/site-packages/django/views generic
dbg: follow_module /tmp/VIRTUAL/lib/python3.5/site-packages/django/views/generic base
dbg: follow_module /tmp/VIRTUAL/lib/python3.5/site-packages/django/views/generic/base.py TemplateView
dbg: _follow_path: TemplateView in scope <fast.Module: base@1-222>
dbg: finder.filter_name "TemplateView" in (<fast.Module: base@1-222>-<fast.Module: base@1-222>): [<Name: TemplateView@151,6>]@None
dbg: finder._names_to_types: [<Name: TemplateView@151,6>], old: [<eClass of <Class: TemplateView@151-160>>]
dbg: after import: [<eClass of <Class: TemplateView@151-160>>]
dbg: after import: [<eClass of <Class: TemplateView@151-160>>]
[<Definition class TemplateView>]

I patched gotodefinition() in jedihttp/handlers.py to also enable that debug output.

...
def printer(level, msg):
    logger.error('DEBUG: ' + msg)
jedi.set_debug_function(printer)
logger.error( 'Definitions: %r' % script.goto_definitions() )
...

This is the output:

ERROR:jedihttp.handlers:Script: <Script: '/tmp/test.py'>
ERROR:jedihttp.handlers:Script source: u'from django.views.generic import TemplateView\n'
ERROR:jedihttp.handlers:Script path: '/tmp/test.py'
ERROR:jedihttp.handlers:DEBUG: dbg: Parsed /tmp/test.py, with 1 parsers in 1 splits.
ERROR:jedihttp.handlers:DEBUG: speed: parsed 0.0263788700104
ERROR:jedihttp.handlers:DEBUG: dbg: start: u'TemplateView' in <fast.FastModule: test@1-2>
ERROR:jedihttp.handlers:DEBUG: speed: import (<Name: django@1,5>, <Name: views@1,12>, <Name: generic@1,18>) 0.027046918869
ERROR:jedihttp.handlers:DEBUG: dbg: search_module 'django' in '/tmp/test.py'
ERROR:jedihttp.handlers:DEBUG: warning: /tmp/test.py:1:5: E3 Potential ImportError.
ERROR:jedihttp.handlers:DEBUG: speed: import (<Name: django@1,5>, <Name: views@1,12>, <Name: generic@1,18>, <Name: TemplateView@1,33>) 0.0281128883362
ERROR:jedihttp.handlers:DEBUG: dbg: search_module 'django' in '/tmp/test.py'
ERROR:jedihttp.handlers:DEBUG: warning: /tmp/test.py:1:5: E3 Potential ImportError.
ERROR:jedihttp.handlers:DEBUG: dbg: after import: []
ERROR:jedihttp.handlers:Definitions: []
ERROR:jedihttp.handlers:DEBUG: dbg: start: u'TemplateView' in <fast.FastModule: test@1-2>
ERROR:jedihttp.handlers:DEBUG: speed: import (<Name: django@1,5>, <Name: views@1,12>, <Name: generic@1,18>) 0.0297708511353
ERROR:jedihttp.handlers:DEBUG: dbg: search_module 'django' in '/tmp/test.py'
ERROR:jedihttp.handlers:DEBUG: warning: /tmp/test.py:1:5: E3 Potential ImportError.
ERROR:jedihttp.handlers:DEBUG: speed: import (<Name: django@1,5>, <Name: views@1,12>, <Name: generic@1,18>, <Name: TemplateView@1,33>) 0.0305018424988
ERROR:jedihttp.handlers:DEBUG: dbg: search_module 'django' in '/tmp/test.py'
ERROR:jedihttp.handlers:DEBUG: warning: /tmp/test.py:1:5: E3 Potential ImportError.
ERROR:jedihttp.handlers:DEBUG: dbg: after import: []
ERROR:jedihttp.handlers:DEBUG: speed: init 2.31266021729e-05
ERROR:jedihttp.handlers:DEBUG: dbg: Parsed /tmp/test.py, with 0 parsers in 1 splits.
ERROR:jedihttp.handlers:DEBUG: speed: parsed 0.000869035720825
ERROR:jedihttp.handlers:DEBUG: speed: import (<Name: django@1,5>, <Name: views@1,12>, <Name: generic@1,18>) 0.00120902061462
ERROR:jedihttp.handlers:DEBUG: dbg: search_module 'django' in '/tmp/test.py'
ERROR:jedihttp.handlers:DEBUG: warning: /tmp/test.py:1:5: E3 Potential ImportError.
ERROR:jedihttp.handlers:DEBUG: speed: import (<Name: django@1,5>, <Name: views@1,12>, <Name: generic@1,18>, <Name: TemplateView@1,33>) 0.00218605995178
ERROR:jedihttp.handlers:DEBUG: dbg: search_module 'django' in '/tmp/test.py'
ERROR:jedihttp.handlers:DEBUG: warning: /tmp/test.py:1:5: E3 Potential ImportError.
ERROR:jedihttp.handlers:DEBUG: dbg: after import: []

Here's :YcmDebugInfo:

Printing YouCompleteMe debug information...
-- Server has Clang support compiled in: False
-- JediHTTP running at 127.0.0.1:59402
--   python binary: /usr/bin/python2
--   stdout log: /tmp/ycm_temp/jedihttp_59402_stdout.log
--   stderr log: /tmp/ycm_temp/jedihttp_59402_stderr.log
-- Server running at: http://127.0.0.1:60980
-- Server process ID: 2292
-- Server logfiles:
--   /tmp/ycm_temp/server_60980_stdout.log
--   /tmp/ycm_temp/server_60980_stderr.log

And here's the YCM debug log: http://tmp.dbrgn.ch/ycmd-debug-log.txt

The problem probably has to do with the Jedi initialization.

[vheon]

Thanks for the detailed report! I don't have much time on my hand lately but I'll look into this as soon as I can if someone doesn't do it before me :)

Thanks again.

[dbrgn]

Thanks for the detailed report!

Well, that's what you ask for in your intimidating issue reporting requirements :smile: (No worries, I understand the reasons...)

PS: This used to work before. Problem started a few weeks ago, probably. But I might have used a quite old version of YCM. Maybe bisecting the repo would help?

[vheon]

PS: This used to work before. Problem started a few weeks ago, probably. But I might have used a quite old version of YCM. Maybe bisecting the repo would help?

No need. Is possible that you had an old version of YCM where we didn't use JediHTTP yet?

[dbrgn]

That is possible. When did you introduce it?

[micbou]

What you are experiencing is expected because JediHTTP is started with Python system (see the line python binary: /usr/bin/python2 in YcmDebugInfo output) instead of the Python from your virtual environment.

There are two ways to make JediHTTP uses your virtual environment:

• set the g:ycm_python_binary_path option in your vimrc to the Python virtualenv path;
• use the command :YcmCompleter RestartServer <path> where path is the Python virtualenv path. This will restart JediHTTP with the right Python interpreter.

When PR https://github.com/Valloric/ycmd/pull/429 is merged, you will only need to set g:ycm_python_binary_path to 'python' and the Python from PATH (which is the Python from virtualenv if virtualenv is activated) will be automatically picked.