Elpy: Elpy is not using python from my activated virtualenv

[marcwebbie]

I am having troubles when running either elpy-test or elpy-test-django-runner. For some reason they don't use the path from the virtualenv.

My config looks likes this after activating a virtualenv with pyvenv-workon:

Elpy Configuration

Virtualenv........: myapp (/Users/marc/.pyenv/versions/myapp)
RPC Python........: 3.4.3 (/Users/marc/.pyenv/versions/myapp/bin/python)
Interactive Python: ipython (/Users/marc/.pyenv/versions/myapp/bin/ipython)
Emacs.............: 24.4.1
Elpy..............: 1.7.1
Jedi..............: 0.8.1
Rope..............: 0.9.4-1
Importmagic.......: 0.1.3
Syntax checker....: flake8 (/Users/marc/.pyenv/versions/myapp/bin/flake8)

On my django projects:

I've set the command to be ./manage.py test --noinput a simillar error happens:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_utils'

For info:

I am on osx and system python is 2.7 I also use pyenv for versioning python. I've also set my workon dir like this:

(let ((workon-home (expand-file-name "~/.pyenv/versions")))
  (setenv "WORKON_HOME" workon-home)
  (setenv "VIRTUALENVWRAPPER_HOOK_DIR" workon-home))

[jorgenschaefer]

Hello, and thank you for the report!
A few questions :-)

What is the full output in the compilation buffer for the manage.py command?
What is the first line of manage.py? (Should be #!/usr/bin/env python)
What is the value of M-: (getenv "PATH")? (It should start with /Users/marc/.pyenv/versions/myapp/bin/)

a simillar error happens

You only show one error in your report, which error is it similar to?

Thanks!

[marcwebbie]

Hello @jorgenschaefer, thanks for the fast reply:

I've been reading a lot the code of pyvenv and elpy. They are both great packages but unfortunately pyvenv has very little documentation.

init.el:

(use-package elpy
      :bind (("C-c t" . elpy-test-django-runner)
             ("C-c C-f" . elpy-find-file))
      :init
      (elpy-enable)
      (defalias 'workon 'pyvenv-workon)
      :config
      (delete 'elpy-module-highlight-indentation elpy-modules)
      (delete 'elpy-module-flymake elpy-modules)
      (setq elpy-rpc-backend "jedi")
      (setq elpy-test-django-runner-command ("./manage.py" "test" "--noinput"))
      (elpy-use-ipython))

Elpy config when activated venv pyvenv-workon: myapp:

Virtualenv........:  (/Users/marc/.pyenv/versions/myapp/)
RPC Python........: 3.4.3 (/Users/marc/.pyenv/versions/myapp/bin/python)
Interactive Python: ipython (/Users/marc/.pyenv/versions/myapp/bin/ipython)
Emacs.............: 24.4.1
Elpy..............: 1.7.1
Jedi..............: 0.8.1
Rope..............: 0.9.4-1
Importmagic.......: 0.1.3
Syntax checker....: flake8 (/Users/marc/.pyenv/versions/myapp/bin/flake8)

Output after running django-test-runner:

-*- mode: compilation; default-directory: "~/Projects/myapp/" -*-
Compilation started at Sun Apr 19 15:18:05

./manage.py test --noinput
Traceback (most recent call last):
  File "./manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
  File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 1, in <module>
    from django.utils.version import get_version
ImportError: No module named utils.version

Compilation exited abnormally with code 1 at Sun Apr 19 15:18:05

manage.py:

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

Start of output from (getenv "PATH"):

"/Users/marc/.pyenv/versions/myapp/bin:"

I've managed to get it running with:

(add-hook 'pyvenv-post-activate-hook
                '(lambda () (setq elpy-test-django-runner-command (list (executable-find "python") "manage.py" "test" "--noinput"))))

However I'd like to understand why the django runner is not finding the right python executable in this context.

[jorgenschaefer]

It looks like PATH is set correctly for Emacs (else executable-find would not find the correct path there), but it is not exported to subprocesses (else /usr/bin/env should find the same).

What is the value of M-: (getenv "PATH")?
Does it work if you do (setq compilation-environment '("PATH=/Users/marc/.pyenv/versions/myapp/bin/"))?

[marcwebbie]

Still no luck with:

(setq compilation-environment '("PATH=/Users/marc/.pyenv/versions/myapp/bin/"))

Output:

-*- mode: compilation; default-directory: "~/Projects/myapp/" -*-
Compilation started at Mon Apr 20 05:36:36

python manage.py test --noinput
Traceback (most recent call last):
  File "manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
  File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 1, in <module>
    from django.utils.version import get_version
ImportError: No module named utils.version

Output from (getenv "PATH"):

"/Users/marc/.pyenv/versions/myapp/bin:/Users/marc/.pyenv/shims/:/usr/local/bin/:/usr/local/sbin/:/usr/bin/:/bin/:/usr/sbin/:/sbin/:/opt/X11/bin/"

[jorgenschaefer]

I am severely confused! :-)

(executable-find "python") returns /Users/marc/.pyenv/versions/myapp/bin/python, right?

Does M-x compile RET echo $PATH show the same path as (getenv "PATH")?

[marcwebbie]

Now it shows what is happening:

the output of M-x compile RET echo $PATH RET:

/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/marc/.pyenv/versions/myapp/bin:/Users/marc/.pyenv/shims/:/usr/local/bin/:/usr/local/sbin/:/usr/bin/:/bin/:/usr/sbin/:/sbin/:/opt/X11/bin/:/usr/local/MacGPG2/bin/:/usr/local/Cellar/emacs/24.4/libexec/emacs/24.4/x86_64-apple-darwin14.1.0/

Any thoughts on how to set that to compile $PATH?

Thanks

[jorgenschaefer]

/Users/marc/.pyenv/versions/e4l/bin

Is this the path to your virtualenv, or is that a different one?

[marcwebbie]

yeah, Sorry, it should be /Users/marc/.pyenv/versions/myapp/bin. I was on another project when I ran that command.

edited comment above with right path to myapp

[jorgenschaefer]

Ah, ok. So this is what we know:

• pyvenv activates the virtualenv correctly. This is why completion works, as well as executable-find.
• When compile is run, something prepends /usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin to PATH

My suspicion would be that the latter is done by the shell, as compile uses a shell to execute the arguments. Alternatively, compile is doing this.

What is the value of compilation-environment for you?
And what is the value of shell-file-name?

Please do run this code:

(let ((buf (generate-new-buffer "*Test*")))
  (start-process "test" buf "env")
  (display-buffer buf))

It should show a buffer with lots of environment variables. Please verify that PATH starts with /Users/marc/.pyenv/versions/myapp/bin.

Then, run this code:

(let ((buf (generate-new-buffer "*Test*")))
  (start-process "test" buf shell-file-name "-c" "env)
  (display-buffer buf))

Does the value of PATH in the new buffer differ from the value of PATH in the old?

[marcwebbie]

Hello @jorgenschaefer

Yes. They differ:

(let ((buf (generate-new-buffer "*Test*")))
  (start-process "test" buf "env")
  (display-buffer buf))

Output (myapp env as first in PATH):

PATH=/Users/marc/.pyenv/versions/myapp/bin:/Users/marc/.pyenv/shims/:/usr/local/bin/:/usr/local/sbin/:/usr/bin/:/bin/:/usr/sbin/:/sbin/:/opt/X11/bin/:/usr/local/MacGPG2/bin/:/usr/local/Cellar/emacs/24.4/libexec/emacs/24.4/x86_64-apple-darwin14.1.0/

---

(let ((buf (generate-new-buffer "*Test*")))
  (start-process "test" buf "/bin/zsh" "-c" "env")
  (display-buffer buf))

Output:

PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/MacGPG2/bin:/Users/marc/.pyenv/versions/myapp/bin:/Users/marc/.pyenv/shims/:/usr/local/bin/:/usr/local/sbin/:/usr/bin/:/bin/:/usr/sbin/:/sbin/:/opt/X11/bin/:/usr/local/MacGPG2/bin/:/usr/local/Cellar/emacs/24.4/libexec/emacs/24.4/x86_64-apple-darwin14.1.0/

It seems that it is there where the issue is.

[jorgenschaefer]

Aha! :-)

You likely have a .bashrc or similar file that prepends those directories to PATH. Is this the case?

If so, please do move that PATH changing code to your .bash_profile instead, which is only loaded once, on login, not for every shell being started.

[marcwebbie]

Hello @jorgenschaefer, I actually have no .bashrc or .bash_profile since I use prezto. Well, at least not on my home directory. Do you have an idea on where I could be looking at?

[jorgenschaefer]

Oh, zsh! I do not know zsh at all, I'm afraid – you'll have to figure out what kind of per-session and per-call init files it uses. Maybe talk to the prezto people, too?

[ghost]

@marcwebbie if prezto keeps to the default order of evaluation of init files in zsh, then the thing to do to set PATH at the login shell only would be to set it from ~/.zprofile. However many zsh configurations change the initfile locations and order, so you'd need to check prezto's own documentation.

Shameless advertising follows: I might recommend http://grml.org/zsh if it proves too complicated to debug prezto. It is (like prezto and ohmyzsh) a case of using "someone else's config," but it's a great deal less ambitious in scope than those.

[marcwebbie]

Thanks for your help. I will keep this issue up to date as soon as I get more setup.

[IvanMalison]

Aha! :-)

You likely have a .bashrc or similar file that prepends those directories to PATH. Is this the case?

If so, please do move that PATH changing code to your .bash_profile instead, which is only loaded once, on login, not for every shell being started.

I disagree that setting the PATH environment variable in rc files is a bad practice. From the official zsh documentation (http://zsh.sourceforge.net/Intro/intro_3.html)

.zshenv is sourced on all invocations of the shell, unless the -f option is set. It should contain commands to set the command search path, plus other important environment variables. `.zshenv' should not contain commands that produce output or assume the shell is attached to a tty.

I've been doing this (deliberately) for quite some time and it seems to be causing this issue for me as well.

zsh may be different (philosophically speaking) than bash in this respect.

EDIT: your suggestion would essentially amount to moving these path setting commands to .zlogin or .zprofile but according to the page i linked above this is not something that should be done.

'.zlogin' is sourced in login shells. It should contain commands that should be executed only in login shells. '.zlogout' is sourced when login shells exit. '.zprofile' is similar to '.zlogin', except that it is sourced before '.zshrc'. '.zprofile' is meant as an alternative to '.zlogin' for ksh fans; the two are not intended to be used together, although this could certainly be done if desired. '.zlogin' is not the place for alias definitions, options, environment variable settings, etc.; as a general rule, it should not change the shell environment at all. Rather, it should be used to set the terminal type and run a series of external commands (fortune, msgs, etc).

[IvanMalison]

Stack overflow seems to agree with you (for bash) for essentially exactly the reason that it avoids the problem that has been encountered here:
http://superuser.com/questions/409186/environment-variables-in-bash-profile-or-bashrc

Some people seem to disagree, however:
http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html

[IvanMalison]

In addition to the custom PATH changes I was making in .zshrc, I also had to disable

# system-wide environment settings for zsh(1)
if [ -x /usr/libexec/path_helper ]; then
    eval `/usr/libexec/path_helper -s`
fi

which was in my /etc/zshenv

[jorgenschaefer]

I disagree that setting the PATH environment variable in rc files is a bad practice.

The problem here is that virtualenvs work by changing PATH so that your shell finds the programs in the virtualenv directory before any system-wide installations. If you override PATH (as opposed to change or append to) in your rc file, you essentially deactivate the current virtualenv in any subshell. Which means that you made your shell unusable for the purpose of running programs in virtualenvs, because _you made your shell deactivate the virtualenv._

You will encounter similar problems with other programs that try to use your shell to run anything in a virtualenv.

If you insist on overriding PATH on every shell invocation, you can solve the problem using other approaches, too. For example, you can customize elpy-test-*-runner-command to use the full path to the python program in your virtualenv (which means you have to change each of them every time you change the virtualenv). Alternatively, you can change shell-file-name to point to a different shell that does not break PATH for subshells.

I hope that clarifies the problem. :-)

[jorgenschaefer]

Oh, another idea from the zsh mailing list is to use a safeguard environment variable to prevent overriding PATH and other variables in subshells, e.g.

if [ -z "$ENVIRONMENT_IS_SET" ]
then
    PATH="..."
    ENVIRONMENT_IS_SET=true
fi

[IvanMalison]

I hope that clarifies the problem. :-)

Oh yeah I totally get what the issue is. I actually already created a stack overflow issue for it, because it seems that zsh should allow one override the definition of PATH in subshells.

Oh, another idea from the zsh mailing list is to use a safeguard environment variable to prevent overriding PATH and other variables in subshells, e.g.

if [ -z "$ENVIRONMENT_IS_SET" ]
then
PATH="..."
ENVIRONMENT_IS_SET=true
fi

Hah funnily enough this is exactly the approach that i arrived at on my own last night:

https://github.com/IvanMalison/dotfiles/commit/138a3bebd314a0248ad197309bdd67edb5e79d6a#diff-2c9f0b1f04d2bbef64e1983e3583d134R69

In addition to that, I also made an idempotent add_to_front_of_path command that is a no op if the path is already contained in the PATH variable.

I also had to use that same environment_is_set trick in my my /etc/zshenv to deal with the path_helper stuff that I mentioned above.

Perhaps we should add a section to the README suggesting this approach to zsh users?

[jorgenschaefer]

Perhaps we should add a section to the README suggesting this approach to zsh users?

The FAQ seems like a good place for this sort of thing, indeed :-)

[marcwebbie]

I seemed to get it working on my mac.

Actually I use use-package to set my shell on osx. My config was like this:

(use-package exec-path-from-shell
  :ensure t
  :if *is-a-mac*
  :init
  (exec-path-from-shell-initialize)
  (exec-path-from-shell-copy-env "PYTHONPATH")
  )

I've just changed (exec-path-from-shell-initialize) to the :config step:

(use-package exec-path-from-shell
  :ensure t
  :if *is-a-mac*
  :config
  (exec-path-from-shell-initialize)
  (exec-path-from-shell-copy-env "PYTHONPATH")
  )

Also I've set if which pyenv > /dev/null; then eval "$(pyenv init -)"; fi to zprofile instead of zshrc.

Now pyenv path is being prepend to the path on compile and elpy commands are being run as expected.

Thanks

[jorgenschaefer]

Very nice! Glad to hear things work now.

If you have any further problems, please do not hesitate to open a new issue. And enjoy Elpy! :-)

[pavopax]

This was a useful thread. I add some notes for posterity, but feel free to remove or let me know a better place to put this

another tip for conda users

make sure you DO NOT have this in your .emacs

;; DO NOT use this command, this will overwrite virtual envs (conda)
;; (setq python-shell-interpreter "~/miniconda3/bin/python")

the last was preventing my interactive python session from using the venv one

double check settings with M-x elpy-config as this thread suggests. RPC and interactive python should be the same, and both should show the virtual env python you desire

small tip. add to .emacs:

;; in "Virtual Envs" menu, show me conda envs
(setenv "WORKON_HOME" "~/miniconda3/envs")