Elpy: Error "No such file or directory, virtualenv" when running elpy-config

[federicotdn]

Summary

After activating a virtual environment created with python -m venv env using pyvenv-activate, running M-x elpy-config no longer works, with the following messages:

Elpy is creating the RPC virtualenv... (This may take a while, but should only happen once in a while)
pyvenv-create: Searching for program: No such file or directory, virtualenv

I have always used Elpy without virtualenv, without any problems. I think this commit may have changed something there.

Steps to reproduce

1) Create virtual env using python3 -m venv env
2) Activate it in Emacs using pyvenv-activate
3) Run M-x elpy-config, without having virtualenv installed

My configuration

macOS Mojave 10.14.6
Python 3.7.4

Result of (elpy-config)

Not available.

Elpy configuration in my init.el

(elpy-enable)

[galaunay]

I did made some recent changes on how Eply's RPC works.

Elpy is now calling pyvenv-create to generate a virtualenv dedicated to the RPC.
As pyvenv-create relies on virtualenv, you need to have it installed.

I made a PR to pyvenv to avoid that issue in the future.
I also made a temporary fix in Elpy, so it should work now.

[kirk86]

This is not fixed!
Please roll back to previous functionality. IMHO if something works there's no need to modify or change it.

I have a conda env with virtualenv installed
I activate the env with pyvenv-activate from emacs and execute elpy-config

What are these RPC dependencies being installed and where are they installed? If it's on my virtual env then that's not a desirable behaviour at all.

[federicotdn]

@kirk86 another option is to install Elpy from MELPA Stable, if you do not wish to wait for a fix.

[galaunay]

@kirk86 This PR was necessary to avoid having to install Elpy's python dependencies in every virtualenvs you wanted to work in (see #1546 and the PR #1548).

I did as much tests as I could before merging, but there is apparently still some quirks.
Please use Melpa Stable if you want to roll back to the previous behaviour (I made a release just before this PR).

However, It would be nice if you could help me figure this bug out.
I will try conda env, to check if I can reproduce it.

What are these RPC dependencies being installed and where are they installed? If it's on my virtual env then that's not a desirable behaviour at all.

Elpy needs some python packages (jedi, yapf, black,...) to provide some features.
Now those packages are installed in a dedicated virtualenv (named elpy-rpc-venv).
Elpy is not installing anything in your virtualenv.

[kirk86]

However, It would be nice if you could help me figure this bug out.

I'm happy to help as much as I can, but I'm not an elisp expert.

This PR was necessary to avoid having to install Elpy's python dependencies in every virtualenvs you wanted to work in

Got it! What it would make sense to me is to leave explicit in the docs that elpy needs x,y,z packages to provide its functionality and allow the user to denote via a path or something the main env from where elpy should pick up those packages.

For instance, I use conda and my main conda env is very minimal with only the necessary stuff. For everything else I build their own dedicated conda envs with the necessary packages. In a scenario like this it would make sense as a user to install jedi, autopep, etc. in the main conda env and tell elpy to get the prerequisite from that env.

Now from inisde emacs whenever I do pyvenv env x, y, z elpy should append whatever functionality exists in the env x, y, z to the previous config from the main conda env.

Think of it like the PATH variable in shell, where we point it some/random/path and then we can simply append PATH=$PATH + 'another/random/path'

[federicotdn]

@kirk86 This PR was necessary to avoid having to install Elpy's python dependencies in every virtualenvs you wanted to work in (see #1546 and the PR #1548).

Interesting, personally I've always liked that Elpy installs all of its dependencies in the activated virtual environment. It keeps stuff much tidier IMO. This is specially true for projects that have e.g. flake8 as a dev dependency (pinned to a specific version) and use flake8 on a CI environment or on a Makefile recipe. In those cases all of the flymake warnings and errors are exactly the same you'd get running the CI script or the Makefile recipe.

I consider that the advantage of having your IDE use the exact same dev tools that your project specifies outweighs the disadvantage of having to set them up for every new virtual env (which happens only once per project, really).

Have you decided if you're going to maintain an option to keep the original behaviour?

[pbgc]

Don't know if its related .... but elpy stopped working for me .. and everything I do gives me this error message:

elpy-rpc-get-or-create-venv: Opening output file: No such file or directory, /Users/pbeck/.virtualenvs/elpy-rpc-venv/elpy-rpc-python-path-command

EDIT: Had to change to the version of elpa stable that works. I'm on MacOS 10.14.6 and Emacs 26.2

[pbgc]

Ok ... I'm fixed it. I was already using an virtualenv that I activated before launching emacs so elpy would install the dependencies on it. I stopped activating it and elpy started giving the message "installing dependencies" or something like that and now i have:

Elpy Configuration

Emacs.............: 26.2
Elpy..............: 1.31.0
Virtualenv........: elpy-rpc-venv (/Users/pbeck/.virtualenvs/elpy-rpc-venv)
Interactive Python: python3 3.6.4 (/usr/local/bin/python3)
RPC virtualenv....: elpy-rpc-venv (/Users/pbeck/.virtualenvs/elpy-rpc-venv)
Python...........: python3 3.6.4 (/Users/pbeck/.virtualenvs/elpy-rpc-venv/bin/python3)
Jedi.............: 0.15.1
Rope.............: 0.14.0
Autopep8.........: 1.4.4
Yapf.............: 0.28.0
Black............: 19.3b0
Syntax checker....: flake8 (/Users/pbeck/.virtualenvs/elpy-rpc-venv/bin/flake8)

BUT M-. (elpy-goto-definition) stopped working! (don't know if something else)

[kirk86]

I don't like this behaviour at all.
When I was installing from stable melpa I had in mind that I was getting elpy working as it used to be before any changes had been made to the package but I get the same behaviour as @pbgc with the virtualenv.

First of I don't like when programs clutter my home directory with .configx, .configy, .xyzconfig. There exist a .config directory in linux why people don't use that?

Second of all, now I'm in this case where I'll have to activate some virtualenv so that elpy can install those required packages.

Previously I had may default main conda environment with the prerequisite packages for elpy installed and not matter which file I was editing prerequisite packages were being picked up correctly by elpy-config.

Now I have this:

Jedi, Autopep8, etc, are not being picked up properly?

I want elpy with previous behaviour before the update, where should I look and how should I get it?

[pbgc]

I also preferred that there was some config option to say the virtualenv to use to get/install the dependencies!
That's what I was using before and worked like a charm! I just needed to activate that virtualenv before running emacs and had in init.el

`
(setq elpy-rpc-python-command "~/Development/emacs_vv3/bin/python3")

(setq flycheck-python-pycompile-executable "~/Development/emacs_vv3/bin/python3")

(setq flycheck-python-flake8-executable "~/Development/emacs_vv3/bin/python3")
`

[kirk86]

Thankfully I had an older version on my macbook and now I'm back on business as usual.

This is the behaviour that I prefer.
I install everything on my main conda default env and no mater which other env I activate (from inside emacs) or which packages are installed in it things work fine and prerequisite packages are being picked up properly. Exactly like in my previous thread of the example of $PATH variable.

[galaunay]

@federicotdn

Have you decided if you're going to maintain an option to keep the original behaviour?

I understand your concern, PR #1658 will allow to get the old behaviour back and run the RPC in the current virtualenv (I will put some more details in the PR description).

@pbgc

Don't know if its related .... but elpy stopped working for me .. and everything I do gives me this error message:
elpy-rpc-get-or-create-venv: Opening output file: No such file or directory, /Users/pbeck/.virtualenvs/elpy-rpc-venv/elpy-rpc-python-path-command

This should be fixed by #1658 as well.

BUT M-. (elpy-goto-definition) stopped working! (don't know if something else)

I forgot a dependency requirement when re-organizing files, it should be fixed by now.

@kirk86

I want elpy with previous behaviour before the update, where should I look and how should I get it?

Melpa stable will do (just tested now), if you are still seeing RPC virtualenv in elpy-config, it means you are still running the non-stable version. Maybe restarting Emacs will do ?

[galaunay]

PR #1658 will allow to customize the virtualenv used by the RPC with a new option elpy-rpc-venv-path.
It will allow to get back your workflow (by setting buffer-local values for this variable for example).

[federicotdn]

@galaunay looking forward to that! Thanks for the hard work on this library, I use it practically every day.

[maggo007]

For me when "virtualenv" is installed it is not possible to install anything in the virtual environment with pip. When first created by elpy it fails to install dependencies. In the config page I get this error when trying to install them.
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

When the virtual environment get created with "python -m venv" by removing "virtualenv" or by creating them in the command line everything works fine.

edit: seems to be a distro related problem
https://github.com/pypa/virtualenv/issues/1416

[galaunay]

@federicotdn Thanks, it's nice to know that people are enjoying Elpy :).

@maggo007 I just made a commit to use python -m venv in priority. We don't really need the virtualenv additional features just to create virtual environments.

[tviti]

A thought: does it make sense to warn the user that elpy is about to create a venv on its own?

Not that I'm really peeved by this, and understand the justification completely (this is exactly how I have my emacs-julia setup configured as well: with a dedicated julia env/package so that I don't have to worry about dependencies leaking into other projects). But it does seem like it'd be good to prompt the user that elpy is about to try to finagle with files outside of its own directory.

[galaunay]

But it does seem like it'd be good to prompt the user that elpy is about to try to finagle with files outside of its own directory.

I get your point, I updated PR #1658 to ask users before the creation of any virtualenv.

We could also create the venv in Elpy's directory or in a .emacs.d/elpy directory.
But it will make it more complicated for users to access this virtualenv from outside.

[sten0]

Hi @galaunay, Would you please add an option to elpy-rpc-virtualenv-path (and associated code paths) to skip venv creation and use system-wide packages?

[galaunay]

Hi @sten0,
I added it on the todo list.

It should work with (setq elpy-rpc-virtualenv-path "/usr") (or similar if your main python is somewhere else than /usr/bin/python)

[galaunay]

PR #1658 should have addressed most of the concerns raised here:

• elpy-rpc-virtualenv-path can now be customised to use the global environment, the currently activated virtual environment or a specific virtualenv of your choice.
• Elpy is now asking permission to create virtualenvs that are not in the default directory (.emacs.d/elpy/rpc-venv).
• Elpy now relies on venv to create virtualenv ( virtualenv doesn't need to be installed anymore).

If you want to get back the old Elpy behaviour, just set elpy-rpc-virtualenv-path to current.

I am closing this issue, feel free to re-open if there is still to discuss on those points.
Please open a new issue if there is problems following PR #1658.