Asdf: [Feature Request] Managing named environments within `asdf` alike `pyenv-virtualenv`

[goranmoomin]

Firstly, I'm sorry if there is already a method to achieve named environments (explained below) in asdf in a simple way. This feature request is probably a duplicate of #167, but the issue was proposed/closed in 2017 and the feature didn't really gain much traction so I'm making this again here.

The motivation of the feature

Currently asdf allows installing different versions of the same interpreter, for example the v3.7 and v3.8 of python. A direct consequence of this approach (I'm not sure if it's intended or not...) is that it allows users to install different versions of packages (or does anything that touches the global state) in a non-interfering way. This capability is very, very useful when managing multiple projects that use the interpreter, since that means that one can carefully control the set of packages that are used in each project. It allows reproducible environments, or removal/cleanup of environments when they are not needed.

However as asdf primarily being a version manager, it only provides this isolation between two different versions of the same interpreter. For example, it doesn't allow have two instances of python v3.8 with different packages. As asdf already provides the groundwork of managing binary shims and providing isolation, it would be beneficial to provide the same capabilities in a more general method.

A detailed description of how the feature would work

The asdf-install command will be extended to accept a third argument, the environment name, as in asdf install [<name> <version> [<environment-name>]]. The environment name will default to the version.

When the asdf-install command is invoked, the directory ${asdf_data_dir}/installs/${plugin}/${environment_name} is created, and the usual process of invoking the plugin and building the environment should be executed. This can be (relatively) easily achieved by just passing the environment name instead of version when calling get_install_path.

The asdf-global, asdf-local, and asdf-shell should gain support for setting the environment name instead of a version, and the shim scripts would then allow launching from ${asdf_data_dir}/installs/${plugin}/${environment_name}/bin/, and other commands like asdf-list should also support the environments feature.

For example, let's say that a user is developing three python projects, one depending on 3.7.6 and the other two on 3.8.1.

The user could execute the following commands:

$ asdf install python 3.7.6 project-a
python-build 3.7.6 /Users/username/.asdf/installs/python/project-a
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.6.tar.xz...
-> https://www.python.org/ftp/python/3.7.6/Python-3.7.6.tar.xz
...
$ asdf install python 3.8.1 project-b
...
$ asdf install python 3.8.1 project-c
...
$ cd ./project-a && asdf local python project-a && cd ..
$ cd ./project-b && asdf local python project-b && cd ..
$ cd ./project-c && asdf local python project-c && cd ..
$ cd ./project-a && python --version && cd ..
Python 3.7.6
$ cd ./project-b && python --version && cd ..
Python 3.8.1

This would allow three python interpreters be installed on the system, located on ~/.asdf/installs/python/{project-a,project-b,project-c}.

[vic]

A somewhat related issue #523.

[vic]

Currently you can achieve what you want having symlinks like this:

ln -s ~/.asdf/plugins/python ~/.asdf/plugins/python-project-c

And on project c, your .tool-versions file:

python-project-c 3.8.1

Will try to read your proposal with more time tonight and give some feedback.

[denizdogan]

As a person that up until now used nvm and pyenv for lots of different concurrent projects, I would absolutely love to see this added to asdf.

[goranmoomin]

@vic ping...!

[danhper]

This is indeed very related to #523.
I think that this proposal is simple enough and provides pretty much everything needed for "alias" like behavior. Symlink would not allow to have twice the same version of a language so this is not just syntactic sugar but does add functionalities to asdf.
@asdf-vm/core What do you think?

[fxfitz]

I'm throwing in my +1 for this. I just switched to asdf from pyenv/nvm, and this is definitely something I'd like to see.

[gsevla]

I'm using asdf to switch between npm, python, yarn... but i really miss pyenv virtualenv. Right now, I'm working on two python 3.5.9 projects and they use different versions of django and other stuff. While pyenv can create multiple python 3.5.9 environments, I have no problem, but asdf python "only allow one python environment".

As @danhper said, symlinks doesn't work.

I believe aliases are a good solution.

Now going back about python, asdf python uses pyenv under the hood but it doesn't install https://github.com/pyenv/pyenv-virtualenv. Maybe add pyenv(.asdf/plugins/python/pyenv) to path and manually install pyenv-virtualenv can be a workaround. Still not tested, but I'll give it a shot at weekend.

[yujinyuz]

@gsevla Any news about trying pyenv-virtualenv with asdf?

[gsevla]

@yujinyuz Sorry. I tried but couldn't figure out why it didn't work. I Ended up coming back to pyenv (with virtualenv).

[yujinyuz]

@gsevla Thanks for the response! I ended up writing a simple script that works similar to pyenv virtualenv and the auto activating a venv if directory matches a venv name

[gsevla]

@gsevla Thanks for the response! I ended up writing a simple script that works similar to pyenv virtualenv and the auto activating a venv if directory matches a venv name

Nice! Thanks for share this idea.

[MaienM]

For those talking about integrating pyenv-virtualenv in asdf, I hacked together a plugin for this a while ago. I've never shared it as a proper plugin because it's quite hacky, but it's done its job well enough for me so far. It also (obviously) only helps those that want to use this for python.

If anyone's interested in trying it it can be found in my dotfiles: https://github.com/MaienM/dotfiles/tree/master/asdf/plugins/python-venv. Download that folder (don't forget the submodule) and drop it in $ASDF_DIR/plugins/. YMMV, no warranty, you know the drill.

You'd create a virtual env like this (install the used python version first if it isn't already):

ASDF_PYTHON_VERSION=3.8.1 asdf install python-venv name-of-env

You can then use name-of-env as a python version:

asdf local python name-of-env
ASDF_PYTHON_VERSION=name-of-env asdf exec python -V

To remove the environment uninstall it as normal (take care to use the python-venv plugin when uninstalling, not the python one as that wouldn't clean up everything properly):

asdf uninstall python-venv name-of-env

---

+1 on the proposal though, would be much nicer and more generic than this workaround.

[dagadbm]

i am just using asdf-direnv to manage this for now

[justinmayer]

I understand and completely agree that isolated environments are useful. And at least in Python’s case, that’s precisely what virtual environments are for — we don’t need asdf to re-invent them.

For example, if I understand this proposed flow correctly…

$ asdf install python 3.8.1 project-b
...
$ asdf install python 3.8.1 project-c
...

So the exact same interpreter version will be installed twice, one for each project? From my perspective, this is madness. Each interpreter takes up about 250 MB of disk space, so if you have a lot of environments, this adds up quickly. I currently have ~40 environments, which in the above-proposed case would take up 10 GB (!) of space, and that’s before installing anything into them! Virtual environments achieve the same isolation with a single interpreter and would use 97.5% less disk space than 40 entire Python interpreters.

So how would you use asdf with Python virtual environments, you ask? Below I provide two answers: (1) general case for Bash/Zsh folks, and (2) how I do it with Fish shell. (Both assume you’ve installed asdf and Python-compilation dependencies.)

Bash + Virtualenv

First, append the following to ~/.bashrc:

source ~/.asdf/asdf.sh
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/projects

Install the latest version of Python:

source ~/.bashrc
asdf plugin add python
asdf install python latest  # currently Python 3.8.5
asdf global python 3.8.5

Install Virtualenv (because it’s much faster than python -m venv […]):

python -m pip install virtualenv
asdf reshim python

Create a new environment:

virtualenv -p $(asdf where python 3.8.5)/bin/python $WORKON_HOME/project-a

Too much typing? Syntax too hard to remember? No problem. Append these two functions to ~/.bashrc:

mkenv(){
    virtualenv -p $(asdf where python "$1")/bin/python "$WORKON_HOME"/"$2"
}

workon(){
    source "$WORKON_HOME"/"$1"/bin/activate
    [ -d "$PROJECT_HOME"/"$1" ] && cd "$PROJECT_HOME"/"$1"
}

Let’s get ready to try the new functions:

source ~/.bashrc
rm -rf $WORKON_HOME/project-a
mkdir -p ~/projects/project-a ~/projects/project-b

Create a new environment and activate it:

mkenv 3.8.5 project-a
workon project-a
python --version
→ Python 3.8.5
deactivate

Install another version of Python, create a new environment using it, and activate it:

asdf install python 3.7.8
mkenv 3.7.8 project-b
workon project-b
python --version
→ Python 3.7.8
deactivate

And that’s all without getting fancy — it would be trivial to add other time-saving enhancements to these very basic Bash functions that I, someone who doesn’t even use Bash, wrote in five minutes.

(Why not use Virtualenvwrapper, you ask? I think it does too much, and in my limited testing, its PATH manipulation collided with asdf’s and yielded inscrutable errors.)

Fish + VirtualFish

Assuming you are fortunate enough to be a Fish shell fan (and already have it installed), you get an even-better workflow. Starting from the top…

Append to ~/.config/fish/config.fish:

set -x WORKON_HOME "$HOME/.virtualenvs"
set -x PROJECT_HOME "$HOME/projects"
set -g VIRTUALFISH_HOME $WORKON_HOME
set -g VIRTUALFISH_COMPAT_ALIASES "True"
set -g VIRTUALFISH_DEFAULT_PYTHON "$HOME/.asdf/installs/python/3.8.5/bin/python"
source ~/.asdf/asdf.fish

Install the latest version of Python:

exec fish
asdf plugin add python
asdf install python latest  # currently Python 3.8.5
asdf global python 3.8.5

Install VirtualFish (which uses Virtualenv under the hood):

python -m pip install virtualfish
asdf reshim python
vf install compat_aliases projects environment update_python && exec fish
mkdir $WORKON_HOME

Create new project & activate its environment, via asdf-specific support I added to VirtualFish:

vf project -p 3.8.5 project-a
python --version
→ Python 3.8.5
deactivate

Install another version of Python, create a new environment using it, and activate it:

asdf install python 3.7.8
vf project -p 3.7.8 project-b
python --version
→ Python 3.7.8
deactivate

Summary

Again, I understand the desire to have isolated environments, but since we already have great tools to provide them, I don’t think re-implementing them in asdf is warranted or wise. I like the fact that asdf is lean and does one thing very well. 😁

[goranmoomin]

@justinmayer I think you didn't catch the point. The point for this feature is that I can use asdf not only for Python, but for also Ruby, etc...

[justinmayer]

Oh, I understand the point. I just think we're better off using the already-existing environment management tooling at our disposal.

[goranmoomin]

@justinmayer That just leaves asdf meaningless... by you're logic we're better off using the already-existing version management tooling. We just want a central one, and asdf is perfect for that.

[justinmayer]

Yeahhhh... no. asdf is not meaningless. It does its job very well, and adding bloat by re-implementing the painstaking work that other tooling has already done would likely not be to asdf’s credit.

As a plugin — not core — I could understand providing syntactic hooks to VirtualFish, Virtualenv, Rbenv, etc., but anything further I would consider to be needless feature-creep.