Azure-pipelines-tasks: CondaEnvironment should use conda activate, not a manual PATH prepending

[asmeurer]

The CondaEnvironment uses a manual activate routine, that prepends the PATH and sets the conda environment variables https://github.com/Microsoft/vsts-tasks/blob/6fae1bdf535512ecf21495fabcdc777cb7fc1c0c/Tasks/CondaEnvironmentV1/conda_internal.ts#L129

However, this is insufficient, as conda also sources activation scripts that may be included with some packages. For my use-case this breaks the sage package from conda forge, which sets some sage environment variables (see https://github.com/conda-forge/sagelib-feedstock/blob/master/recipe/activate/activate.sh).

Instead of trying to do the activation manually, it should reuse conda's own logic for this and source conda's activate.sh.

[brcrista]

You're right, that's what the task should do. The issue is that each build step runs in its own process. So if we just ran activate, any environment state wouldn't persist in later steps.

We do have a way to pass environment between build steps, but it's something the task has to do. What we need to do is basically implement "sourcing" logic for the task, where we run activate and any environment context written by it or a delegated script gets picked up and propagated.

So this is a known limitation of the task and is on our backlog. It looks like the workaround you found is the right one though.

[brcrista]

For anyone else out there who has this issue, scripting is also an option. You just have a script where you run activate and then everything else you need. Or, you can still break a job up into individual steps (say you want that finer-grained passed / failed status), but they need to be in script and you need to re-run activate at the top of each one.

[asmeurer]

FWIW I've found it annoying even in other contexts that environment variables aren't shared across tasks.

[brcrista]

If it helps, think of it like launching programs from a shell. Each invocation spawns a subprocess. If you want to bounce back environment to the host process, you have to do so explicitly, like when you source a script in the shell.

[mtaron]

I also ran into this issue; it manifested as DLL load failures for numpy: Importing the multiarray numpy extension module failed.

This is because the path prepending done by the CondaEnvironment task is incomplete. activate actually adds a dozen paths to PATH. For myenv on Windows, these are:

• C:\ProgramData\Miniconda3\envs\myenv
• C:\ProgramData\Miniconda3\envs\myenv\Library\mingw-w64\bin
• C:\ProgramData\Miniconda3\envs\myenv\Library\usr\bin
• C:\ProgramData\Miniconda3\envs\myenv\Library\bin
• C:\ProgramData\Miniconda3\envs\myenv\Scripts
• C:\ProgramData\Miniconda3\envs\myenv\bin
• C:\ProgramData\Miniconda3
• C:\ProgramData\Miniconda3\Library\mingw-w64\bin
• C:\ProgramData\Miniconda3\Library\usr\bin
• C:\ProgramData\Miniconda3\Library\bin
• C:\ProgramData\Miniconda3\Scripts
• C:\ProgramData\Miniconda3\bin

The bolded path is the key to the numpy loading error. When it is set, the correct version of numpy is loaded without error.

A script that runs activate <env name> as the first step isn't actually a workaround for this issue. If you try this, you'll notice that the entire script exits immediately after the activate and no other commands are run. :'(

Here is what I had to do to work around this issue:

- job: myjob
  variables:
    CONDA_DEFAULT_ENV: myenv
    CONDA_EXE: $(CONDA)\Scripts\conda.exe
    CONDA_PREFIX: $(CONDA)\envs\myenv
    CONDA_PREFIX_1: $(CONDA)
    CONDA_PYTHON_EXE: $(CONDA)\python.exe

  steps:
    # The Conda Environment task does not activate the environment correctly,
    # so the path needs to be manually updated.
  - script: |
      echo '##vso[task.setvariable variable=PATH]$(CONDA_PREFIX);$(CONDA_PREFIX)\Library\mingw-w64\bin;$(CONDA_PREFIX)\Library\usr\bin;$(CONDA_PREFIX)\Library\bin;$(CONDA_PREFIX)\Scripts;$(CONDA_PREFIX)\bin;$(CONDA_PREFIX_1);$(CONDA_PREFIX_1)\Library\mingw-w64\bin;$(CONDA_PREFIX_1)\Library\usr\bin;$(CONDA_PREFIX_1)\Library\bin;$(CONDA_PREFIX_1)\Scripts;$(CONDA_PREFIX_1)\bin;$(PATH)'
    displayName: Add conda to path

    # We can now use conda commands
    # Use a script since CondaEnvironment does not support conda env commands
  - script: 'conda env create -f myenv.yml --quiet'
    displayName: Create conda environment

I actually didn't end up using the CondaEnvironment task at all since it doesn't support conda env create, but I'll file another issue for that.

[brcrista]

I hadn't noticed before, but it looks like you're right -- on Windows, activating the environment at the beginning of a script causes the rest of the script not to be executed. I tested this on Hosted macOS and Hosted Ubuntu also and they don't have this problem.

Thanks for sharing your workaround. Let me see what I can come up with here.

[brcrista]

@mtaron my mistake, you have to call activate <env> -- see https://github.com/conda/conda/issues/794

I'll admit, Batch scripting is one of my lesser strengths. I'll fix the doc page.

[jenshnielsen]

In the newest version of conda i.e. with updateConda: true and conda 4.6.3 this seems to also break loading of the sqlite3 dynamic library and in turn the whole of the python sqlite3 module.

[brcrista]

We've decided to deprecate the Conda Environment task for the next release (see #9573). Instead, we recommend invoking the conda CLI directly. The task made sense when we were still focused on the old "Web designer"-based build definitions that lean heavily on tasks. Today with YAML, script is much easier.

I created a guide with some YAML snippets here: https://aka.ms/pipelines-anaconda-guide. Feedback welcome!