conda env export omits pip packages (again)

Created on 24 Jan 2019  路  3Comments  路  Source: conda/conda

Current Behavior


conda env export omits pip install'ed packages. I can see this has come up before. This time it's on conda 4.6.0. I tried changing the pip version to 10.*, 18.1, 18.0, but this did not fix the issue.

Even when exporting with the -vv flag there are no informative outputs printed to the log. The failure to include pip appears to be completely silent.

Steps to Reproduce


I test using conda-forge and and the base channels. First conda-forge:

(base) 09:36:54 login27:~/python/neuroconda$ conda create -n test-forge -c conda-forge python=3.6
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /imaging/local/software/centos7/anaconda3/envs/test-forge

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/linux-64::ca-certificates-2018.11.29-ha4d7672_0
  certifi            conda-forge/linux-64::certifi-2018.11.29-py36_1000
  libffi             conda-forge/linux-64::libffi-3.2.1-hf484d3e_1005
  libgcc-ng          conda-forge/linux-64::libgcc-ng-7.3.0-hdf63c60_0
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-7.3.0-hdf63c60_0
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  openssl            conda-forge/linux-64::openssl-1.0.2p-h14c3975_1002
  pip                conda-forge/linux-64::pip-18.1-py36_1000
  python             conda-forge/linux-64::python-3.6.7-hd21baee_1001
  readline           conda-forge/linux-64::readline-7.0-hf8c457e_1001
  setuptools         conda-forge/linux-64::setuptools-40.6.3-py36_0
  sqlite             conda-forge/linux-64::sqlite-3.26.0-h67949de_1000
  tk                 conda-forge/linux-64::tk-8.6.9-h84994c4_1000
  wheel              conda-forge/linux-64::wheel-0.32.3-py36_0
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h14c3975_1004


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test-forge
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) 09:37:38 login27:~/python/neuroconda$ conda activate test-forge
(test-forge) 09:38:20 login27:~/python/neuroconda$ pip install --no-deps pybids
Collecting pybids
Installing collected packages: pybids
Successfully installed pybids-0.7.0
(test-forge) 09:38:50 login27:~/python/neuroconda$ conda env export -f test.yml
(test-forge) 09:39:22 login27:~/python/neuroconda$ cat test.yml
name: test-forge
channels:
  - conda-forge
  - defaults
  - r
dependencies:
  - ca-certificates=2018.11.29=ha4d7672_0
  - certifi=2018.11.29=py36_1000
  - libffi=3.2.1=hf484d3e_1005
  - libgcc-ng=7.3.0=hdf63c60_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - ncurses=6.1=hf484d3e_1002
  - openssl=1.0.2p=h14c3975_1002
  - pip=18.1=py36_1000
  - python=3.6.7=hd21baee_1001
  - readline=7.0=hf8c457e_1001
  - setuptools=40.6.3=py36_0
  - sqlite=3.26.0=h67949de_1000
  - tk=8.6.9=h84994c4_1000
  - wheel=0.32.3=py36_0
  - xz=5.2.4=h14c3975_1001
  - zlib=1.2.11=h14c3975_1004
prefix: /imaging/local/software/centos7/anaconda3/envs/test-forge
(test-forge) 09:46:10 login27:~/python/neuroconda$ conda list pybids
# packages in environment at /imaging/local/software/centos7/anaconda3/envs/test-forge:
#
# Name                    Version                   Build  Channel
pybids                    0.7.0                    pypi_0    pypi

Then base:

(base) 09:49:44 login27:~/python/neuroconda$ conda create -n test-base -c base python=3.6
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /imaging/local/software/centos7/anaconda3/envs/test-base

  added / updated specs:
    - python=3.6


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2018.12.5  |                0         123 KB
    certifi-2018.11.29         |           py36_0         146 KB
    libedit-3.1.20181209       |       hc058e9b_0         188 KB
    ncurses-6.1                |       he6710b0_1         958 KB
    openssl-1.1.1a             |       h7b6447c_0         5.0 MB
    pip-18.1                   |           py36_0         1.8 MB
    python-3.6.8               |       h0371630_0        34.4 MB
    readline-7.0               |       h7b6447c_5         392 KB
    setuptools-40.6.3          |           py36_0         625 KB
    sqlite-3.26.0              |       h7b6447c_0         1.9 MB
    tk-8.6.8                   |       hbc83047_0         3.1 MB
    wheel-0.32.3               |           py36_0          35 KB
    zlib-1.2.11                |       h7b6447c_3         120 KB
    ------------------------------------------------------------
                                           Total:        48.8 MB

The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/linux-64::ca-certificates-2018.12.5-0
  certifi            pkgs/main/linux-64::certifi-2018.11.29-py36_0
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1a-h7b6447c_0
  pip                pkgs/main/linux-64::pip-18.1-py36_0
  python             pkgs/main/linux-64::python-3.6.8-h0371630_0
  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-64::setuptools-40.6.3-py36_0
  sqlite             pkgs/main/linux-64::sqlite-3.26.0-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.32.3-py36_0
  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Proceed ([y]/n)? y


Downloading and Extracting Packages
openssl-1.1.1a       | 5.0 MB    | ######################################################################################################################################################################## | 100%
zlib-1.2.11          | 120 KB    | ######################################################################################################################################################################## | 100%
certifi-2018.11.29   | 146 KB    | ######################################################################################################################################################################## | 100%
setuptools-40.6.3    | 625 KB    | ######################################################################################################################################################################## | 100%
ca-certificates-2018 | 123 KB    | ######################################################################################################################################################################## | 100%
libedit-3.1.20181209 | 188 KB    | ######################################################################################################################################################################## | 100%
sqlite-3.26.0        | 1.9 MB    | ######################################################################################################################################################################## | 100%
ncurses-6.1          | 958 KB    | ######################################################################################################################################################################## | 100%
tk-8.6.8             | 3.1 MB    | ######################################################################################################################################################################## | 100%
python-3.6.8         | 34.4 MB   | ######################################################################################################################################################################## | 100%
readline-7.0         | 392 KB    | ######################################################################################################################################################################## | 100%
wheel-0.32.3         | 35 KB     | ######################################################################################################################################################################## | 100%
pip-18.1             | 1.8 MB    | ######################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test-base
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) 09:50:45 login27:~/python/neuroconda$ conda activate test-base
(test-base) 09:51:13 login27:~/python/neuroconda$ pip install --no-deps pybids
Collecting pybids
Installing collected packages: pybids
Successfully installed pybids-0.7.0
(test-base) 09:51:32 login27:~/python/neuroconda$ conda env export -f test.yml
(test-base) 09:51:57 login27:~/python/neuroconda$ cat test.yml
name: test-base
channels:
  - defaults
  - conda-forge
  - r
dependencies:
  - ca-certificates=2018.12.5=0
  - certifi=2018.11.29=py36_0
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - ncurses=6.1=he6710b0_1
  - openssl=1.1.1a=h7b6447c_0
  - pip=18.1=py36_0
  - python=3.6.8=h0371630_0
  - readline=7.0=h7b6447c_5
  - setuptools=40.6.3=py36_0
  - sqlite=3.26.0=h7b6447c_0
  - tk=8.6.8=hbc83047_0
  - wheel=0.32.3=py36_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
prefix: /imaging/local/software/centos7/anaconda3/envs/test-base
(test-base) 09:52:00 login27:~/python/neuroconda$ conda list pybids
# packages in environment at /imaging/local/software/centos7/anaconda3/envs/test-base:
#
# Name                    Version                   Build  Channel
pybids                    0.7.0                    pypi_0    pypi

Expected Behavior


pip installed packages should appear in '-pip:' sub category. If this functionality is broken for whatever reason conda should complain loudly about this when exporting.

Environment Information

conda info


(test-base) 09:54:19 login27:~/python/neuroconda$ conda info

     active environment : test-base
    active env location : /imaging/local/software/centos7/anaconda3/envs/test-base
            shell level : 3
       user config file : /home/jc01/.condarc
 populated config files : /home/jc01/.condarc
          conda version : 4.6.0
    conda-build version : 3.10.5
         python version : 3.6.5.final.0
       base environment : /imaging/local/software/centos7/anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/r/linux-64
                          https://conda.anaconda.org/r/noarch
          package cache : /imaging/local/software/centos7/anaconda3/pkgs
                          /home/jc01/.conda/pkgs
       envs directories : /imaging/local/software/centos7/anaconda3/envs
                          /home/jc01/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.0 requests/2.18.4 CPython/3.6.5 Linux/3.10.0-862.el7.x86_64 centos/7.5.1804 glibc/2.17
                UID:GID : 6303:50
             netrc file : None
           offline mode : False

conda config --show-sources


==> /home/jc01/.condarc <==
ssl_verify: False
channels:
  - conda-forge
  - defaults
  - r

conda list --show-channel-urls


# packages in environment at /imaging/local/software/centos7/anaconda3/envs/test-base:
#
# Name                    Version                   Build  Channel
ca-certificates           2018.12.5                     0    defaults
certifi                   2018.11.29               py36_0    defaults
libedit                   3.1.20181209         hc058e9b_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc-ng                 8.2.0                hdf63c60_1    defaults
libstdcxx-ng              8.2.0                hdf63c60_1    defaults
ncurses                   6.1                  he6710b0_1    defaults
openssl                   1.1.1a               h7b6447c_0    defaults
pip                       18.1                     py36_0    defaults
pybids                    0.7.0                    pypi_0    pypi
python                    3.6.8                h0371630_0    defaults
readline                  7.0                  h7b6447c_5    defaults
setuptools                40.6.3                   py36_0    defaults
sqlite                    3.26.0               h7b6447c_0    defaults
tk                        8.6.8                hbc83047_0    defaults
wheel                     0.32.3                   py36_0    defaults
xz                        5.2.4                h14c3975_4    defaults
zlib                      1.2.11               h7b6447c_3    defaults

picture jooh

Most helpful comment

... And now that the 4.6.1 release is official, I tried the update again (from conda-forge). Same problem. However, it turns out that you can get pip export to work again if you turn on the new pip_interop_enabled flag:

(cbu_nipy_1_03dev) 02:34:09 login27:~/python/neuroconda$ conda config --set pip_interop_enabled False
(cbu_nipy_1_03dev) 02:34:15 login27:~/python/neuroconda$ conda env export -f test.yml
(cbu_nipy_1_03dev) 02:34:23 login27:~/python/neuroconda$ cat test.yml | grep pip
  - apipkg=1.5=py_0
  - pip=10.0.1=py36_0
(cbu_nipy_1_03dev) 02:34:39 login27:~/python/neuroconda$ conda config --set pip_interop_enabled True
(cbu_nipy_1_03dev) 02:34:44 login27:~/python/neuroconda$ conda env export -f test.yml
(cbu_nipy_1_03dev) 02:34:56 login27:~/python/neuroconda$ cat test.yml | grep pip
  - apipkg=1.5=py_0
  - pip=10.0.1=py36_0
  - pip:

Hope this is useful for someone.

picture jooh on 29 Jan 2019
👍6

All 3 comments

... However, downgrading to conda 4.5.9 restores normal pip export function. But broke my installation as detailed in #7788. I repaired it using the instructions there (removing # conda version lines from conda-meta/history). I then was able to conda update conda... to 4.5.12, which is now the latest available version on base and conda-forge. Anyway, exporting on 4.5.12 also works fine.

Which raises the question of how I ended up on 4.6.0 in the first place. I looked in the history file, and it seems conda update -n base conda put me on 4.6.0 on 16 January. So it looks like there was a conda-forge 4.6.0 release that got pulled. And that's where my problems started.

jooh picture jooh on 25 Jan 2019

... And now that the 4.6.1 release is official, I tried the update again (from conda-forge). Same problem. However, it turns out that you can get pip export to work again if you turn on the new pip_interop_enabled flag:

(cbu_nipy_1_03dev) 02:34:09 login27:~/python/neuroconda$ conda config --set pip_interop_enabled False
(cbu_nipy_1_03dev) 02:34:15 login27:~/python/neuroconda$ conda env export -f test.yml
(cbu_nipy_1_03dev) 02:34:23 login27:~/python/neuroconda$ cat test.yml | grep pip
  - apipkg=1.5=py_0
  - pip=10.0.1=py36_0
(cbu_nipy_1_03dev) 02:34:39 login27:~/python/neuroconda$ conda config --set pip_interop_enabled True
(cbu_nipy_1_03dev) 02:34:44 login27:~/python/neuroconda$ conda env export -f test.yml
(cbu_nipy_1_03dev) 02:34:56 login27:~/python/neuroconda$ cat test.yml | grep pip
  - apipkg=1.5=py_0
  - pip=10.0.1=py36_0
  - pip:

Hope this is useful for someone.

jooh picture jooh on 29 Jan 2019
👍6

thanks! was trolling thru the source code trying to figure out how to re-enable this

kevlubkcm picture kevlubkcm on 29 Jan 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kadekutama picture kadekutama  路  3Comments
bunop picture bunop  路  3Comments
jkarimi91 picture jkarimi91  路  3Comments
mhlr picture mhlr  路  3Comments
ark00 picture ark00  路  3Comments