Repo2docker: Installing from a Pipfile.lock failing to upgrade a dependency (certifi)

[consideRatio]

I tried to build a repo with a Pipfile.lock but failed due to certifi==2019.3.9 was installed already and uninstalling it failed, which was done in order to install certifi==2019.6.16.

I'm not confident on how to go about this, but step one would be to understand the issue better. Any input appreciated!

My understanding

repo2docker's PipfileBuildPack installs things things directly to the --system environment, which is the KERNEL_PYTHON_PREFIX environment associated with /srv/conda/envs/notebook, this is because the Pipfile buildpack is derived from the CondaBuildPack. So there is a foundation from Conda within here already. The issue arise because of these preinstalled packages, and pipenvs inability to upgrade.

Error logs

# running this command...

repo2docker https://github.com/Sunita76/z2jh-cost-estimator

# resulted in...

Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/cli/command.py", line 254, in install
[pipenv.exceptions.InstallError]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 1874, in do_install
[pipenv.exceptions.InstallError]:       keep_outdated=keep_outdated
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 862, in do_install_dependencies
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, False, failed_deps_queue, retry=False)
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting certifi==2019.6.16 (from -r /tmp/pipenv-02yzxcut-requirements/pipenv-dpvc8kc3-requirement.txt (line 1))', '  Using cached https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl', 'Installing collected packages: certifi', '  Found existing installation: certifi 2019.3.9']
[pipenv.exceptions.InstallError]: ["ERROR: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead toonly a partial uninstall."]
ERROR: ERROR: Package installation failed...

Full build log

Waiting for build to start...
Picked Git content provider.
Cloning into '/tmp/repo2dockerp9tbf4qu'...
HEAD is now at e28184a Autoformatting
Building conda environment for python=3.7Using PipfileBuildPack builder
Building conda environment for python=3.7Building conda environment for python=3.7Step 1/49 : FROM buildpack-deps:bionic
 ---> b8bbcf540d80
Step 2/49 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 1501c11a0276
Step 3/49 : RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> a816cdb791e2
Step 4/49 : RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
 ---> Using cache
 ---> e7aeb659e6fd
Step 5/49 : ENV LC_ALL en_US.UTF-8
 ---> Using cache
 ---> ecfc16b9bc62
Step 6/49 : ENV LANG en_US.UTF-8
 ---> Using cache
 ---> bac10ff78823
Step 7/49 : ENV LANGUAGE en_US.UTF-8
 ---> Using cache
 ---> b74304555afe
Step 8/49 : ENV SHELL /bin/bash
 ---> Using cache
 ---> 7937fced70bf
Step 9/49 : ARG NB_USER
 ---> Using cache
 ---> b5252df81044
Step 10/49 : ARG NB_UID
 ---> Using cache
 ---> caa4982d7d6c
Step 11/49 : ENV USER ${NB_USER}
 ---> Using cache
 ---> c6d1ba79efe9
Step 12/49 : ENV HOME /home/${NB_USER}
 ---> Using cache
 ---> 048df7aa5736
Step 13/49 : RUN adduser --disabled-password     --gecos "Default user"     --uid ${NB_UID}${NB_USER}
 ---> Using cache
 ---> ac5b1dc5afe4
Step 14/49 : RUN wget --quiet -O - https://deb.nodesource.com/gpgkey/nodesource.gpg.key |  apt-key add - &&     DISTRO="bionic" &&     echo "deb https://deb.nodesource.com/node_10.x $DISTRO main" >> /etc/apt/sources.list.d/nodesource.list &&     echo "deb-src https://deb.nodesource.com/node_10.x $DISTRO main" >> /etc/apt/sources.list.d/nodesource.list
 ---> Using cache
 ---> 0dba220fa193
Step 15/49 : RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends    less        nodejs        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> fbdbfd7a774c
Step 16/49 : EXPOSE 8888
 ---> Using cache
 ---> 667826aba3bb
Step 17/49 : ENV APP_BASE /srv
 ---> Using cache
 ---> e2571a094b71
Step 18/49 : ENV NPM_DIR ${APP_BASE}/npm
 ---> Using cache
 ---> b5df4b485570
Step 19/49 : ENV NPM_CONFIG_GLOBALCONFIG ${NPM_DIR}/npmrc
 ---> Using cache
 ---> 335ba886dd23
Step 20/49 : ENV CONDA_DIR ${APP_BASE}/conda
 ---> Using cache
 ---> d1f3c573b7c9
Step 21/49 : ENV NB_PYTHON_PREFIX ${CONDA_DIR}/envs/notebook
 ---> Using cache
 ---> 34173b4722a5
Step 22/49 : ENV KERNEL_PYTHON_PREFIX ${NB_PYTHON_PREFIX}
 ---> Using cache
 ---> 4bbc3f91c555
Step 23/49 : ENV PATH ${NB_PYTHON_PREFIX}/bin:${CONDA_DIR}/bin:${NPM_DIR}/bin:${PATH}
 ---> Using cache
 ---> 7298b99e58ef
Step 24/49 : COPY conda/install-miniconda.bash /tmp/install-miniconda.bash
 ---> Using cache
 ---> ac2b0407c343
Step 25/49 : COPY conda/activate-conda.sh /etc/profile.d/activate-conda.sh
 ---> Using cache
 ---> 01b3a2b4eb26
Step 26/49 : COPY conda/environment.py-3.7.frozen.yml /tmp/environment.yml
 ---> Using cache
 ---> 0077b68aca22
Step 27/49 : RUN mkdir -p ${NPM_DIR} && chown -R ${NB_USER}:${NB_USER} ${NPM_DIR}
 ---> Using cache
 ---> e79e13718aa7
Step 28/49 : USER ${NB_USER}
 ---> Using cache
 ---> 04b8217bac15
Step 29/49 : RUN npm config --global set prefix ${NPM_DIR}
 ---> Using cache
 ---> f52fc3cbc881
Step 30/49 : USER root
 ---> Using cache
 ---> 19eba3676127
Step 31/49 : RUN bash /tmp/install-miniconda.bash && rm /tmp/install-miniconda.bash /tmp/environment.yml
 ---> Using cache
 ---> 221d90b25e78
Step 32/49 : ARG REPO_DIR=${HOME}
 ---> Using cache
 ---> 8d6e22ee0a7f
Step 33/49 : ENV REPO_DIR ${REPO_DIR}
 ---> Using cache
 ---> e97a19785a49
Step 34/49 : WORKDIR ${REPO_DIR}
 ---> Using cache
 ---> 6141bb1c2095
Step 35/49 : ENV PATH ${HOME}/.local/bin:${REPO_DIR}/.local/bin:${PATH}
 ---> Using cache
 ---> bbc8f7179edc
Step 36/49 : ENV CONDA_DEFAULT_ENV ${KERNEL_PYTHON_PREFIX}
 ---> Using cache
 ---> 573d5ceeb000
Step 37/49 : USER root
 ---> Using cache
 ---> 45ddece5861f
Step 38/49 : COPY src/ ${REPO_DIR}
 ---> 83b8a90fcac9
Step 39/49 : RUN chown -R ${NB_USER}:${NB_USER} ${REPO_DIR}
 ---> Running in eb6029542da2
Removing intermediate container eb6029542da2
 ---> 7f6492872ef2
Step 40/49 : USER ${NB_USER}
 ---> Running in 0196245f7bb1
Removing intermediate container 0196245f7bb1
 ---> 3f8723dc6196
Step 41/49 : RUN ${KERNEL_PYTHON_PREFIX}/bin/pip install pipenv==2018.11.26
 ---> Running in d96c741339b9
Collecting pipenv==2018.11.26
  Downloading https://files.pythonhosted.org/packages/13/b4/3ffa55f77161cff9a5220f162670f7c5eb00df52e00939e203f601b0f579/pipenv-2018.11.26-py3-none-any.whl (5.2MB)
Collecting virtualenv-clone>=0.2.5 (from pipenv==2018.11.26)
  Downloading https://files.pythonhosted.org/packages/ba/f8/50c2b7dbc99e05fce5e5b9d9a31f37c988c99acd4e8dedd720b7b8d4011d/virtualenv_clone-0.5.3-py2.py3-none-any.whl
Requirement already satisfied: pip>=9.0.1 in /srv/conda/envs/notebook/lib/python3.7/site-packages (from pipenv==2018.11.26) (19.1)
Collecting virtualenv (from pipenv==2018.11.26)
  Downloading https://files.pythonhosted.org/packages/c4/9a/a3f62ac5122a65dec34ad4b5ed8d802633dae4bc06a0fc62e55fe3e96fe1/virtualenv-16.6.1-py2.py3-none-any.whl (2.0MB)
Requirement already satisfied: setuptools>=36.2.1 in /srv/conda/envs/notebook/lib/python3.7/site-packages (from pipenv==2018.11.26) (41.0.1)
Requirement already satisfied: certifi in /srv/conda/envs/notebook/lib/python3.7/site-packages (from pipenv==2018.11.26) (2019.3.9)
Installing collected packages: virtualenv-clone, virtualenv, pipenv
Successfully installed pipenv-2018.11.26 virtualenv-16.6.1 virtualenv-clone-0.5.3
Removing intermediate container d96c741339b9
 ---> e74655fd6fc1
Step 42/49 : RUN (cd . &&                     PATH="${KERNEL_PYTHON_PREFIX}/bin:$PATH"               pipenv install --ignore-pipfile --system --dev                 )
 ---> Running in 5c73212764b9
Installing dependencies from Pipfile.lock (698e66)…
An error occurred while installing certifi==2019.6.16 --hash=sha256:046832c04d4e752f37383b628bc601a7ea7211496b4638f6514d0e5b9acc4939 --hash=sha256:945e3ba63a0b9f577b1395204e13c3a231f9bc0223888be653286534e5873695 --hash=sha256:046832c04d4e752f37383b628bc601a7ea7211496b4638f6514d0e5b9acc4939 --hash=sha256:945e3ba63a0b9f577b1395204e13c3a231f9bc0223888be653286534e5873695! Will try again.
Ignoring importlib-resources: markers 'python_version < "3.7"' don't match your environment
An error occurred while installing pandas==0.24.2 --hash=sha256:071e42b89b57baa17031af8c6b6bbd2e9a5c68c595bc6bf9adabd7a9ed125d3b --hash=sha256:17450e25ae69e2e6b303817bdf26b2cd57f69595d8550a77c308be0cd0fd58fa --hash=sha256:17916d818592c9ec891cbef2e90f98cc85e0f1e89ed0924c9b5220dc3209c846 --hash=sha256:2538f099ab0e9f9c9d09bbcd94b47fd889bad06dc7ae96b1ed583f1dc1a7a822 --hash=sha256:366f30710172cb45a6b4f43b66c220653b1ea50303fbbd94e50571637ffb9167 --hash=sha256:42e5ad741a0d09232efbc7fc648226ed93306551772fc8aecc6dce9f0e676794 --hash=sha256:4e718e7f395ba5bfe8b6f6aaf2ff1c65a09bb77a36af6394621434e7cc813204 --hash=sha256:4f919f409c433577a501e023943e582c57355d50a724c589e78bc1d551a535a2 --hash=sha256:4fe0d7e6438212e839fc5010c78b822664f1a824c0d263fd858f44131d9166e2 --hash=sha256:5149a6db3e74f23dc3f5a216c2c9ae2e12920aa2d4a5b77e44e5b804a5f93248 --hash=sha256:627594338d6dd995cfc0bacd8e654cd9e1252d2a7c959449228df6740d737eb8 --hash=sha256:83c702615052f2a0a7fb1dd289726e29ec87a27272d775cb77affe749cca28f8 --hash=sha256:8c872f7fdf3018b7891e1e3e86c55b190e6c5cee70cab771e8f246c855001296 --hash=sha256:90f116086063934afd51e61a802a943826d2aac572b2f7d55caaac51c13db5b5 --hash=sha256:a3352bacac12e1fc646213b998bce586f965c9d431773d9e91db27c7c48a1f7d --hash=sha256:bcdd06007cca02d51350f96debe51331dec429ac8f93930a43eb8fb5639e3eb5 --hash=sha256:c1bd07ebc15285535f61ddd8c0c75d0d6293e80e1ee6d9a8d73f3f36954342d0 --hash=sha256:c9a4b7c55115eb278c19aa14b34fcf5920c8fe7797a09b7b053ddd6195ea89b3 --hash=sha256:cc8fc0c7a8d5951dc738f1c1447f71c43734244453616f32b8aa0ef6013a5dfb --hash=sha256:d7b460bc316064540ce0c41c1438c416a40746fd8a4fb2999668bf18f3c4acf1! Will try again.
An error occurred while installing plotly==4.0.0a2 --hash=sha256:2a1de761a95fb8689b22064a05485a1888739e9e7f0dc29eed2a8caab6f44463 --hash=sha256:dda6d5bcb6771fd4282ef2aebba6fc03914321720e3aedf7f490415f2f6cbdd6! Will try again.
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/cli/command.py", line 254, in install
[pipenv.exceptions.InstallError]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 1874, in do_install
[pipenv.exceptions.InstallError]:       keep_outdated=keep_outdated
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 862, in do_install_dependencies
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, False, failed_deps_queue, retry=False)
[pipenv.exceptions.InstallError]:   File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting certifi==2019.6.16 (from -r /tmp/pipenv-02yzxcut-requirements/pipenv-dpvc8kc3-requirement.txt (line 1))', '  Using cached https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl', 'Installing collected packages: certifi', '  Found existing installation: certifi 2019.3.9']
[pipenv.exceptions.InstallError]: ["ERROR: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead toonly a partial uninstall."]
ERROR: ERROR: Package installation failed...
Removing intermediate container 5c73212764b9
<class 'str'>
https://github.com/sunita76/z2jh-cost-estimator
The command '/bin/sh -c (cd . &&                     PATH="${KERNEL_PYTHON_PREFIX}/bin:$PATH"                      pipenv install --ignore-pipfile --system --dev                 )' returned a non-zero code: 1

[sgibson91]

Hi @consideRatio, I had a similar issue with another repo. The problem was that pip no longer supports the distutils package and I had to downgrade my pip installation to v10.* in order to work around this message. This is by no means an elegant or correct solution, but hopefully gives you a jumping off point 🙂

EDIT: This issue may help: https://github.com/pypa/pip/issues/5247

[sgibson91]

I don't know if pipfiles support similar arguments to pure pip, but the --ignore-installed flag seemed to work for others.

[consideRatio]

@sgibson91 ah, pipenv relies on pip under the hood. I'll look into the possibility of influencing pip using environment variables, as I cannot do it through pipenv I think. Or I'll look into that further...

---

I was happy to learn I could start a docker container from a build layer and debug the last run command live.

Within the container, just before the pipenv build fails, I did the following, and can conclude that the pipenv error is really a pip error under the hood...

erik@505e0f3c1900:~$ pip install --upgrade certifi
Collecting certifi
  Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)
     |████████████████████████████████| 163kB 6.8MB/s 
Installing collected packages: certifi
  Found existing installation: certifi 2019.3.9
ERROR: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

__Note, I got to this point by first:__

# try build, but fail
repo2docker https://github.com/Sunita76/z2jh-cost-estimator

# find the layer just before the failure that was successfully built

# start a container associated with this layer
docker run -it --rm 47dc505f2a37 bash

[consideRatio]

Aha!

@sgibson91 I verified that I could do the following while I could not if the environment variable wasn't set!

PIP_IGNORE_INSTALLED=1 pip install --upgrade certifi

[consideRatio]

The question then becomes... What is the fallout of adding the --ignored-installed flag, if we do it for repo2docker? Hmmm...

[consideRatio]

I'll add it, run all tests, and see if they still succeed to start with.

[betatim]

I thought pipenv creates a new virtualenv for you in which "nothing" is installed unless specified in the Pipfile which would mean we don't have to deal with uninstalling thing :-/

[sgibson91]

@betatim I think distutils and certifi initially come from the Python installation. So if you want to add a package that has a different dependency on certifi than your core Python install, and you're using pip >= 10.* to install that package, then you come across this problem.

[consideRatio]

@betatim while pipenv install would, pipenv install --system won't, as we do in the PipfileBuildPack.

I think it is correct to do it like this as well, just like PythonBuildPack does not first create a virtual env to install things but instead uses the KERNEL environments pip to install it directly there. This environment is bootstrapped by CondaBuildPack. I think pipenv should do the same, and this issue is really just an issue of finding ourselves with an explicit environment that typical requirements.txt environments didn't have.

[consideRatio]

@sgibson91 excellent summary!

[minrk]

The root of this one was missing metadata in the certifi conda package, which should be fixed now. In general, conda packages shouldn't lack pip metadata, but some old and/or low-level packages do due to the requirement to bootstrap packages (certifi cannot depend on pip because pip depends on certifi). This should always be considered a bug in the conda packaging. A future refreeze of repo2docker ought to resolve the direct cause of this particular bug report, but the general issue of fully frozen environments having conflicts with the 'base' environment is an ongoing one with no easy answer.