Mybinder.org-deploy: ResolvePackageNotFound errors when building repository with `environment.yml`

[ncitron]

Trying to load a Binder with environment.yml file produces following error:

nda list -n root' returned a non-zero code: 1Waiting for build to start...
/usr/local/lib/python3.7/site-packages/repo2docker/utils.py:214: FutureWarning: Possible nested set at position 1815
""", re.VERBOSE)
Picked Git content provider.
Cloning into '/tmp/repo2docker9izsww0f'...
HEAD is now at 01a53eb Add files via upload
Building conda environment for python=3.7Using CondaBuildPack builder
Building conda environment for python=3.7Building conda environment for python=3.7Step 1/44 : FROM buildpack-deps:bionic
---> 4c2d54e8ddb0
Step 2/44 : ENV DEBIAN_FRONTEND=noninteractive
---> Using cache
---> 2c9f706bebc1
Step 3/44 : RUN apt-get -qq update && apt-get -qq install --yes --no-install-recommends locales > /dev/null && apt-get -qq purge && apt-get -qq clean && rm -rf /var/lib/apt/lists/*
---> Using cache
---> a8d26c1d1e6f
Step 4/44 : RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen
---> Using cache
---> 564c35304e31
Step 5/44 : ENV LC_ALL en_US.UTF-8
---> Using cache
---> 27a09dd44ae2
Step 6/44 : ENV LANG en_US.UTF-8
---> Using cache
---> 60867662d92b
Step 7/44 : ENV LANGUAGE en_US.UTF-8
---> Using cache
---> 137ad5167329
Step 8/44 : ENV SHELL /bin/bash
---> Using cache
---> 474bc59386ed
Step 9/44 : ARG NB_USER
---> Using cache
---> 3b179c7191f3
Step 10/44 : ARG NB_UID
---> Using cache
---> a421b8e34c3e
Step 11/44 : ENV USER ${NB_USER}
---> Using cache
---> 6dfdd3bca0ab
Step 12/44 : ENV HOME /home/${NB_USER}
---> Using cache
---> 15321d6ad7ef
Step 13/44 : RUN adduser --disabled-password --gecos "Default user" --uid ${NB_UID} ${NB_USER}
---> Using cache
---> 0673988a4fa0
Step 14/44 : RUN wget --quiet -O - https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && DISTRO="bionic" && echo "deb https://deb.nodesource.com/node_10.x $DISTRO main" >> /etc/apt/sources.list.d/nodesource.list && echo "deb-src https://deb.nodesource.com/node_10.x $DISTRO main" >> /etc/apt/sources.list.d/nodesource.list
---> Using cache
---> 1d6d15e415d1
Step 15/44 : RUN apt-get -qq update && apt-get -qq install --yes --no-install-recommends less nodejs unzip > /dev/null && apt-get -qq purge && apt-get -qq clean && rm -rf /var/lib/apt/lists/*
---> Using cache
---> 1d6f49bd7515
Step 16/44 : EXPOSE 8888
---> Using cache
---> 52a3226ef6be
Step 17/44 : ENV APP_BASE /srv
---> Using cache
---> 7520fd3f1655
Step 18/44 : ENV NPM_DIR ${APP_BASE}/npm
---> Using cache
---> 235421176310
Step 19/44 : ENV NPM_CONFIG_GLOBALCONFIG ${NPM_DIR}/npmrc
---> Using cache
---> 198590c85d56
Step 20/44 : ENV CONDA_DIR ${APP_BASE}/conda
---> Using cache
---> 70b458fae735
Step 21/44 : ENV NB_PYTHON_PREFIX ${CONDA_DIR}
---> Using cache
---> df92d180c9ef
Step 22/44 : ENV KERNEL_PYTHON_PREFIX ${NB_PYTHON_PREFIX}
---> Using cache
---> 061fa3891342
Step 23/44 : ENV PATH ${CONDA_DIR}/bin:${NPM_DIR}/bin:${PATH}
---> Using cache
---> 85a6e2281a48
Step 24/44 : COPY conda/install-miniconda.bash /tmp/install-miniconda.bash
---> Using cache
---> 6a9975b43525
Step 25/44 : COPY conda/environment.py-3.7.frozen.yml /tmp/environment.yml
---> Using cache
---> 9e269fb78b3f
Step 26/44 : RUN mkdir -p ${NPM_DIR} && chown -R ${NB_USER}:${NB_USER} ${NPM_DIR}
---> Using cache
---> 1768bd60ee01
Step 27/44 : USER ${NB_USER}
---> Using cache
---> 17994cba7af2
Step 28/44 : RUN npm config --global set prefix ${NPM_DIR}
---> Using cache
---> 3104b531bb70
Step 29/44 : USER root
---> Using cache
---> 0aadac0b82d1
Step 30/44 : RUN bash /tmp/install-miniconda.bash && rm /tmp/install-miniconda.bash /tmp/environment.yml
---> Using cache
---> efdd289f8648
Step 31/44 : ARG REPO_DIR=${HOME}
---> Using cache
---> 56814d46fd5d
Step 32/44 : ENV REPO_DIR ${REPO_DIR}
---> Using cache
---> ba277f643654
Step 33/44 : WORKDIR ${REPO_DIR}
---> Using cache
---> 3bd7da7ac851
Step 34/44 : ENV PATH ${HOME}/.local/bin:${REPO_DIR}/.local/bin:${PATH}
---> Using cache
---> 633352473280
Step 35/44 : USER root
---> Using cache
---> c75518d79d7c
Step 36/44 : COPY src/ ${REPO_DIR}
---> be8ef3c3c865
Step 37/44 : RUN chown -R ${NB_USER}:${NB_USER} ${REPO_DIR}
---> Running in d865e9a6dbda
Removing intermediate container d865e9a6dbda
---> 21d7a9758f36
Step 38/44 : USER ${NB_USER}
---> Running in ac96d51af19d
Removing intermediate container ac96d51af19d
---> c0457d70cf2b
Step 39/44 : RUN conda env update -n root -f "environment.yml" && conda clean -tipsy && conda list -n root
---> Running in ab6779b88a72
Solving environment: ...working... failed

ResolvePackageNotFound:

• python==3.7.1=haf84260_7
• libedit==3.1.20170329=hb402a30_2
• markupsafe==1.1.0=py37h1de35cc_0
• xz==5.2.4=h1de35cc_4
• pyzmq==17.1.2=py37h1de35cc_0
• tk==8.6.8=ha441bb4_0
• readline==7.0=h1de35cc_5
• ncurses==6.1=h0a44026_1
• libffi==3.2.1=h475c297_4
• mistune==0.8.4=py37h1de35cc_0
• sqlite==3.26.0=ha441bb4_0
• libsodium==1.0.16=h3efe00b_0
• tornado==5.1.1=py37h1de35cc_0
• zlib==1.2.11=h1de35cc_3
• openssl==1.1.1a=h1de35cc_0
• zeromq==4.2.5=h0a44026_1
• icu==58.2=h4b95b61_1
• libiconv==1.15=hdd342a3_7
• matplotlib==3.0.2=py37h54f8f79_0
• dbus==1.13.2=h760590f_1
• pcre==8.42=h378b8a2_0
• numpy==1.15.4=py37hacdab7b_0
• kiwisolver==1.0.1=py37h0a44026_0
• glib==2.56.2=hd9629dc_0
• freetype==2.9.1=hb4e5f40_0
• scipy==1.1.0=py37h1410ff5_2
• pandas==0.23.4=py37h6440ff4_0
• libgfortran==3.0.1=h93005f0_2
• mkl_fft==1.0.6=py37h27c97d8_0
• gettext==0.19.8.1=h15daf44_3
• sip==4.19.8=py37h0a44026_0
• libcxxabi==4.0.1=hcfea43d_1
• qt==5.9.7=h468cd18_1
• libcxx==4.0.1=hcfea43d_1
• numpy-base==1.15.4=py37h6575580_0
• libpng==1.6.35=ha441bb4_0
• expat==2.2.6=h0a44026_0
• appnope==0.1.0=py37_0
• pyqt==5.9.2=py37h655552a_2
• jpeg==9b=he5867d9_2
• mkl_random==1.0.2=py37h27c97d8_0

Removing intermediate container ab6779b88a72
The command '/bin/sh -c conda env update -n root -f "environment.yml" && conda clean -tipsy && conda list -n root' returned a non-zero code: 1

[choldgraf]

Hey there - it looks like there was a package not found in your list of packages. Can you provide a link to your binder repository so folks can debug? Also, can you confirm that your environment will build locally on your machine using conda env create ...?

[ncitron]

My environment builds on my machine perfectly. Heres a link to my binder: https://mybinder.org/v2/gh/noah7545/Galaxy-Evolution/master

[choldgraf]

I'm not sure if this is causing the problem, but I'd try specifying slightly less-specific dependencies. E.g. I've never seen environment.yml files with the py37h6440ff4_0 at the end...perhaps that's why conda isn't able to resolve those packages. Try starting with just the packages themselves, then add version numbers and see if it builds.

[ncitron]

Ill give that a try. Is there any reason my environment files has such specific versions? I generated them using: conda env export > environment.yml Could it have to do with my creating the environment using the anaconda navigator?

[minrk]

conda env export creates a strict export of all packages. This is the most robust for reproducibility, but it does bake in potential platform-specific packages, so you can only use an exported environment on the same platform. This looks like an env created on a mac (appnope is only a package on mac), which isn't portable to linux.

You can use repo2docker itself (the package that Binder uses to build the image) to run conda env export given a loose conda environment.yml. For example:

jupyter repo2docker https://github.com/binder-examples/conda conda env export -n root

will install the loose environment defined at https://github.com/binder-examples/conda and then export a strict environment that you can use.

[betatim]

conda env export --no-builds gives you an export which removes the platform specific tags. It doesn't help with the platform specific packages like appnope though.

I think the full procedure to get a "everything pinned" environment.yml that works with repo2docker/Binder would be:

1. jupyter repo2docker https://github.com/binder-examples/conda /bin/bash
2. conda install <packagesyouwant>
3. conda env export to get an environment file that will work with repo2docker/binder
4. place the environment.yml from step 3 in your original repository and launch it on mybinder.org

(did I forget something?)

We should work on automating those steps as a new repo2docker sub-command. I think there are issues related to that already.

[minrk]

I think a repo2docker freeze would be a great subcommand. We might need a few implementations of it depending on what inputs we want to accept (e.g. requirements.txt, environment.yml or hand-specified packages)

[betatim]

I added a FAQ entry to the repo2docker documentation that should answer this. It would be great if you could try to follow the guide and give feedback @noah7545.

Once that is merged we should close this issue.

[choldgraf]

@betatim this was also partially addressed here as well: https://github.com/jupyterhub/binder/issues/149

[betatim]

Closed by jupyter/repo2docker#598