Docker-stacks: Removal of Python 2 should be more documented, with a workaround

[guillaumerosinosky]

Hi

The removal of Python 2 in jupyter/scipy-notebook provoked lots of errors and astonishment in my scripts. I speak of the following pull requests : https://github.com/jupyter/docker-stacks/pull/432 and https://github.com/jupyter/docker-stacks/pull/433 .

As I probably won't be alone to have this problem, perhaps you could precise in the README.md a workaround (or your maybe planned derived image ?). For instance, I use jupyter/scipy-notebook:b4dd11e16ae4 instead.

Thank you for all the good work.

[parente]

To close this out: need an entry on the recipes page about how to add Python 2 back in via a child Dockerfile.

[parente]

Documented steps to add a Python 2.x environment here:

https://github.com/jupyter/docker-stacks/wiki/Docker-recipes#add-a-python-2x-environment

[doesnotexist]

The verbatim in the documentation does not work, at least not for me. With some slight modifications to the Dockerfile I got it to work. Here's what works for me:

```# Choose your desired base image
FROM jupyter/scipy-notebook:latest

Create a Python 2.x environment using conda including at least the ipython kernel

and the kernda utility.

RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython kernda ipykernel && \
conda clean -tipsy

USER root

Create a global kernelspec in the image and modify it so that it properly activates

the python2 conda environment.

RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && \
$CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json

USER $NB_USER
```

[rancas]

@doesnotexist I've built a docker using your indications and I am now able to run notebooks with python2.
Unfortunately, I've no scipy and pandas installed for Python 2.7 notebooks.
Can this be fixed or do I need to install 2.7 modules during docker build?

[doesnotexist]

@rancas I've don't really know much about conda, so I could be wrong, but I imagine you could just tack on any packages that are conda installable to the end of the conda create line, and it would install those packages for that version of python when its creating the env.

[parente]

Can this be fixed or do I need to install 2.7 modules during docker build?

You'll need to add any additional packages you want to use with the Python 2.x kernel into your dockerfile. I've updated the recipe at the bottom of https://github.com/jupyter/docker-stacks/wiki/Docker-recipes with a comment to this effect.

[fzhcary]

Parente: The script to add python2 back at https://github.com/jupyter/docker-stacks/wiki/Docker-recipes has errors. It complains about some package "qt" .. missing. Have you test it?

[fzhcary]

guillaumerosinosky : I can't agree with you more. The default python2 caused my python2 script lots of errors. Why remove python2? It took me hours to get this page....

[parente]

Parente: The script to add python2 back at https://github.com/jupyter/docker-stacks/wiki/Docker-recipes has errors. It complains about some package "qt" .. missing. Have you test it?

I did when it also had matplotlib in the sample which has since been removed. I've removed the --remove qt line from the recipe on the wiki to match @doesnotexist 's example above.

[mattshma]

when run conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython kernda ipykernel on Centos7+ 64bit, which maybe raise:

PackageNotFoundError: Package missing in current linux-64 channels:
  - kernda

so just replace it to conda create --quiet --yes -p $CONDA_DIR/envs/python2 -c conda-forge python=2.7 ipython kernda=0.2 ipykernel would be ok.

[sardbaba]

@parente the recipe should be updated as follow:

# ---------------- Create a Python 2.x environment using conda
# includes at least the ipython kernel and the kernda utility. Add any additional packages you want
# available for use in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.)
# @see https://github.com/jupyter/docker-stacks/wiki/Docker-recipes#add-a-python-2x-environment

RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 -c conda-forge python=2.7 ipython kernda=0.2 ipykernel && \
conda clean -tipsy

USER root

# Create a global kernelspec in the image and modify it so that it properly activates the python2 conda environment.

RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && \
$CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json

USER $NB_USER

Please note $CONDA_DIR/envs/python2/bin/kernda as well as the @mattshma directions.

Thanks

[ghost]

The Dockerfile recipes provided here and on the main page:

https://github.com/jupyter/docker-stacks/wiki/Docker-recipes#add-a-python-2x-environment

are still not letting me run jupyter notebooks with python2.x as the default. I still get python3, as confirmed by 'import sys; sys.version_info'.

What am I doing wrong???

I do notice the container has both python 2.x and 3.x installed. I even rewrote the Dockerfile and tried replacing the linked file '/opt/conda/bin/python' with the version sitting under '/usr/bin/python2'. But for some reason the change isn't actually happening in the image (a bit stumped on that at the moment).

Any thoughts about what i should try?

[archcra]

When I build the image, I got such error:

The command '/bin/sh -c conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython ipykernel kernda && conda clean -tipsy' returned a non-zero code: 1

Any help will be appreciated

[stijndehaes]

Workaround also stopped working for me with the following error:

Error: /opt/conda/envs/python2/bin does not contain a conda activate script

[MarcSaric]

Same here. The proposed Dockerfile

# Choose your desired base image
FROM jupyter/scipy-notebook:latest

# Create a Python 2.x environment using conda including at least the ipython kernel
# and the kernda utility. Add any additional packages you want available for use
# in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.)
RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython ipykernel kernda && \
    conda clean -tipsy

USER root

# Create a global kernelspec in the image and modify it so that it properly activates
# the python2 conda environment.
RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && \
$CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json

USER $NB_USER

leads to

marc@thinkpad:~/git/docker-stacks/crossmap-notebook$ docker build . -t marcsaric/crossmap-notebook
Sending build context to Docker daemon  3.584kB
Step 1/5 : FROM jupyter/scipy-notebook:latest
latest: Pulling from jupyter/scipy-notebook
22dc81ace0ea: Already exists 
1a8b3c87dba3: Already exists 
91390a1c435a: Already exists 
07844b14977e: Already exists 
b78396653dae: Already exists 
f6ed4d15e19b: Pull complete 
4b867dd0bba7: Pull complete 
2bc99ec6f992: Pull complete 
7037e70e2c06: Pull complete 
33568953b30a: Pull complete 
037d62f84cd0: Pull complete 
7440b3093c97: Pull complete 
1d38a9cc5843: Pull complete 
700a5fab71bf: Pull complete 
c3f13f09eda3: Pull complete 
156b0bab9f11: Pull complete 
ac1f56503f9d: Pull complete 
3817393dc2b6: Pull complete 
9f142b11681f: Pull complete 
facd77546042: Pull complete 
39c538d12c38: Pull complete 
32e3c7a1155f: Pull complete 
7b269e4196bf: Pull complete 
Digest: sha256:eea675aedc8b953a48fb35d3d93da82958246f12964f84069c2279bd6b0eb49a
Status: Downloaded newer image for jupyter/scipy-notebook:latest
 ---> de154b690dcf
Step 2/5 : RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython ipykernel kernda &&     conda clean -tipsy
 ---> Running in a623abe46333
Solving environment: ...working... done

## Package Plan ##

  environment location: /opt/conda/envs/python2

  added / updated specs: 
    - ipykernel
    - ipython
    - kernda
    - python=2.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    zlib-1.2.11                |                0          93 KB  conda-forge
    pyzmq-17.0.0               |           py27_4         991 KB  conda-forge
    simplegeneric-0.8.1        |           py27_0           6 KB  conda-forge
    backports.shutil_get_terminal_size-1.0.0|             py_3           7 KB  conda-forge
    pygments-2.2.0             |           py27_0         1.4 MB  conda-forge
    pexpect-4.5.0              |           py27_0          73 KB  conda-forge
    jupyter_client-5.2.3       |           py27_0         125 KB  conda-forge
    certifi-2018.4.16          |           py27_0         142 KB  conda-forge
    tornado-5.0.2              |           py27_0         633 KB  conda-forge
    enum34-1.1.6               |           py27_1          54 KB  conda-forge
    readline-7.0               |                0         838 KB  conda-forge
    kernda-0.2.1               |           py27_0           8 KB  conda-forge
    futures-3.2.0              |           py27_0          24 KB  conda-forge
    python-2.7.15              |                0        13.9 MB  conda-forge
    pickleshare-0.7.4          |           py27_0          11 KB  conda-forge
    ptyprocess-0.5.2           |           py27_0          21 KB  conda-forge
    ncurses-5.9                |               10         1.1 MB  conda-forge
    scandir-1.7                |           py27_0          48 KB  conda-forge
    pip-9.0.3                  |           py27_0         1.8 MB  conda-forge
    singledispatch-3.4.0.3     |           py27_0          12 KB  conda-forge
    wcwidth-0.1.7              |           py27_0          22 KB  conda-forge
    setuptools-39.1.0          |           py27_0         580 KB  conda-forge
    sqlite-3.20.1              |                2         1.3 MB  conda-forge
    ipython_genutils-0.2.0     |           py27_0          35 KB  conda-forge
    wheel-0.31.0               |           py27_0          61 KB  conda-forge
    ipykernel-4.8.2            |           py27_0         147 KB  conda-forge
    ipython-5.6.0              |           py27_0         1.0 MB  conda-forge
    prompt_toolkit-1.0.15      |           py27_0         332 KB  conda-forge
    backports_abc-0.5          |           py27_0           6 KB  conda-forge
    six-1.11.0                 |           py27_1          20 KB  conda-forge
    pathlib2-2.3.2             |           py27_0          30 KB  conda-forge
    traitlets-4.3.2            |           py27_0         126 KB  conda-forge
    tk-8.6.7                   |                0         3.1 MB  conda-forge
    backports-1.0              |           py27_1           4 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        27.9 MB

The following NEW packages will be INSTALLED:

    backports:                          1.0-py27_1       conda-forge
    backports.shutil_get_terminal_size: 1.0.0-py_3       conda-forge
    backports_abc:                      0.5-py27_0       conda-forge
    ca-certificates:                    2018.4.16-0      conda-forge
    certifi:                            2018.4.16-py27_0 conda-forge
    decorator:                          4.3.0-py_0       conda-forge
    enum34:                             1.1.6-py27_1     conda-forge
    futures:                            3.2.0-py27_0     conda-forge
    ipykernel:                          4.8.2-py27_0     conda-forge
    ipython:                            5.6.0-py27_0     conda-forge
    ipython_genutils:                   0.2.0-py27_0     conda-forge
    jupyter_client:                     5.2.3-py27_0     conda-forge
    jupyter_core:                       4.4.0-py_0       conda-forge
    kernda:                             0.2.1-py27_0     conda-forge
    libsodium:                          1.0.16-0         conda-forge
    ncurses:                            5.9-10           conda-forge
    openssl:                            1.0.2o-0         conda-forge
    pathlib2:                           2.3.2-py27_0     conda-forge
    pexpect:                            4.5.0-py27_0     conda-forge
    pickleshare:                        0.7.4-py27_0     conda-forge
    pip:                                9.0.3-py27_0     conda-forge
    prompt_toolkit:                     1.0.15-py27_0    conda-forge
    ptyprocess:                         0.5.2-py27_0     conda-forge
    pygments:                           2.2.0-py27_0     conda-forge
    python:                             2.7.15-0         conda-forge
    python-dateutil:                    2.7.2-py_0       conda-forge
    pyzmq:                              17.0.0-py27_4    conda-forge
    readline:                           7.0-0            conda-forge
    scandir:                            1.7-py27_0       conda-forge
    setuptools:                         39.1.0-py27_0    conda-forge
    simplegeneric:                      0.8.1-py27_0     conda-forge
    singledispatch:                     3.4.0.3-py27_0   conda-forge
    six:                                1.11.0-py27_1    conda-forge
    sqlite:                             3.20.1-2         conda-forge
    tk:                                 8.6.7-0          conda-forge
    tornado:                            5.0.2-py27_0     conda-forge
    traitlets:                          4.3.2-py27_0     conda-forge
    wcwidth:                            0.1.7-py27_0     conda-forge
    wheel:                              0.31.0-py27_0    conda-forge
    zeromq:                             4.2.5-1          conda-forge
    zlib:                               1.2.11-0         conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.3

Please update conda by running

    $ conda update -n base conda


Cache location: /opt/conda/pkgs
Will remove the following tarballs:

/opt/conda/pkgs
---------------
prompt_toolkit-1.0.15-py27_0.tar.bz2         332 KB
ipykernel-4.8.2-py27_0.tar.bz2               147 KB
python-2.7.15-0.tar.bz2                     13.9 MB
futures-3.2.0-py27_0.tar.bz2                  24 KB
backports.shutil_get_terminal_size-1.0.0-py_3.tar.bz2       7 KB
zlib-1.2.11-0.tar.bz2                         93 KB
pygments-2.2.0-py27_0.tar.bz2                1.4 MB
pathlib2-2.3.2-py27_0.tar.bz2                 30 KB
ptyprocess-0.5.2-py27_0.tar.bz2               21 KB
tornado-5.0.2-py27_0.tar.bz2                 633 KB
readline-7.0-0.tar.bz2                       838 KB
six-1.11.0-py27_1.tar.bz2                     20 KB
pexpect-4.5.0-py27_0.tar.bz2                  73 KB
pip-9.0.3-py27_0.tar.bz2                     1.8 MB
backports_abc-0.5-py27_0.tar.bz2               6 KB
traitlets-4.3.2-py27_0.tar.bz2               126 KB
pickleshare-0.7.4-py27_0.tar.bz2              11 KB
jupyter_client-5.2.3-py27_0.tar.bz2          125 KB
backports-1.0-py27_1.tar.bz2                   4 KB
ipython-5.6.0-py27_0.tar.bz2                 1.0 MB
simplegeneric-0.8.1-py27_0.tar.bz2             6 KB
ipython_genutils-0.2.0-py27_0.tar.bz2         35 KB
scandir-1.7-py27_0.tar.bz2                    48 KB
enum34-1.1.6-py27_1.tar.bz2                   54 KB
sqlite-3.20.1-2.tar.bz2                      1.3 MB
wcwidth-0.1.7-py27_0.tar.bz2                  22 KB
ncurses-5.9-10.tar.bz2                       1.1 MB
tk-8.6.7-0.tar.bz2                           3.1 MB
setuptools-39.1.0-py27_0.tar.bz2             580 KB
certifi-2018.4.16-py27_0.tar.bz2             142 KB
singledispatch-3.4.0.3-py27_0.tar.bz2         12 KB
pyzmq-17.0.0-py27_4.tar.bz2                  991 KB
kernda-0.2.1-py27_0.tar.bz2                    8 KB
wheel-0.31.0-py27_0.tar.bz2                   61 KB

---------------------------------------------------
Total:                                      27.9 MB

Removed prompt_toolkit-1.0.15-py27_0.tar.bz2
Removed ipykernel-4.8.2-py27_0.tar.bz2
Removed python-2.7.15-0.tar.bz2
Removed futures-3.2.0-py27_0.tar.bz2
Removed backports.shutil_get_terminal_size-1.0.0-py_3.tar.bz2
Removed zlib-1.2.11-0.tar.bz2
Removed pygments-2.2.0-py27_0.tar.bz2
Removed pathlib2-2.3.2-py27_0.tar.bz2
Removed ptyprocess-0.5.2-py27_0.tar.bz2
Removed tornado-5.0.2-py27_0.tar.bz2
Removed readline-7.0-0.tar.bz2
Removed six-1.11.0-py27_1.tar.bz2
Removed pexpect-4.5.0-py27_0.tar.bz2
Removed pip-9.0.3-py27_0.tar.bz2
Removed backports_abc-0.5-py27_0.tar.bz2
Removed traitlets-4.3.2-py27_0.tar.bz2
Removed pickleshare-0.7.4-py27_0.tar.bz2
Removed jupyter_client-5.2.3-py27_0.tar.bz2
Removed backports-1.0-py27_1.tar.bz2
Removed ipython-5.6.0-py27_0.tar.bz2
Removed simplegeneric-0.8.1-py27_0.tar.bz2
Removed ipython_genutils-0.2.0-py27_0.tar.bz2
Removed scandir-1.7-py27_0.tar.bz2
Removed enum34-1.1.6-py27_1.tar.bz2
Removed sqlite-3.20.1-2.tar.bz2
Removed wcwidth-0.1.7-py27_0.tar.bz2
Removed ncurses-5.9-10.tar.bz2
Removed tk-8.6.7-0.tar.bz2
Removed setuptools-39.1.0-py27_0.tar.bz2
Removed certifi-2018.4.16-py27_0.tar.bz2
Removed singledispatch-3.4.0.3-py27_0.tar.bz2
Removed pyzmq-17.0.0-py27_4.tar.bz2
Removed kernda-0.2.1-py27_0.tar.bz2
Removed wheel-0.31.0-py27_0.tar.bz2
Cache location: 
There are no unused packages to remove
source cache (/opt/conda/conda-bld/src_cache)
Size:                                           0 B

git cache (/opt/conda/conda-bld/git_cache)
Size:                                           0 B

hg cache (/opt/conda/conda-bld/hg_cache)
Size:                                           0 B

svn cache (/opt/conda/conda-bld/svn_cache)
Size:                                           0 B

Total:                                          0 B
Removing /opt/conda/conda-bld/src_cache
Removing /opt/conda/conda-bld/git_cache
Removing /opt/conda/conda-bld/hg_cache
Removing /opt/conda/conda-bld/svn_cache
Removing intermediate container a623abe46333
 ---> a240dbc22437
Step 3/5 : USER root
 ---> Running in 703fb56fa98d
Removing intermediate container 703fb56fa98d
 ---> ece2d87058e3
Step 4/5 : RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && $CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json
 ---> Running in 13fc05de7dc1
Installed kernelspec python2 in /usr/local/share/jupyter/kernels/python2
{u'display_name': u'Python 2', u'argv': [u'/opt/conda/envs/python2/bin/python', u'-m', u'ipykernel_launcher', u'-f', u'{connection_file}'], u'language': u'python'}
Error: /opt/conda/envs/python2/bin does not contain a conda activate script
The command '/bin/sh -c $CONDA_DIR/envs/python2/bin/python -m ipykernel install && $CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json' returned a non-zero code: 1

[parente]

conda 4.5 changed how environment activation behaves. I think the recipes page on the wiki is open to anyone to update. A quick workaround is to remove the kernda call (https://github.com/Valassis-Digital-Media/kernda/issues/10).