Docker-stacks: Python 3.7 only in minimal-notebook, not in scipy-notebook
Hi, I noticed inconsistent Python versions across the docker stack.
What docker image you are using?
Example: jupyter/scipy-notebook
What complete docker command do you run to launch the container (omitting sensitive values)?
docker run -it jupyter/minimal-notebook:latest python --version
What steps do you take once the container is running to reproduce the issue?
petrucci ➜ docker-images git:(master) ✗ docker pull jupyter/minimal-notebook:latest
latest: Pulling from jupyter/minimal-notebook
Digest: sha256:1691f3f2472692db58f92319c893a91aa826b88ee9976693d47086dc16880979
Status: Image is up to date for jupyter/minimal-notebook:latest
petrucci ➜ docker-images git:(master) ✗ docker pull jupyter/scipy-notebook:latest
latest: Pulling from jupyter/scipy-notebook
Digest: sha256:24eff1eeafdc22f744e003771d1f40b97997b4593763a8ca2d4967b314f2879d
Status: Image is up to date for jupyter/scipy-notebook:latest
petrucci ➜ docker-images git:(master) ✗ docker run -it jupyter/minimal-notebook:latest python --version
Python 3.7.1
petrucci ➜ docker-images git:(master) ✗ docker run -it jupyter/scipy-notebook:latest python --version
Python 3.6.6
What do you expect to happen?
Images having the same Python version.
What actually happens?
Images have different Python version.
RafalSkolasinski
All 12 comments
It's possible there's been a change in how Docker Hub is decides whether to use cached parent images or pull anew. It looks like the last build of scipy-notebook used a cached local image instead of pulling the newest minimal-notebook that built immediately before it.
It's also possible something in the images is causing a downgrade of Python to 3.6, but that's less believable.
Needs more digging. I wish the Docker Hub build logs were public (https://cloud.docker.com/u/jupyter/repository/docker/jupyter/base-notebook/builds).
parente
on 28 Nov 2018
It's also possible something in the images is causing a downgrade of Python to 3.6, but that's less believable.
This is the case. I did a bit more manual test but forgot to update the issue and now I lost logs.
But the problem seemed to be related to h5py forcing downgrade.
RafalSkolasinski
on 28 Nov 2018
Looks like h5py was built for 3.7 upstream in conda-forge here: https://github.com/conda-forge/h5py-feedstock/pull/40. Might be some dependency is still 3.6 only or the 3.7 build never completed the build and push to the channel on the CI system. conda search does show the package is available for 3.7.
h5py 2.8.0 py37h7eb728f_3 conda-forge
h5py 2.8.0 py37h097b052_4 conda-forge
I tested it again, I run bash in jupyter/minimal-notebook and executed conda install by hand with packages from scipy-notebook.
This is the error message
UnsatisfiableError: The following specifications were found to be in conflict:
- h5py=2.7 -> python=3.5 -> readline=6.2
- h5py=2.7 -> python=3.5 -> sqlite=3.13
- h5py=2.7 -> python=3.5 -> tk=8.5
- python=3.7
Use "conda info <package>" to see the dependencies for each package.
Looks like an update to h5py to 2.8 is in order. Would you like to send the PR to change the version number here? https://github.com/jupyter/docker-stacks/blob/master/scipy-notebook/Dockerfile#L41
parente
on 28 Nov 2018
I will test locally if image builds with that change and if it fixes the problem I will open PR.
RafalSkolasinski
on 28 Nov 2018
Number seems to be a problem too, see
UnsatisfiableError: The following specifications were found to be in conflict:
- numba=0.38 -> llvmlite==0.23.0 -> enum34 -> python=2.7
- python=3.7
trying numba=0.40*
RafalSkolasinski
on 28 Nov 2018
Setting numba=0.40* remove the conflict but Python gets downgraded anyway.
I'd have to specify python=3.7 by hand to avoid the downgrade.
RafalSkolasinski
on 28 Nov 2018
Interesting. We may want to pin the Python version in the base-notebook image anyway using a pin file to avoid this kind of behavior and raise incompatibilities as errors at build time instead of allowing conda to downgrade Python.
parente
on 28 Nov 2018
https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile#L82 is an example of the pinning we do for tini. I'm thinking we could do the same for Python.
parente
on 28 Nov 2018
Permalink for above pinning-for-tini reference, which had gone stale:
javabrett
on 20 Mar 2019
The associated PR #778 has the continuation of the discussion here and link to the upstream work that is happening in conda-forge to release tensorflow 1.13 which is blocking the upgrade of python to 3.7 in tensorflow-notebook. scipy-notebook currently has 3.7 and probably has for a bit now (https://github.com/jupyter/docker-stacks/wiki/scipy-notebook-59b402ce701d).
I'm going to close this issue out and let the PR live on as the spot for us to track the TF upgrade.
parente
on 31 Mar 2019
Related issues
iramsey85
·
4Comments
tarelli
·
4Comments
MridulS
·
4Comments
ainiml
·
3Comments
aar0nTw
·
4Comments
Most helpful comment
Interesting. We may want to pin the Python version in the base-notebook image anyway using a pin file to avoid this kind of behavior and raise incompatibilities as errors at build time instead of allowing conda to downgrade Python.