Readthedocs.org: Conda build not working with python 3.6

[wheerd]

Details

I am trying to build my python 3.6 only project's documentation using a conda environment. However, this fails because the package requirements are not satisfiable for 3.6. Locally my docs build fine with python 3.6. Not sure if that is because the package python versions are not updated on conda or because of the hardcoded version for some of the packages.

• Project URL: http://readthedocs.org/projects/multiset/
• Build URL: http://readthedocs.org/projects/multiset/builds/4880183/
• environment.yml: https://github.com/wheerd/multiset/blob/master/environment.yml
• Read the Docs username: Wheerd

Actual Result

Conda cannot find a working constallation for python 3.6:

The following specifications were found to be in conflict:
  - alabaster >=0.7,<0.8,!=0.7.5 -> python 2.7*|3.4*|3.5*
  - certifi (target=certifi-2016.9.26-py36_0.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*|3.5*|3.6*
  - docutils ==0.12 -> python 2.7*|3.4*|3.5*
  - mock -> python 2.7*|3.4*|3.5*
  - pillow ==3.0.0 -> python 2.7*|3.4*|3.5*
  - pip (target=pip-9.0.1-py36_1.tar.bz2) -> python 2.7*|3.4*|3.5*
  - pip (target=pip-9.0.1-py36_1.tar.bz2) -> setuptools
  - pygments ==2.1.1 -> python 2.7*|3.4*|3.5*
  - python 3.6*
  - setuptools (target=setuptools-32.3.1-py36_0.tar.bz2) -> certifi
  - setuptools (target=setuptools-32.3.1-py36_0.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*|3.5*|3.6*
  - sphinx ==1.3.5 -> python 2.7*|3.4*|3.5*
  - sphinx_rtd_theme ==0.1.7 -> python 2.7*|3.4*|3.5*
  - wheel (target=wheel-0.29.0-py36_0.tar.bz2) -> python 2.7*|3.3*|3.4*|3.5*|3.6*

[willingc]

We saw the same issue on Jupyter docs and temporarily have pinned python to 3.5 in environment.yml. See https://github.com/jupyter/nbconvert/pull/506 for temporary fix by @michaelpacer.

Issue also filed on conda https://github.com/conda/conda/issues/4168.

[wheerd]

I am afraid that will not work for me, because I use autdoc and my project does not work under 3.5 anymore. I guess those are the downsides of trying to use the latest and greatest...

[willingc]

@wheerd Bummer... Perhaps a temporary work around for the autodoc would be a 2 step build: 1) locally build the source files that use autodoc and save those. 2) Use the locally generated files as input source files. We did that in the early days of the notebook docs. Not a great long term strategy but ok in a pinch. If I see any cleaner solutions, I'll post here.

[wheerd]

I think I will have to wait for python 2.5.3 which is due to arrive in a week and fixes the typing module. I guess the docs will have to wait until then to get updated. But thanks for your help anyways @willingc :)
Getting python 3.6 support with conda on rtd would be nice though.

[jakirkham]

Could try using conda-forge. Many things have already been rebuilt for Python 3.6. This amounts to adding the code block below to your environment.yml file.

channels:
  - conda-forge

[willingc]

@jakirkham I will double check our files. We do have conda-forge on some of our Jupyter repos 👍

[jakirkham]

If you do find there are Python 3.6 packages missing from conda-forge, it is pretty easy to fix. One merely needs to re-render the feedstock in question. Though we definitely have Sphinx on Python 3.6. If you need help configuring the Jupyter repos for RTD, would be happy to chip in.

[wheerd]

I am already using conda-forge. However, the problem ist that RTD locks the sphinx version to 1.3.5 for which there is no package at all in conda-forge and no python 3.6 package in the normal channel apparently. The same is probably true for the other packages that are version locked.

And since there is not python 3.5.3 package yet, I cannot use that either. I have already create an issue over there to get that version bumped (conda-forge/python-feedstock#110).

[willingc]

Yep, I went backed and looked at some of the Jupyter repos even those with the conda-forge channel were failing. @jakirkham Thoughts?

[jakirkham]

However, the problem ist that RTD locks the sphinx version to 1.3.5...

That's a pretty old version. Where in the logs are you seeing this getting pinned?

[jakirkham]

Yep, I went backed and looked at some of the Jupyter repos even those with the conda-forge channel were failing. @jakirkham Thoughts?

Could you please share a log from one of the failures? Feel free to email it if you prefer.

[willingc]

@jakirkham Here's an example from this morning.

RTD Build (Failed): https://readthedocs.org/projects/jupyter-client/builds/4913160/

jupyter_client environment.yml (PR change to pin to 3.5): https://github.com/jupyter/jupyter_client/commit/74cde1ff6963

RTD Build (Passed): https://readthedocs.org/projects/jupyter-client/builds/4913426/

Second step in the RTD build process has the failure.

[willingc]

Related PR to fix this issue https://github.com/rtfd/readthedocs.org/pull/2589

[willingc]

Also PR https://github.com/rtfd/readthedocs.org/pull/2378

Likely the best solution would be to automate the tests/builds when new sphinx versions are released.

[jakirkham]

So Python 3.5.3 is being built, which hopefully helps, @wheerd.

In general, we (conda-forge) could try to build Python 3.6 packages for all of these old versions that RTD is using. Though maybe we would be better off to work with RTD to get them to upgrade to newer versions. Found an issue ( https://github.com/rtfd/readthedocs.org/issues/2548 ) to bump the Sphinx version to 1.5, which would help.

[wheerd]

@jakirkham thanks for the help, that should get my docs building again. But yes, I guess the best solution for building docs with python 3.6 docs would be to update the RTD dependencies to newer versions.

[pcattori]

Perhaps a temporary work around for the autodoc would be a 2 step build: 1) locally build the source files that use autodoc and save those. 2) Use the locally generated files as input source files. We did that in the early days of the notebook docs.

@willingc How would I upload the generated files to rtd? I am also on a project that uses python 3.6 features.

[humitos]

Finally, the Sphinx 1.5.3 version was added to conda-forge. So, now there are two more issues opened regarding this topic that when they get closed, this issue you are having will _disappear_ :)

Those are #2713 and #2714. I'm adding them here in case you want to subscribe also.

[cdeil]

We're having the same issue. We're starting a Python 3.6 or later package, would like to install dependencies with conda, and can't get around this issue:

UnsatisfiableError: The following specifications were found to be in conflict:
  - python 3.6*
  - sphinx_rtd_theme ==0.1.7 -> python 3.5* -> xz 5.0.5
Use "conda info <package>" to see the dependencies for each package.

https://readthedocs.org/projects/hips/builds/5477266/
https://github.com/hipspy/hips
If a workaround exists (tell RTD to not try to install these old versions of Sphinx and sphinx_rtd_theme at all), please let us know.

[humitos]

@cdeil there is no a workaround for this but this week we did this PR #2876 that removes the pinned versions for conda environments so it shouldn't conflict with your own packages.

I think it will be deployed next week. So, please try again after that. Thank you for your report.

[cdeil]

@humitos - thank you very much!!!

The point of using conda is that it gives you something that's independent of system config. So I was very confused when my RTD conda builds failed trying to conda install stuff that I never asked for (like the old version sphinx_rtd_theme ==0.1.7 mentioned above).

What exactly does #2876 do? Does it remove the conda install command from the build that was failing for me in https://readthedocs.org/projects/hips/builds/5477266/ ?

Is it simple to reproduce the RTD build / check locally if it will work now? I.e. are there a few docker / other commands listed somewhere that I could use to check if my project will now build after your change?

[humitos]

What exactly does #2876 do? Does it remove the conda install command from the build that was failing for me in https://readthedocs.org/projects/hips/builds/5477266/ ?

These are the steps used to create a project with conda environment:

1. conda env create -f user-environment.yml
2. conda install sphinx==1.5.3 rtd_sphinx_theme==0.1.7 ...
3. conda env update -f user-environment.yml

Now, with #2876 we are doing: 1) and in 2) we are not pinning at all and 3) is not ran.`

2) has to be ran since RTD needs a couple of extra packages that the user doesn't need for their own project. We think this new way of execution will work better. So, in case the user has some conflicts with the packages that RTD needs, the user could pin the RTD packages in their own environment.yml

Is it simple to reproduce the RTD build / check locally if it will work now? I.e. are there a few docker / other commands listed somewhere that I could use to check if my project will now build after your change?

Yes, it's easy to reproduce how RTD works (without installing the whole project) by running the commands I mentioned or taking a look at your own build -there are all the commands ran.

Please, let me know any comment you have on this. Besides, if you have a better idea, share it :)

[cdeil]

@humitos - That's pretty good, thanks for this detailed infos!
I think it might even be nicer to have full control over the build, being able to e.g. provide a Dockerfile.

At the moment my main problem is the failing pip install in a conda build: #2827
I'll try to reproduce that locally now by copy & pasting the commands from the build log.
(but I'd be surprised if I can reproduce the issue locally ... why is no pip available???)

[cdeil]

@humitos - The problem I'm having is with the pip install command you generate and run on RTD. I'm not able to reproduce the issue locally: https://github.com/rtfd/readthedocs.org/issues/2827#issuecomment-304534964

Does this pip install command go away with your changes in #2876 ?
Could it make sense to activate the conda env?