Readthedocs.org: Ability to pin Sphinx and friends in a conda environment

[jakirkham]

Details

• Project URL: https://readthedocs.org/projects/dask-image/
• Build URL (if applicable): https://readthedocs.org/projects/dask-image/builds/6925299/
• Read the Docs username (if applicable): (same as GitHub)

Expected Result

We are currently pinning Sphinx with the expectation that RTD honors this pinning.

Actual Result

However it appears that RTD overrides our Sphinx pinning later, which is causing a different problem ( https://github.com/rtfd/readthedocs.org/issues/3769 ).

cc @humitos

[humitos]

In my comment https://github.com/rtfd/readthedocs.org/issues/3769#issuecomment-375014874 I mentioned that the problem is the order of the commands executed.

RTD first:

1. create the env using the yaml provided by the user
2. install the needed packages to build the documentation (mock pillow sphinx sphinx_rtd_theme)

The last step that installs these packages will UPGRADE them if they are already installed in the conda env.

I've been checking the option --no-update-dependencies (from conda install) but it seems that doesn't work as I want. So, at the moment I don't know how to do something like "ensure installed but not upgrade it".

/tmp ⌚ 12:06:29
$ conda install --no-update-dependencies sphinx          

The following NEW packages will be INSTALLED:

    packaging:                17.1-py27_0                     
    pyparsing:                2.2.0-py27hf1513f8_1            

[...]

The following packages will be UPDATED:

    sphinx:                   1.5.1-py27_0         conda-forge --> 1.7.1-py27_0    

The following packages will be SUPERSEDED by a higher-priority channel:

    asn1crypto:               0.24.0-py27_0        conda-forge --> 0.24.0-py27_0   
[...]

Proceed ([y]/n)? n

Exiting

(output cropped)

I don't have too much knowledge about conda, but it would be great if someone who knows it more help us here saying if there is an option that solves this.

Otherwise, we will need to think in another solution. For example, "remove that command" from the flow, which involves force the user to always add those default packages (sphinx, mock, pillow, rtd_sphinx_theme) to their YAML file, which I think it's not a good "solution".

cc @juanlu001, @willingc do you know if there is an option in conda install to ensure a package is installed (or install it of not) but do not upgrade it if it's already installed? Thanks

[jakirkham]

cc @kalefranz

[kalefranz]

"ensure installed but not upgrade it"

Until https://github.com/conda/conda/issues/6845 was filed, I hadn't recognized that we didn't cover that use case. Right now, unfortunately, there's no good way :(

[stsewd]

This is the relevant part of the code

https://github.com/rtfd/readthedocs.org/blob/673bfc7938d8d50d9bda39a62aed08ff41d481ed/readthedocs/doc_builder/python_environments.py#L333-L357

Where conda is installed first and then rtd installs its pip requirements

@humitos isn't an option pin the sphinx version like is done with the python environment?

https://github.com/rtfd/readthedocs.org/blob/673bfc7938d8d50d9bda39a62aed08ff41d481ed/readthedocs/doc_builder/python_environments.py#L219-L227

[stsewd]

Kind of related to #1364

[humitos]

@stsewd we decided to unpin them some time ago because we had incompatibility problems with conda. Check this PR https://github.com/rtfd/readthedocs.org/pull/2876

I think what we are doing now is correct, but we need conda to behaves like pip in that way of "ensure installed without upgrade"

I'm marking this as blocked on https://github.com/conda/conda/issues/6845 for now (when that feature gets implemented I think this will be fixed automatically in RTD without code changes)

If there are someone with a better solution for the general problem that works in these different scenarios, we should probably implement it --but for now, we don't have a better solution :(

[wkerzendorf]

@stsewd does this mean you can't pin sphinx at the moment (sorry am not quite understanding it)

[stsewd]

@wkerzendorf no, you can't. If you pin sphinx it will be updated by rtd later, this issue depends on https://github.com/conda/conda/issues/6845

[wkerzendorf]

That’s unfortunate- any idea for some work around. Could I revert to pip or does that also not work. Could RTD just change the order of when which conda is called

[stsewd]

@wkerzendorf you can do it with pip, this is only a conda problem.

[kalefranz]

https://github.com/conda/conda/issues/6845 should be resolved in conda 4.6 with https://github.com/conda/conda/pull/7291

[wkerzendorf]

Any idea when this is coming out?

[kalefranz]

Best estimate right now is to have 4.6 enter the beta stage in canary in six weeks, which would mean a release to defaults around six weeks after that.

Sent from my iPhone

On May 17, 2018, at 4:55 PM, Wolfgang Kerzendorf [e-mail] wrote:

Any idea when this is coming out?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[willingc]

@kalefranz I wonder what happens if you have Sphinx install in the pip section of the environment.yml. I'm on limited bandwidth right now or I would try it now.

[kalefranz]

I don't think we add any type of --upgrade flag when we shell out to pip for the environment.yml stuff. So my hunch is that it works as you want.

[humitos]

I'm marking this as blocked on conda/conda#6845 for now (when that feature gets implemented I think this will be fixed automatically in RTD without code changes)

This issue was closed in conda (implemented by https://github.com/conda/conda/pull/7291)

I suppose that we will need to wait for the next mini conda release that contains this feature and then:

• update our docker image with that conda version
• update our build servers with that docker image
• update our python code to call the conda command with the -S, --satisfied-skip-solve argument

[humitos]

Although the conda issue is already solved in code, it will be available in 4.6.0 which is unreleased at this time. I'd like to work on the PR to add/fix this once it's released.

[akchinSTC]

~Are there plans to update the RTD images to latest conda and build processes to include the skip upgrade flag now that conda/conda#6845 has closed ?~

+1 to updating the build process with the new flag.

[stsewd]

The new feature isn't released yet by conda https://github.com/conda/conda/blob/master/CHANGELOG.md#460-unreleased

[akchinSTC]

The new feature isn't released yet by conda https://github.com/conda/conda/blob/master/CHANGELOG.md#460-unreleased

My mistake, confused conda package manager versioning with Anaconda distro.

[humitos]

Even if we add -S to our conda commands, I think we won't get the desired behavior because packages will be skipped _only if all_ the dependencies are satisfied.

I do think that we need to add the -S argument but it won't solve all the situations, though. So, after using -S the user will need to install all the dependencies that RTD will install later (pinning sphinx to the one he/she wants) as a workaround to satisfy all the dependencies.

"At least", adding the -S gives us a chance for a workaround :) . Thoughts here?

[stsewd]

4.6.0 was released on January 15th https://github.com/conda/conda/blob/master/CHANGELOG.md#460-2019-01-15

We should update our dockerfile https://github.com/rtfd/readthedocs-docker-images/blob/dd04d480e02c09520d2994dab34338e22ecef7aa/Dockerfile#L14

[humitos]

I think that Miniconda was not released yet. I does not appear in the repo where we download the file: https://repo.anaconda.com/miniconda/

[chrizzFTD]

Installers for Miniconda 4.6.14 now available on https://repo.anaconda.com/miniconda/
Do we know what the next steps are? :p

[stsewd]

We need to update the docker file and test some projects with the new flag to see if nothing breaks or projects that were broken are fixed after that :)

[humitos]

I suppose that we will need to wait for the next mini conda release that contains this feature and then:

• update our docker image with that conda version

I create a PR for this at https://github.com/rtfd/readthedocs-docker-images/pull/102

• update our build servers with that docker image

Once the previous PR get merged, I can upgrade the testing image in our build servers.

If you want to help us to test this image, let me know and I will enable a flag on your project.

• update our python code to call the conda command with the -S, --satisfied-skip-solve argument

We need this PR merged and deployed as well: #5631

[humitos]

I'm marking this as blocked again :disappointed: . See https://github.com/rtfd/readthedocs.org/pull/5631#issuecomment-486724453

[stsewd]

https://github.com/conda/conda/issues/6845#issuecomment-444656635

[humitos]

Hi everybody!

This issue has been opened for a long time and we tried different ways to achieve it. Unfortunately, most of them failed and we found that conda does not have the same behavior that pip has regarding packages that are already installed in the environment.

Now, we deployed a new Feature Flag (CONDA_APPEND_CORE_REQUIREMENTS) to try to attack this problem and finally be able to pin "Read the Docs common dependencies" (sphinx, recommonmark, sphinx-rtd-theme, mock, pillow, etc). This work is done at #5956 and #5986

This feature is under a Feature Flag because it's not broadly tested yet and may need some extra work once we have more data on projects using it and having common issues. I'd be happy to enable this flag to any project that has been involved in this issue and get some feedback from you.

As a side note, I also added another Feature Flag (UPDATE_CONDA_STARTUP) to update the version of conda to the latest release at the beginning of the process, which it may be good to enable as well.

Please, let me know what are the projects you would like to enable these flags, and I will do it next week probably.

Thanks you all for the support on this!

[adam-urbanczyk]

I believe CQ is also suffering from this issue: https://readthedocs.org/projects/cadquery/builds/10978472/
Could you enable this CONDA_APPEND_CORE_REQUIREMENTS flag for CQ @humitos ?

CC: @jmwright

[humitos]

@adam-urbanczyk Done! Let me know if there is any problem (better in a new issue or support email, so we don't notify all the people involved in this issue). Thanks!