Readthedocs.org: Add support for arbitrary package names in `python.install`

[webknjaz]

I was trying to follow https://docs.readthedocs.io/en/stable/config-file/v2.html#packages but it seems to miss one important use-case.

My project requires pip 19 for building (because of PEP517 dep).
So I tried to do this:

version: 2
...
python:
  ...
  install:
  - method: pip
    path: pip >= 19.0.3
  ...

And it failed because the validator expects a file system path.

I however need this and I don't want to create a requirements file just for this. It also doesn't make sense to put it into dist extras because it needs to be there before building the dist.

I think it should be pretty easy to extend the config. OTOH just having an up-to-date Pip version capable of PEP 517 would also deal with my problem.

Details

• Read the Docs project URL: https://readthedocs.org/projects/octomachinery/
• Build URL (if applicable): https://readthedocs.org/projects/octomachinery/builds/8869160/
• Read the Docs username (if applicable): webknjaz

Expected Result

Build env have pip>=19.0.3

Actual Result

Failure to parse config

[stsewd]

We already install the latest version of pip, this is done just after the virtual env creation and before the package/requirements installation.

[webknjaz]

Yeah, I see that now. Now I realized that I'm hitting Pip's bug: basically, setuptools from system site-packages leak into the isolated build...

[webknjaz]

As you can see, bumping things in the virtualenv manually doesn't help: https://readthedocs.org/projects/octomachinery/builds/8869252/

[webknjaz]

Looks like https://github.com/pypa/pip/issues/6264

[webknjaz]

By the way, did anyone suggest providing the ability to just specify tox env for building things? This would save me because I've made it work there.

[webknjaz]

Anyway, _I think_ bumping setuptools/pip in the system site-packages would "fix" this. At least, it seems like a valid workaround.

[hroncok]

Bumping system site-packages setuptools to at least 40.8.0 or removing it entirely works as a workaround.

[webknjaz]

Nice, but in my case, it's 40.9.0 which is required because I rely on a feature introduced there (missing setup.py, having only setup.cfg).

[stsewd]

The pip in the global system package comes from the default in ubuntu https://github.com/rtfd/readthedocs-docker-images/blob/71714aa45ffd2b00dc599fd1353651a706ed9f49/Dockerfile#L124-L125

[stsewd]

By the way, did anyone suggest providing the ability to just specify tox env for building things? This would save me because I've made it work there.

I don't think we will support that, we depend on a specific workflow, you may find this interesting #1083

[webknjaz]

Yeah... I see. I know I could migrate to gh-pages but if it's possible to fix this on RTD, maybe we should try.
I'm curious whether you could inject pip install -U pip setuptools --user so that it'd use user install. Maybe It'd help. I also see you use pyenv so probably "system" site-packages are coming from there, not OS, because OS-installed things probably point to a different version of Python anyway. So perhaps you should update not what installed with apt, but Python which is userspace-installed by pyenv...

[stsewd]

If the buggy lib is coming from the installation in pyenv, it would be updated the next time we update the docker images

https://github.com/rtfd/readthedocs-docker-images/blob/71714aa45ffd2b00dc599fd1353651a706ed9f49/Dockerfile#L177-L178

[webknjaz]

@stsewd you update pip there but not setuptools

[webknjaz]

@stsewd it would be nice to indicate which docker image was used on the build page...
Can you please tell me how can I reproduce this locally? Which image should I use? latest one year outdated (#5553), stable doesn't have 3.7. I'm guessing testing?

[webknjaz]

Alright, it looks like latest in config points to 4.x branch...

[webknjaz]

I've also found another issue in the build log:

/home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/bin/python -m pip install --upgrade --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/.cache/pip Pygments==2.3.1 setuptools<41 docutils==0.14 mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<0.6

Requirement already up-to-date: Pygments==2.3.1 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (2.3.1)
Requirement already up-to-date: setuptools<41 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (40.9.0)
Requirement already up-to-date: docutils==0.14 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (0.14)
Requirement already up-to-date: mock==1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (1.0.1)
Requirement already up-to-date: pillow==5.4.1 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (5.4.1)
Requirement already up-to-date: alabaster!=0.7.5,<0.8,>=0.7 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (0.7.12)
Requirement already up-to-date: commonmark==0.8.1 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (0.8.1)
Requirement already up-to-date: recommonmark==0.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (0.5.0)
Requirement already up-to-date: sphinx<2 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (1.8.5)
Requirement already up-to-date: sphinx-rtd-theme<0.5 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (0.4.3)
Requirement already up-to-date: readthedocs-sphinx-ext<0.6 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (0.5.17)
Requirement already satisfied, skipping upgrade: future in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from commonmark==0.8.1) (0.17.1)
Requirement already satisfied, skipping upgrade: babel!=2.0,>=1.3 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (2.6.0)
Requirement already satisfied, skipping upgrade: imagesize in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (1.1.0)
Requirement already satisfied, skipping upgrade: packaging in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (19.0)
Requirement already satisfied, skipping upgrade: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (1.12.0)
Requirement already satisfied, skipping upgrade: snowballstemmer>=1.1 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (1.2.1)
Requirement already satisfied, skipping upgrade: requests>=2.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (2.21.0)
Requirement already satisfied, skipping upgrade: sphinxcontrib-websupport in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (1.1.0)
Requirement already satisfied, skipping upgrade: Jinja2>=2.3 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from sphinx<2) (2.10)
Requirement already satisfied, skipping upgrade: pytz>=0a in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from babel!=2.0,>=1.3->sphinx<2) (2018.9)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from packaging->sphinx<2) (2.3.1)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from requests>=2.0.0->sphinx<2) (1.24.1)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from requests>=2.0.0->sphinx<2) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from requests>=2.0.0->sphinx<2) (2.8)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from requests>=2.0.0->sphinx<2) (2019.3.9)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /home/docs/checkouts/readthedocs.org/user_builds/octomachinery/envs/latest/lib/python3.7/site-packages (from Jinja2>=2.3->sphinx<2) (1.1.1)

for some reason, it installs setuptools<41

[webknjaz]

Ah, that's fine...

[webknjaz]

So I've tried out testing tag and doing pip3.7 install -U setuptools against the pyenv-managed Python outside the virtualenv helped.

@stsewd when should we expect

P.S. btw a bit unrelated: since you're already using pyenv why don't you use its pyenv-virtualenv plugin for managing venvs?

[stsewd]

I'm not really sure about the docker images in dockerhub, I'm using them without any problem, I guess they work for me bc I added them in a custom dockerfile.

I time ago I tried updating pip from the global system/pyenv, but I was facing some complications (I don't remember exactly), that's why I ended up updating pip inside the virtualenv. I'm going to try to play around updating setuptools and see if it works.

P.S. btw a bit unrelated: since you're already using pyenv why don't you use its pyenv-virtualenv plugin for managing venvs?

I guess for historical reasons, when we weren't using it? Also, we don't really activate the virtualenv, we just use the executable from there. I was thinking more in using the venv module for python3, but we still support python2 builds, so isn't an easy refactor.

[webknjaz]

So I went for a dirty hack and now it fails with SphinX:
https://readthedocs.org/projects/octomachinery/builds/8872497/

Any ideas?
https://github.com/sanitizers/octomachinery/blob/b0fc5300e54fec8167892761adc48d9dedecdc1a/docs/conf.py#L14-L52

[stsewd]

What version of sphinx are you using locally?

[webknjaz]

latest

[webknjaz]

I think I've figured it out. I'll need to exec and restart it. Because old sphinx is already in memory which messes things up...

[stsewd]

We install 1.8.x by default, we don't support 2.0 yet, so you may experiment some weird things (maybe)

[webknjaz]

I improved the hack: https://github.com/sanitizers/octomachinery/blob/819713b/docs/conf.py#L14-L65

And it worked: https://readthedocs.org/projects/octomachinery/builds/8872645/

_don't repeat this at home!_

[webknjaz]

Yeah, it says Sphinx 2.0.0 https://docs.octomachinery.dev/ here in the bottom. I didn't set any version limitations. But the problem with hack was replacing things which have already been loaded into runtime via a separate process...

[humitos]

We install 1.8.x by default, we don't support 2.0 yet, so you may experiment some weird things (maybe)

We do support Sphinx 2.0. Although, our theme is not 100% ready for it.

[humitos]

@webknjaz I'm happy to have you on board!

Anyway, _I think_ bumping setuptools/pip in the system site-packages would "fix" this. At least, it seems like a valid workaround.

I think this should be reported into our https://github.com/rtfd/readthedocs-docker-images repository and we should install the latest versions setuptools as well, here:

https://github.com/rtfd/readthedocs-docker-images/blob/71714aa45ffd2b00dc599fd1353651a706ed9f49/Dockerfile#L178

[humitos]

We have very old versions currently,

Python 2.7.14 (default, Jan 14 2019, 21:06:11) 
>>> setuptools.version.__version__
'28.8.0'
>>> 

Python 3.7.1 (default, Jan 14 2019, 21:13:02) 
>>> setuptools.version.__version__
'39.0.1'
>>> 

[stsewd]

We updated this in the dockerfile, we only need to wait for a new release of the docker images https://github.com/rtfd/readthedocs-docker-images/pull/101

[webknjaz]

@stsewd that's a workaround. This issue is about allowing to do pip install <pkg_name>, not just pip install -r <req_file>

[webknjaz]

@stsewd today with the release of the new Pip which is now breaking some of the use-cases it's still relevant: https://github.com/pypa/pip/issues/6434.

[webknjaz]

@stsewd so I'd say this should be reopened and implemented as a feature request.

[stsewd]

We do support installing packages from local system https://docs.readthedocs.io/en/stable/config-file/v2.html#packages or from pip using a requirements file or like an option in the extra_requirements of the setup.py file (we respect the installation order).

If you mean adding _inline_ packages in the rtd config file, it feels like re implementing a requirements file.

Also, we used to install an old version of pip, but we trust that pip wouldn't break, so we install the latest release by default p: I mean, looks like more a pip issue rather than rtd after the setuptools update.

If you think this still can be useful I can reopen as a feature request

[webknjaz]

Yes, I still think that it's useful: I often choose to not have requirements.txt files in favor of something else and it's odd if I'd be forced to have one just for RTD. It's an inconsistent experience.
Not using extras is another choice sometimes because pip doesn't have "install only extras" mode yet.

That issue I mentioned is Pip's own. And the latest Pip renders some of the projects unusable failing when trying to install them.
But my point is that RTD should provide a way to point to non-faulty versions if needed. In my case, I need to put that in YAML config for consistency.

[agjohnson]

This was a feature that we cut from the first implementation of our config file, but i think it's important for users to have this functionality.

We rely too heavily on packaging patterns of yore, like pushing users towards requirements files, and this doesn't translate to a first class experience for users using more elegant solutions that exist for packaging now. I'd also advocate for enabling editable installs, so that pip install --editable '.[docs]' is possible without installing the package (install only extras), though this is a separate issue and can be at least remedied by allowing arbitrary package installation. So, I'd vote that this is the next feature we enable in our python.install config.

Removing design decision we discussed the need for this and we just need to assign priority now.

[ericholscher]

I just ran into this issue, it would indeed be useful 👍

[agjohnson]

Same, I hit this yesterday and found it redundant that I had to make a separate requirements file just for a single package. I don't think we have anything blocking us from implementing this now that we already support a list of requirements files.

[stsewd]

We still need a design decision on how the scheme will look like. One suggestion I have is

python:
  install:
      - packages:
            - mkdocs>=1.1
            - sphinx

[webknjaz]

Sounds good. But don't do too much validation as pip supports many formats including dist @ http://...