Readthedocs.org: Support Poetry

[danielknell]

It would be nice to be able to build docs for libraries managed with poetry.

maybe by checking for a [tool.poetry] section in pyproject.toml, having to keep a separate requirements.txt file in sync feels a bit fragile.

[stsewd]

What would be the poetry workflow in read the docs? Pipfile support will be out soon btw. If this gets implemented, it would live as an option in the configuration file.

[danielknell]

locally i run poetry install to create a virtualenv and install the dependencies, then poetry run sphinx-build ... to run sphinx-build inside that virtualenv.

although if there is a virtualenv already active then poetry will install the dependencies into that instead, which means it should work the same as for requirements.txt or Pipfile?

[sobolevn]

My workflow is pretty similar:

1. poetry install, which installs both dev and prod dependencies by default
2. cd docs, optional step
3. poetry run make html, to build the static website

Real project example that uses both poetry and rtd: https://github.com/wemake-services/wemake-django-template

[humitos]

So, replacing the pip install step by poetry install is all that's needed here or there are more tricks that I'm not seeing?

[sobolevn]

@humitos yes, I think so.

Mind, that we also have to work with venv differently: we need to prefix all commands with poetry run.

[danielknell]

if you have the venv active, you can run poetry install and it will install into the current environment the same way that pip install -r requirements.txt will, you can then use the venv as you would had normally.

poetry run is just a shorthand to run things in the venv without having to remember where it is or activate it first.

[stsewd]

dev and prod dependencies

Is there a use case where users only want to install dev/prod deps? There are any other configurations that users may want to set?

[danielknell]

poetry install will install both dev and regular dependencies
poetry install --no-dev will install only regular dependencies and no dev stuff

the latter is more along the lines of setup.py install with the caveat that it wont install the module itself, i cant think of a reason off the top of my head not to install dev dependencies for docs generation...

you can also use poetry install --extras "foo bar" to install extra dependency groups foo and bar

full docs are at https://poetry.eustace.io/docs/cli/#install

[uSpike]

I think this problem will resolve itself once the next version of pip is released with PEP-517 support (I've heard in January 2019?). It was merged - https://github.com/pypa/pip/pull/5743
Then, a project only needs to add

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

to the pyproject.toml, and the normal pip install should work.

[danielknell]

except the build script runs python setup.py install --force by default instead of a pip install . so pep-517 wont come into play?

[stsewd]

@danielknell we do support installing the project using pip.

@uSpike I have read the pip issue and pep, but I wasn't able to understand it very well yet :/ will this replace pipenv too?

[uSpike]

@stsewd pip might be able to install from a Pipfile at some point in the future, but AFAIK it is not on the road map any time soon.

[cjw296]

@danielknell - yeah, check here: https://docs.readthedocs.io/en/latest/yaml-config.html#python-pip-install, might be able to do it through the web config too. So, guess we're just waiting on https://github.com/pypa/pip/issues/6106. @stsewd : once pip 0.19.0 is out, how long before it makes it into a readthedocs image that we can use?

[stsewd]

We install the latest version of pip before each build #4938, so it should be available after they release it :)

[GPMueller]

I'm not very familiar with Python virtual environments, but there may be another issue there.

I run poetry from a conf.py script, but get the following error, which I don't see how it could be fixed from user side:

Creating virtualenv clang-api-doc-py3.5 in /home/docs/.cache/pypoetry/virtualenvs
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/home/docs/.cache/pypoetry/virtualenvs/clang-api-doc-py3.5/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

[lorinkoz]

https://github.com/pypa/pip/issues/6106#issuecomment-456528521

[sobolevn]

Since new pip@19 is release, here's the short overview of what is going on.
I have locally checked how pip@19 works with two setups:

1. Application: https://github.com/wemake-services/wemake-django-template
2. Library: https://github.com/wemake-services/wemake-python-styleguide

Application

Trying: pip install .
Result:

``` "No file/folder found for package {}".format(name)
poetry.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package wemake-django-template

----------------------------------------

Command "/Users/sobolev/Documents/github/wemake-django-template/.venv/bin/python3 /Users/sobolev/Documents/github/wemake-django-template/.venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/qn/2gssw9hx48g81chw0398hlrr0000gn/T/tmpbzmmkx48" failed with error code 1 in /private/var/folders/qn/2gssw9hx48g81chw0398hlrr0000gn/T/pip-req-build-2qbtac_8
```

Library

Trying: pip install .
Result: It installs everything as it should.

[danielknell]

does it work within rtfd? i keep meaning to try but have not found time to update my projects yet with the config to install via pip.

[uSpike]

@sobolevn I wouldn't expect https://github.com/wemake-services/wemake-django-template to install since it is a template (cookiecutter) project.

[sobolevn]

@uSpike poetry install works, so I had expected pip install to work as well.

[danielknell]

poetry install will only install dev/prod dependencies, not the module itself
pip install . will try and install the module itself, and with it the prod dependencies.

the fact it does not install dev dependencies is concerning and maybe pep 517 isn't enough to fully support poetry projects?

[sobolevn]

poetry install also installs the project itself in the editable mode. https://github.com/sdispater/poetry/releases/tag/0.12.0

I have also tried to research this dev/no-dev problem, looks like there is no known solution yet.

[danielknell]

that means poetry install is equivalent to pip install -e .

[stsewd]

Can anyone try a project in rtd? We have the latest pip installed, let us know if there is anything else to do from our side to support poetry.

[uSpike]

@sobolevn that project (https://github.com/wemake-services/wemake-django-template) is a template, you're not supposed to pip install it. The equivalent is having a requirements.txt, but they used poetry instead. The actual project is inside https://github.com/wemake-services/wemake-django-template/tree/master/%7B%7Bcookiecutter.project_name%7D%7D but you must run cookiecutter to make it.

@danielknell aah I forgot about that... there might be a way to pass flags to poetry via pip?

[danielknell]

@uSpike doubtful, pip and the pep itself don't really have anything along those lines, which is understandable as its outside of the scope of building and installing from source...

it looks like to install a project with dev dependencies the only option is to run poetry install inside the virtualenv.

[lorinkoz]

What settings should I use in order to trigger an install?
If I check build using setup.py it complains. If I uncheck in won't try
to install anything.

El jue., 24 ene. 2019 a las 9:27, Daniel Knell ([e-mail])
escribió:

@uSpike https://github.com/uSpike doubtful, pip and the pep itself
don't really have anything along those lines, which is understandable as
its outside of the scope of building and installing from source...

it looks like to install a project with dev dependencies the only option
is to run poetry install inside the virtualenv.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rtfd/readthedocs.org/issues/4912#issuecomment-457215488,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADdcSmGZO6hcPwZkAol5cbWdimuVNRAhks5vGcLYgaJpZM4Ynph9
.

[uSpike]

@danielknell I thought PEP-517 included some flags for pip to pass things to the backend: https://www.python.org/dev/peps/pep-0517/#config-settings

Those could then be caught by poetry in https://github.com/sdispater/poetry/blob/master/poetry/masonry/api.py and change how the dist is built, maybe to include dev dependencies?

[danielknell]

@lorinkoz

# .readthedocs.yml
python:
   pip_install: true

[danielknell]

the only way forward i see at the moment unless using vanilla sphinx is to define anything sphinx needs as optional dependencies, then add them to an extras group, then use a config like:

build:
    image: latest

python:
    version: 3.6
    pip_install: true
    extra_requirements:
        - docs

https://github.com/sdispater/poetry#extras

[uSpike]

@danielknell I agree, until https://github.com/pypa/pip/issues/5771 is done and poetry adds support for parsing said config to include dev dependencies in a build.

[lorinkoz]

Brief comment: pip install -e is not yet supported without a setup.py.

[lorinkoz]

I am getting the usual Directory '.' is not installable. File 'setup.py' not found. that makes me think -e is being passed.

[sobolevn]

Yes, I understand that I can not install a project with pip install . without it being a library. But, my question remains: how is that possible to install my deps from pyproject.toml inside rtd?

In other words: how is that possible to replace pip install -r requirements.txt that is currently executed with something like poetry install?

[stsewd]

@sobolevn we don't support executing poetry install.

[danielknell]

@stsewd I am seeing failures, even though the project is setup to install via poetry and installs ok with pip install .[docs] locally it still fails because of no setup.py during build despite both reporting pip 19.0.1, is there any way it could be using a different pip in that step or something?

see:

build: https://readthedocs.org/projects/twelvefactor/builds/8456552/
code: https://github.com/artisanofcode/python-twelvefactor

interestingly this one previously passed, with no changes that should have an influence on the install:
https://readthedocs.org/projects/twelvefactor/builds/8456164/

diff between the two: https://github.com/artisanofcode/python-twelvefactor/compare/18b3e2b59765695053945778852e7860dd3cd8c6...cb6e69c70bc58e2c315a63cae7c56ae64c7b60b1

[guillaumerenault]

https://github.com/orsinium/poetry-setup can be used to generate a setup.py and requirements.txt while this is still a problem.

[lorinkoz]

What's the status of this? My docs started generating okay.

[sobolevn]

@lorinkoz can you please share your setup?

[lorinkoz]

Default setup in readthedocs regarding installing. My project only has a pyproject.toml. From the logs, RTD is not installing my dev dependencies, but in my case, they are not mandatory for building the docs.

[dmfigol]

I am poetry user as well and if I could store all dependencies in pyproject.toml, I'd be happy.
I am not sure that it is poetry job to change how they build the wheel with PEP517 API - I don't think they should include dev dependencies in the metadata.
Wouldn't it be possible to install poetry along with the pip, add some flag in .readthedocs.yml like python.poetry_install=true and then use poetry install in the project root after git checkout? This will install the project in editable mode with all dev dependencies.

[pawamoy]

I'm trying this again for aria2p.

Here is my RTD config:

build:
    image: latest

python:
    version: 3.6
    pip_install: true

requirements_file: docs/requirements.txt

Previously, I had my own aria2p package in this requirement file to make it work:

sphinx==1.8.3
sphinx-rtd-theme==0.4.2
sphinxcontrib-spelling==4.2.0
toml==0.10.0
recommonmark==0.4.0
aria2p>=0.2.0

It means that the docs were built using the latest available release, not the local fetched code.

Today I changed it to ., to use the local code:

sphinx==1.8.3
sphinx-rtd-theme==0.4.2
sphinxcontrib-spelling==4.2.0
toml==0.10.0
recommonmark==0.4.0
.

It seems to work perfectly in a Docker container, but I get an error on RTD.

Docker session history (project root mounted on /code):

docker run -v $(pwd):/code --rm -it python:3.6 bash

cd /code/
apt-get update
apt-get install python-enchant
pip install -U pip
pip install virtualenv
python -m virtualenv --no-site-packages --no-download VENV
/code/VENV/bin/python -m pip install --exists-action=w --cache-dir /root/.cache/pip -r docs/requirements.txt 
sphinx-build -E -b html docs build/docs

Error on RTD:

$ /home/docs/checkouts/readthedocs.org/user_builds/aria2p/envs/latest/bin/python -m pip install --exists-action=w --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/aria2p/.cache/pip -r docs/requirements.txt
Processing /home/docs/checkouts/readthedocs.org/user_builds/aria2p/checkouts/latest
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/docs/checkouts/readthedocs.org/user_builds/aria2p/envs/latest/lib/python3.6/tokenize.py", line 452, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-2d9t32el/setup.py'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-2d9t32el/

It seems that pip is still searching for a setup.py. Could this be a cache problem?

EDIT: indeed it was a cache issue. You can wipe the cache before rebuilding, look at this page https://docs.readthedocs.io/en/stable/guides/wipe-environment.html (thank you @stsewd)

[pawamoy]

One thing that could help debugging these kind of issues would be the ability to see the config used for the current build: the readthedocs.yml if any, and the settings from Admin:Settings and Admin:Advanced Settings.

To summarize, here are the things I tried:

• use a requirements.txt file for the docs, add your package in it: my-package>=0.1.0. It won't use the latest code, but the latest release (which is enough most of the time)
• use a requirements.txt file for the docs, add . in it. Your package will be installed with pip install ., using poetry, and your latest version of the docs will be in sync with your master branch.
• use a requirements.txt file for the docs, with only the dependencies for the docs. Add pip_install: true to the python section of your readthedocs.yml configuration file. Dependencies will be installed normally, then extra command pip install . will be run. It's like the previous solution, but in two steps.

Of course, when RTD and poetry are more integrated together, the need for a requirements.txt could disappear.

My last question and I'll wait for feedback before continuing my flooding: is there a way to invalidate the build cache?

[stsewd]

My last question and I'll wait for feedback before continuing my flooding: is there a way to invalidate the build cache?

Yes, you can wipe the environment before triggering a new build https://docs.readthedocs.io/en/stable/guides/wipe-environment.html

[pawamoy]

Yes! That did the trick, thank you :heart: Will edit my comments above

[agjohnson]

@rtfd/core discussed this a bit today -- and I've mostly only skimmed this issue just now -- but it sounds like pip will eventually support installation with poetry? If so, I think our stance at the moment is that we won't implement this in RTD directly, and instead wait for pip to support installation via poetry.

If there is anyone more intimately familiar with the plans of pip here, I think we'd still like input from you here. Our worry is spreading ourselves thin implementing more specific Python tooling, but we also would like to support common tooling to the Python community.

I'm going to close this issue for now, but I'm fine keeping this conversation going here if we need to re-evaluate this issue.

[dmfigol]

@agjohnson
pip already supports non-editable PEP-517 installs and this is why it is possible to use the following workaround:

build:
  image: latest

python:
  version: 3.6
  pip_install: true
  extra_requirements:
    - docs

However, there is an inconvenience: RTD dependencies like sphinx and other plugins should be listed as extras in pyproject.toml, which in turn will add this metadata to the published package on PyPi. Instead I'd prefer to keep these dependencies in dev section of pyproject.toml along with other dev tools like pytest, flake8, etc.
To enable this workflow, RTD would need to install poetry and call poetry install instead of pip install .[docs] which could be controlled with some flag in .readthedocs.yml

[chrahunt]

The fundamental problem is that Poetry "dev dependencies" don't map directly to a concept that pip understands. pip install installs the package itself (+ any specified extras), which means only things listed as dependencies of the package make it into the virtual environment. At no point currently are dev dependencies considered actual dependencies except within poetry itself.

Users are not necessarily helpless though - I have published rtd-poetry which should transparently download dev dependencies used in conjunction with the a .readthedocs.yml like:

---
build:
  image: latest

python:
  version: 3.6
  pip_install: true

It just checks whether you're on RTD and appends dev dependencies to the wheel metadata if so. I was able to use it successfully for my docs build here, but chrahunt/rtd-poetry#2 needs to be resolved before it is actually usable.

[thejohnfreeman]

@dmfigol is there an unstated assumption in your example? I tried the .readthedocs.yml you wrote, and it didn't work for me:

build:
  image: latest
python:
  version: 3.6
  pip_install: true
  extra_requirements:
    - docs

Processing /home/docs/checkouts/readthedocs.org/user_builds/project-template-python/checkouts/latest
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/docs/checkouts/readthedocs.org/user_builds/project-template-python/envs/latest/lib/python3.6/tokenize.py", line 452, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-74uo7l0b/setup.py'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-74uo7l0b/

[pawamoy]

You might need to wipe your build cache: https://docs.readthedocs.io/en/stable/guides/wipe-environment.html

[thejohnfreeman]

Wiping the environment did it. Thanks for the instructions, @pawamoy!

[jtpavlock]

I got the following to work for config v2

# .readthedocs.yml

version: 2

sphinx:
  configuration: docs/conf.py

python:
  version: 3.6
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

# pyproject.toml

[tool.poetry.dependencies]
sphinx = {version = "^3", optional = true}

[tool.poetry.extras]
docs = ["sphinx"]

I'll echo @dmfigol's sentiment that this isn't ideal for reasons he stated.

[deepio]

None of the options worked for me, so I programmatically installed them in the conf.py file.

# -- Import statements -------------------------------------------------------

import os
import subprocess

if os.environ.get('READTHEDOCS') == 'True':
  # Install m2r for example
  subprocess.check_output(["pip", "install", "m2r"])

[anentropic]

Just a +1 for @jtpavlock method detailed above, it worked for me too.

Note that you need to have such packages listed both as optional = true in the main tool.poetry.dependencies and then again in the extras > docs list.