Readthedocs.org: How to use the readthedocs/build docker image locally to test build

[tlvu]

Expected Result

A way to launch the same build as on ReadTheDocs website but locally to test build.

Actual Result

Keep pushing to Github to trigger builds to test. This surely waste resources on ReadTheDocs server and make the testing feedback loop longer.

[stsewd]

The easy way is to install read the docs locally

[davidfischer]

The easy way is to install read the docs locally

Easy...

More seriously, what are you looking to test locally? You can do a local Sphinx/MkDocs build on your system without pushing to RTD.

[tlvu]

Okay, let me just backup and explain why I need this.

In Ouranosinc/raven#148 we ran out of memory while doing the build. I want to perform the same build, in the same docker image as RTD to see how much memory we need so I can request a sensible amount.

Also I am not looking to install RTD locally ... why should I when there's a docker image ... right? Or am I missing something?

[stsewd]

It's mostly because all the commands come from the codebase, and we set some env variables, not really easy to follow all the code to have a container running _exactly_ the same.

[stsewd]

Can you link your project? So we can see if we can increase the memory/time limits

[tlvu]

The doc build is coming from this repo https://github.com/Ouranosinc/raven.

[tlvu]

I was trying to strip down what we need to install and with this very minimal environment.yml, I already run out of memory, see build url https://readthedocs.org/projects/raventlvu/builds/9278183

$ cat environment-rtfd.yml 
name: raven
channels:
- birdhouse
- conda-forge
- defaults
dependencies:
- pywps>=4.2
- xarray
- gdal>=2.3

[stsewd]

which is the original read the docs project? I guess https://readthedocs.org/projects/raventlvu/ this one is for testing

[tlvu]

I have to ask my colleague for the original RTD project. Yes https://readthedocs.org/projects/raventlvu/ is for testing.

[Zeitsperre]

Hi and thanks for spending the time looking into this issue for us!

The original project can be found at https://readthedocs.org/projects/pavics-raven/. It's been failing from bad configuration reasons for the past 3.5 months but I went in and tried to fix it a few weeks ago with a PR https://github.com/Ouranosinc/raven/pull/134. Since then, the issue has been a memory allocation error.

It could be that there are some bad libraries in our conf.py, but the docs are building locally so I'm tempted to say that isn't the case.

[stsewd]

I've increased the memory limits to 1.5 GB, but it still runs out of memory. You could try using https://sphinx-autoapi.readthedocs.io/en/latest/reference/directives.html which extracts your docs without the need to install the project dependencies.

[tlvu]

I've increased the memory limits to 1.5 GB, but it still runs out of memory. You could try using https://sphinx-autoapi.readthedocs.io/en/latest/reference/directives.html which extracts your docs without the need to install the project dependencies.

That is what I've been testing on my https://readthedocs.org/projects/raventlvu/builds/9278183 and I was also running out of memory. I still had to install a subset of the conda environment. Probably I did it wrong and misuse or misunderstood that autoapi feature. Still new to Sphinx docs and RTD.

[stsewd]

So, the autodoc feature of sphinx requires the project to be installed to extract the docs (because it actually executes the code). Autoapi extracts the docs in a static way (no need to run the code). So, when using autoapi, tou can skip the dependencies installation for the project.

You need to replace the directives of autodoc http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html to the ones used by autoapi https://sphinx-autoapi.readthedocs.io/en/latest/reference/directives.html

And btw, I don't think you need to install your project to generate your documentation https://github.com/Ouranosinc/raven/tree/master/docs/source (I don't see any call to autodoc at first sight)

[huard]

docs/source/processes.rst includes autoprocess directives, an autodoc extension we created to document PyWPS processes automatically from its attributes. However, these attributes are extracted from a class instance, so the autoapi option won't work.

[tlvu]

@stsewd The memory limit check seems to be buggy. I have this passing build https://readthedocs.org/projects/pavics-raven/builds/9479705/ have more items in conda environment.yml than this failed build https://readthedocs.org/projects/pavics-raven/builds/9480361/ due to "excessive memory consumption".

I have managed to trim down the installs needed to only the following few packages https://github.com/Ouranosinc/raven/blob/533a3d2819dc6616d93772673e14df23fb17ea25/environment-docs.yml, still consuming too much memory seems weird.

[tlvu]

I actually got our doc build to work on ReadTheDocs again, see PR https://github.com/Ouranosinc/raven/pull/167 but I hope the memory limit check is not an intermittent issue, as stated in my previous comment.

[humitos]

@tlvu Hi! The way that conda uses the memory is weird. We've been having problems with conda and OOM in the last months but there is not too much we can do.

Since conda says that their latest versions handle the memory in an smarter way, we added a flag (UPDATE_CONDA_STARTUP) for projects to update conda _before_ creating the command. If you are happy with that, I can enable it for you in your project.

See https://docs.readthedocs.io/en/latest/guides/feature-flags.html

You may be interested in CONDA_APPEND_CORE_REQUIREMENTS from that page as well. Let me know!

[tlvu]

@humitos yes for UPDATE_CONDA_STARTUP because why not :D

No for CONDA_APPEND_CORE_REQUIREMENTS since I want the initial conda create step to be as small as possible to not burst the memory limit for each step. From what I understand, this option will add more packages to be installed to the initial conda create step to save a separate step which means the initial conda create step will be bigger and that's the step I want to trim down.

Thanks for UPDATE_CONDA_STARTUP and for the support of all the ReadTheDocs team.

Should I leave issue open because I still think having the ability for the user to supply their own docker image will basically make the build faster (no more install step) and provide the user with the ability to have a reproducible doc build environment (they can locally run their doc build using the exact same docker image).

My issue is resolved now so I don't mind closing this issue either.

By the way, I guess I am not the only project with OOM problem because of the awsome sphinx autodoc feature. I solved my issue by mocking import (using autodoc_mock_imports in sphinx conf.py file, see my PR https://github.com/Ouranosinc/raven/pull/167) of all the 3rd party dependencies so they do not have to be installed for real and by adding my own sources into the sys.path for autodoc to parse for docstrings and auto generate the doc. I basically work-around the requirement to install my project and all its dependencies, resulting in less memory usage and faster build.

You might want to document this somewhere on ReadTheDocs docs so other users using sphinx autodoc will benefit and prevent other similar OOM support call in the future.

[humitos]

@humitos yes for UPDATE_CONDA_STARTUP because why not :D

Done.

I triggered a new build at https://readthedocs.org/projects/pavics-raven/builds/9483814/. You can see that conda was updated before running create.

No for CONDA_APPEND_CORE_REQUIREMENTS since I want the initial conda create step to be as small as possible to not burst the memory limit for each step. From what I understand, this option will add more packages to be installed to the initial conda create step to save a separate step which means the initial conda create step will be bigger and that's the step I want to trim down.

Yes. It will add the packages installed in the next step (conda install) into the environment.yaml file. Since we don't know why conda breaks on OOM, I'm not sure if this will be worse or better :smile:

Should I leave issue open because I still think having the ability for the user to supply their own docker image will basically make the build faster (no more install step) and provide the user with the ability to have a reproducible doc build environment (they can locally run their doc build using the exact same docker image).

We are not going to support providing a custom user Docker image. At least, it's not in our plans for now. So, we can close this issue for now.

I'm closing the issue, but feel free to re-open if you consider necessary. Also, if you find any other problem, open a new issue. Happy to help if time permits :heart:

You might want to document this somewhere on ReadTheDocs docs so other users using sphinx autodoc will benefit and prevent other similar OOM support call in the future.

We have this already documented at https://docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules. Although, it may be good to place it at https://docs.readthedocs.io/en/latest/guides/build-using-too-many-resources.html and expand _why_ this could be needed.

On the other hand, we recommend to use autoapi if what you want is to auto document the modules. It's static and does not need to install anything to get the modules documentation :wink:

[humitos]

I triggered a new build at https://readthedocs.org/projects/pavics-raven/builds/9483814/. You can see that conda was updated before running create.

In fact, it takes 280 compared with 845 without the flag (see https://readthedocs.org/projects/pavics-raven/builds/9483280/)

Also, the conda environment creation takes 163 seconds while without the flag it takes 473.

Please, confirm that this work properly because I'm seeing good improvements from outside.

[tlvu]

We have this already documented at https://docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules.

My case was not C modules so never thought to look there.

Although, it may be good to place it at https://docs.readthedocs.io/en/latest/guides/build-using-too-many-resources.html and expand _why_ this could be needed.

Yeah this would be a more natural location to document this "mock" trick to decrease resource usage (install less stuff).

On the other hand, we recommend to use autoapi if what you want is to auto document the modules. It's static and does not need to install anything to get the modules documentation.

I am new to this project but I think we could not use autoapi, we had to use autodoc. I see old autoapi configuration in our sphinx conf.py file too so I think we have previously tried to use autoapi.

[tlvu]

I triggered a new build at https://readthedocs.org/projects/pavics-raven/builds/9483814/. You can see that conda was updated before running create.

In fact, it takes 280 compared with 845 without the flag (see https://readthedocs.org/projects/pavics-raven/builds/9483280/)

Also, the conda environment creation takes 163 seconds while without the flag it takes 473.

Please, confirm that this work properly because I'm seeing good improvements from outside.

Yup, it works perfectly fine for us, thanks a lot.

[humitos]

I am new to this project but I think we could not use autoapi, we had to use autodoc. I see old autoapi configuration in our sphinx conf.py file too so I think we have previously tried to use autoapi.

I would appreciate if you can share the reasons. We are trying to improve it and any feedback that you may have would be really helpful to make it better.

[tlvu]

I am new to this project but I think we could not use autoapi, we had to use autodoc. I see old autoapi configuration in our sphinx conf.py file too so I think we have previously tried to use autoapi.

I would appreciate if you can share the reasons. We are trying to improve it and any feedback that you may have would be really helpful to make it better.

I'll let my co-workers @huard and @Zeitsperre elaborate why autodoc is chosen over autoapi since they've been around on this project much longer than me.

[huard]

We're using an autodoc extension I wrote for PyWPS Process objects. The information required to document those objects is defined once an instance is created, so we really need to run some code to document each Process.