Readthedocs.org: OpenMW documentation builds failed, OOM

[psi29a]

Details

• Project URL: https://readthedocs.org/projects/openmw/
• Build URL (if applicable): https://readthedocs.org/projects/openmw/builds/5740932/
• Read the Docs username (if applicable): psi29a

Expected Result

builds creates and posts html based documentation

Actual Result

builds have been failing for awhile now, we are currently using up to 1.6 GiB of memory to generate our documentation. If we can bump the limit of 1 GiB up to 2 GiB for OpenMW, that would solve all our issues.

Thank you!

[humitos]

Are you still having this issue? Your builds seem to be failing for another reason.

[psi29a]

We still have this issue if we want to a full build, we tried to work around it now by only building part of our documentation.

[stsewd]

So, the issue here is a problem with the time/memory limits?

[psi29a]

We can only do a partial build of our documentation as we need around 1.6 GiB to do a full build. Anyway we can bump to 2GiB please?

[ericholscher]

Have you done any debugging to see why your doc builds require so much memory. That seems like an incredibly large amount to generate documentation.

[ericholscher]

Just saw the data here: https://github.com/aiidateam/aiida_core/issues/1472#issuecomment-391610600 -- we should also be looking in the Sphinx theme it seems, though perhaps this is just a case of having a ton of stuff need to be in memory.

[psi29a]

I know why, we use doxygen (C++ codebase), autodoc and the breath sphinx extensions... on a rather large project (OpenMW). We marked it at about 1.6 GiB of RAM before finishing.

For the time being, I turned this off, but now we have no API documentation being generated.

[humitos]

Hi @psi29a!

I just saw that you @giovannipizzi did a change that seemed to work (https://github.com/giovannipizzi/aiida_core/commit/82dfcf1eaf304c8b0f46879205de2717858f5735)

All your last 10 builds were built without issues. Was that enough for you or you disabled something else from your code?

Let me know if this is still and issue for you or we can close it for now.

[psi29a]

Thanks for reply! :)

The last 10 builds built without issues because I disabled the C++ autodoc code, so still no API docs. The change above won't change the fact that sphinx with breath plugin and doxygen takes up 1.6GiB of memory. So until the memory limit gets raised, the documentation can't be built. :/

[giovannipizzi]

Hi, in my case the memory problem was occurring only in the singlehtml build. The RTFD folks now enabled a way to disable selectively only this build (see discussion here https://github.com/rtfd/readthedocs.org/issues/3220, it required committing a suitable .readthedocs.yml file.
Now, the builds are working normally.

[humitos]

@psi29a Hi!

I increased your mem limit to 2g and I'd like you to re-enable the C++ autodoc code and test again.

I want to let you know that I'm testing how this behaves in the server since our resources are limited and it may break the builder where your project is being built. It seems that your project build once a day more or less, so I suppose it shouldn't be a big problem.

In case that we notice some problems with the memory consumption, I will set the default value for the memory again.

Also, I wrote some thoughts on how to deal with this issues in general in #4403

[psi29a]

Super, I'll give it a spin.

[psi29a]

OK... just reverted my commit to bring back breath/doxygen mix. I re-ran my build process and measured memory usage at its peak:

bcurtis 10108 97.4 5.0 1719388 1668600 pts/2 R+ 14:53 2:58 /home/bcurtis/Workspace/Private/OpenMW/openmw/docs/venv/bin/python2 /home/bcurtis/Workspace/Private/OpenMW/openmw/docs/venv/bin/sphinx-build -T -E -D language=en . _build/html

We came in at around 1.7 GiB memory usage.

Going to commit it and see what RTD does.

[psi29a]

Is building now:
https://readthedocs.org/projects/openmw/builds/7511574/

[stsewd]

It failed while building the localmedia stuff, you can disable that with the yaml file https://docs.readthedocs.io/en/latest/yaml-config.html#formats

[humitos]

@psi29a I think it's good to follow what @stsewd said, and also maybe it worth to create a different branch for these tests so we can run a build different from master/lastest without you reverting the commits every time you/we want to test this. What do you think?

[psi29a]

Nope!

Command killed due to excessive memory consumption

Anything else we can do?

[psi29a]

I hope this is what you meant:
https://gitlab.com/OpenMW/openmw/blob/master/readthedocs.yml

[psi29a]

Failed again?
https://readthedocs.org/projects/openmw/builds/7511851/

[stsewd]

This time it failed when building the actual documentation :/, I suppose the previous build was successful because the builders were less busy

[psi29a]

I'm not going to leave it like this for long, I'll eventually need to revert to having docs building again. :)

[humitos]

@psi29a at the moment, I think we can't offer more memory than my test of 2g :(

As I said in a previous comment, I suggest you keep the branch of your current docs clean and make these changes for test the autodoc build in a different branch.

[humitos]

@psi29a we changed some settings in our servers and we are testing if we can offer more memory for projects that were having this kind of issues. So, I increased the memory available for openwm and I'd like you to re-enable breath/doxygen so we can see that it builds properly.

Please, keep me informed. Thanks!

[psi29a]

I'll give it a spin when I have time. Cheers!

[humitos]

I found our queue routing wasn't working properly. So, although your project is properly configured to have more resources available, it's not going to the correct Celery queue.

This will be fixed once #4553 gets merged and deployed. I'm sorry for the confusion.

[humitos]

@psi29a This should be fixed and your project should have more resources available to build your docs. I'm closing this issue here, but feel free to reopen if you consider.

[psi29a]

Thanks, I'll give it a spin and report back.