Readthedocs.org: Unable to build project online

[svenevs]

Details

• Project URL: http://fake-nanogui.readthedocs.io/en/latest/
• Build URL (if applicable): https://readthedocs.org/projects/fake-nanogui/builds/6699993/
• Read the Docs username (if applicable): sjm324

Expected Result

Build of docs with 26 warnings (template issues with breathe).

Actual Result

After cat docs/conf.py it spins for a bit, but then nothing continues.

Build id: 6699993

FWIW this is a testing site for the real project, which is facing the same issues right now. We were trying to solve the "no setup.py" issue, but now the rest of the build fails. I'm admin on the fake docs, so lets work with the above build ID and solve it there.

Is there a way for me to view the build log directly? Is one of the extensions causing the problem?

Thanks for any help!

[stsewd]

Hi, I just build your docs on my local instance successfully, could you try building the docs again? (probably was a temporary problem on the servers).

[svenevs]

Hi @stsewd, thanks for checking. Unfortunately, it seems to have failed again in the same way (Build #6707242).

Is it possible that having submodules may somehow be a problem? Are there any kinds of build logs you could potentially post here that I could help stare at?

[stsewd]

That's very strange, could you try wiping the environment? https://readthedocs.org/wipe/fake-nanogui/latest/

The logs from my local instance looks very similar to yours!

[svenevs]

Harumph. Yeah I've tried wiping a couple of times, it installs cleanly each time (though it does use cached versions of the extensions in the install logs, but I seriously doubt that matters).

I'm digging through the stuff added to conf.py by RTD, the only thing that looked dubious was this

# Add RTD Static Path. Add to the end because it overwrites previous files.
if not 'html_static_path' in globals():
    html_static_path = []
if os.path.exists('_static'):
    html_static_path.append('_static')
html_static_path.append('/home/docs/checkouts/readthedocs.org/readthedocs/templates/sphinx/_static')

since I already had '_static' in the html_static_path, but removing it from the original conf.py has no change.

I'm the author of one of the extensions (exhale) which had a new release, but my testing site builds just fine with it as well as another user confirmed the latest release worked for them. Even so, if there was an exception or something from the extensions, that would show up in the RTD logs availble to me when sphinx-build is run, right?

[stsewd]

@svenevs I just imported your project on the org site, and indeed, it fails. I'm not sure why on my local instance succeeds, here are some errors that I found on the output on my local instance. Sorry for not being more helpful.

/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/class_nanogui__detail__FormWidget_LT__Color_COMMA__std__true_type__GT.rst:29: WARNING: Too many template argument lists compared to parameter lists. Argument lists: 1, Parameter lists: 0, Extra empty parameters lists prepended: 1. Declaration:
    nanogui::detail::FormWidget<Color, std::true_type>
/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/class_nanogui__detail__FormWidget_LT__T_COMMA__typename_std__is_enum_LT__T__GT___type__GT.rst:29: WARNING: Too many template argument lists compared to parameter lists. Argument lists: 1, Parameter lists: 0, Extra empty parameters lists prepended: 1. Declaration:
    nanogui::detail::FormWidget<T, typename std::is_enum<T>::type>
/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/class_nanogui__detail__FormWidget_LT__T_COMMA__typename_std__is_floating_point_LT__T__GT___type__GT.rst:29: WARNING: Too many template argument lists compared to parameter lists. Argument lists: 1, Parameter lists: 0, Extra empty parameters lists prepended: 1. Declaration:
    nanogui::detail::FormWidget<T, typename std::is_floating_point<T>::type>
/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/class_nanogui__detail__FormWidget_LT__T_COMMA__typename_std__is_integral_LT__T__GT___type__GT.rst:29: WARNING: Too many template argument lists compared to parameter lists. Argument lists: 1, Parameter lists: 0, Extra empty parameters lists prepended: 1. Declaration:
    nanogui::detail::FormWidget<T, typename std::is_integral<T>::type>
/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/class_nanogui__detail__FormWidget_LT__bool_COMMA__std__true_type__GT.rst:29: WARNING: Too many template argument lists compared to parameter lists. Argument lists: 1, Parameter lists: 0, Extra empty parameters lists prepended: 1. Declaration:
    nanogui::detail::FormWidget<bool, std::true_type>
/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/class_nanogui__detail__FormWidget_LT__std__string_COMMA__std__true_type__GT.rst:29: WARNING: Too many template argument lists compared to parameter lists. Argument lists: 1, Parameter lists: 0, Extra empty parameters lists prepended: 1. Declaration:
    nanogui::detail::FormWidget<std::string, std::true_type>
/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/function_nanogui__file_dialog.rst:15: WARNING: doxygenfunction: Unable to resolve multiple matches for function "nanogui::file_dialog" with arguments () in doxygen xml output for project "NanoGUI" from directory: ./doxyoutput/xml.
Potential matches:
    - std::string nanogui::file_dialog(const std::vector<std::pair<std::string, std::string>>&, bool)
    - std::vector<std::string> nanogui::file_dialog(const std::vector<std::pair<std::string, std::string>>&, bool, bool)
/home/stsewd/rtd/readthedocs.org/user_builds/cat/checkouts/latest/docs/api/typedef_nanogui__Matrix3f.rst:15: WARNING: Error in type declaration.
If typedef-like declaration:
  Type must be either just a name or a typedef-like declaration.
  If just a name:
    Invalid definition: Expected end of definition. [error at 18]
      nanogui::Matrix3f = typedef Eigen::Matrix3f
      ------------------^

[svenevs]

No need to apologize! Those warnings are expected, breathe has some issues with how it processes C++11 using directives, as well as template signatures. On a fully-updated local build, those are WARNING's (but the build still succeeds).

Has the RTD policy changed such that these warnings are now errors? Is there a way to relax this so that these warnings are allowed to pass through, even if they make some weird signatures?

Update: Hmmm. I don't think the warnings are the problem here. I just removed all of the template stuff / warnings etc, and it still fails after cat conf.py. The most recent build is here, build number 6707416. This was after a wipe for good measure, but there seems to be something else going on x0

I appreciate the help. I don't know what would be causing this anymore :/

[svenevs]

@stsewd does anything show up in the online logs after cat conf.py? I've wiped / rebuilt a few times, the latest build 6751769 is still silently failing after cat conf.py. I don't know if this is helpful, but I needed the master branch back so the latest now points to a fix_the_docs_online branch that should produce zero build warnings.

[stsewd]

@svenevs now I believe this could be a problem with memory limits, see https://github.com/rtfd/readthedocs.org/issues/3613

[svenevs]

Possibly, though I'd be surprised. Local testing averages a maximum of 150 megabytes of memory usage by sphinx-build. It is known that the underlying XML parser in breathe produces memory leaks, but I don't think machine differences on the servers would lead to 150 megabytes -> over 1GB.

I'll start back-tracking breathe versions and see if there are different results. Do you happen to know if exceptions raised during the sphinx-build execution are logged for the user?

[svenevs]

Oh snap. I think I've found the problem. I capture the output of doxygen and write it to the console, I silenced all printing to the console and logged the output to /dev/null and it seems to get past it. It seems more likely that the subprocess buffers are overflowing, even though I was actually very very careful about how that was done (writing to a tmpfile).

It'll take a little more effort to create this the "right" way (I just commented out a LOT of code hehe). When I get that setup more cleanly, I'll post back here with the actual cause and solution in case somebody ever faces the same issue.

[stsewd]

@svenevs thanks for investigating the root of this problem, the problem was because of a memory limit caused by the flooding output of doxygen, right? If that is the case I can add this issue to #3613 for future consideration.

[svenevs]

the problem was because of a memory limit caused by the flooding output of doxygen, right?

Ok status confirmed. The issue was the subprocess buffer overflowing (which is a form of memory limit). This particular scenario is actually not catchable (as fare as I understand it), an excellent article on how / why here. I had actually written my code to avoid subprocess.PIPE and use tmpfiles, but this seems to still have failed. So it could be that creating or reading the temporary files was the actual problem. For smaller sites, this was not a problem. For larger projects (and therefore more doxygen output), it overflowed in some way.

So the solution, if anybody ever reading this needs it, is to send doxygen output to /dev/null when on ReadTheDocs.

https://github.com/svenevs/exhale/blob/94042bb29c7a66a8f82c9576f21a9ce72221853e/exhale/configs.py#L962

That's how you can know. I'll link to the implementation, but there's more going on in that method than you probably need. Execution of doxygen is here, you can take whatever you want from that (BSD3 license).