Sphinx: C++ domain: oscillating build failures using function overload cross-refs

[svenevs]

Subject: oscillating build failures using function overload cross-refs

CC: @mquinson

Problem

This one is really quite strange. Initial build -> success, repeat build with no changes -> failure, repeat build success, etc. So every other build will fail.

This is using Sphinx 1.8.0.b1, and using the newly added (and beautiful) function overload cross-references -- the ones that include return_type func_name(param_1, param_2) [const]).

I have spent a few hours trying to resolve this myself, as it is unclear if this is a Sphinx bug or the Breathe extension doing something dubious (possibly with the pending_xref's). I believe that the issue is actually with Sphinx because of the symptoms of the build failure. There are two extensions involved, exhale (which sits on top of breathe), and just using breathe itself without exhale.

Context

namespace simgrid {
    namespace s4u {
        class Actor {
        public:
            void kill();
            static void kill(aid_t pid);
        };
    }// namespace s4u
}// namespace simgrid

Problem 1

exhale generates a large number of reStructuredText documents (which ultimately use the breathe directives), one of which is called library_root.rst which will perform a .. include:: unabridged_api.rst. When the following link is used

:cpp:func:`void simgrid::s4u::Actor::kill()`

First build success with no warnings. Any subsequent builds will warn about unabridged_api.rst not being included, even though it is.

Problem 2

Using exhale or just breathe on its own, the following link

:cpp:func:`void simgrid::s4u::Actor::kill(aid_t)`

Is initially successful on the first build. If you re-build once, you will get

Exception occurred:
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_breathe/venv/lib/python2.7/site-packages/sphinx/domains/cpp.py", line 4170, in find_declaration
    candId = symbol.declaration.get_newest_id()
AttributeError: 'NoneType' object has no attribute 'get_newest_id'

But every other build is successful.

Summary

The symptoms above (warnings that shouldn't exist about a document not being included, or every-other-build-failing) are what lead me to believe something is broken in Sphinx specifically. The mangled names from Breathe as compared to Sphinx are identical, so I can't really see how breathe would cause this, but of course that cannot be ruled out.

I am grateful that these new cross-ref links exist! If you think this should be discussed on the breathe issue tracker please say so. I concluded it was a sphinx bug in resolving references, but this is an area of sphinx I am very much not familiar with.

Procedure to reproduce the problem

I have produced a repository to be able to test this: https://github.com/svenevs/sphinx-cpp-cross-ref-bug

The docs_exhale and docs_breathe generated _build/html/index.html include extra information. For example, if you comment out the Explicit Overload v2 one:

--- a/docs_exhale/index.rst
+++ b/docs_exhale/index.rst
@@ -50,7 +50,7 @@ without warnings, but the next ``make html`` will produce the warning about
 Explicit Overload v2
 ++++++++++++++++++++

-- ``void simgrid::s4u::Actor::kill(aid_t)``: :cpp:func:`void simgrid::s4u::Actor::kill(aid_t)`
+.. - ``void simgrid::s4u::Actor::kill(aid_t)``: :cpp:func:`void simgrid::s4u::Actor::kill(aid_t)`

 Every other build will fail on this.

Then builds will always succeed, but the second and remaining builds produce an "incorrect" warning:

checking consistency... /home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/api/unabridged_api.rst: WARNING: document isn't included in any toctree

Error logs / results

Example logs are included in the docs_exhale and docs_breathe folders, here is the one for docs_exhale, it doesn't really give much useful information though:

# Sphinx version: 1.8.0b1
# Python version: 2.7.13 (CPython)
# Docutils version: 0.14 
# Jinja2 version: 2.10
# Last messages:
#   writing output... [ 18%] api/classsimgrid_1_1s4u_1_1Actor
#   writing output... [ 27%] api/dir_include
#   writing output... [ 36%] api/file_include_the_code.hpp
#   writing output... [ 45%] api/file_view_hierarchy
#   writing output... [ 54%] api/library_root
#   writing output... [ 63%] api/namespace_simgrid
#   writing output... [ 72%] api/namespace_simgrid__s4u
#   writing output... [ 81%] api/program_listing_file_include_the_code.hpp
#   writing output... [ 90%] api/unabridged_api
#   writing output... [100%] index
# Loaded extensions:
#   sphinx.ext.mathjax (1.8.0b1) from /home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/ext/mathjax.pyc
#   alabaster (0.7.11) from /home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/alabaster/__init__.pyc
#   breathe (4.10.0) from /home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/breathe/__init__.pyc
#   exhale (0.2.0) from /home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/exhale/__init__.pyc
Traceback (most recent call last):
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/cmd/build.py", line 304, in build_main
    app.build(args.force_all, filenames)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/application.py", line 341, in build
    self.builder.build_update()
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 350, in build_update
    len(to_build))
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 415, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 596, in write
    self._write_serial(sorted(docnames))
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 604, in _write_serial
    doctree = self.env.get_and_resolve_doctree(docname, self)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/environment/__init__.py", line 623, in get_and_resolve_doctree
    self.apply_post_transforms(doctree, docname)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/environment/__init__.py", line 670, in apply_post_transforms
    transformer.apply_transforms()
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/transforms/__init__.py", line 90, in apply_transforms
    Transformer.apply_transforms(self)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/docutils/transforms/__init__.py", line 171, in apply_transforms
    transform.apply(**kwargs)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/transforms/post_transforms/__init__.py", line 90, in apply
    typ, target, node, contnode)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/domains/cpp.py", line 6767, in resolve_xref
    target, node, contnode)[0]
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/domains/cpp.py", line 6683, in _resolve_xref_inner
    matchSelf=True, recurseInAnon=True)
  File "/home/sven/Desktop/sphinx-cpp-cross-ref-bug/docs_exhale/venv/lib/python2.7/site-packages/sphinx/domains/cpp.py", line 4170, in find_declaration
    candId = symbol.declaration.get_newest_id()
AttributeError: 'NoneType' object has no attribute 'get_newest_id'

Expected results

Every build is always successful ;)

Environment info

• OS: reproduced on OSX, Fedora, and Debian
• Python version: 2.7 in the logs, but same error occurs with 3.6 and 3.7
• Sphinx version: 1.8.0b1

[jakobandersen]

Thanks a lot for the bug report. For a long time I have noticed that partial builds would sometimes break some symbol lookups, and your example perfectly reproduces the problem (though with more harsh consequences due to the new cross-references).
I didn't try the Exhale example, but the Breathe example triggers the problem. The pure Sphinx example seems to work on my machine though.

[jakobandersen]

This indeed uncovered some old problems, but I it should all be fixed in #5354. @svenevs, do you have time to test this branch on the full project you encountered the problem before I merge it?

[svenevs]

Wow! Yes, I'll be back at a computer in ~15 minutes!

[svenevs]

@jakobandersen I can confirm that this fixes the breaking ref lookups!!!!!!!!! It fixes both the docs_breathe one, as well as the original code framework that this came from.

"Problem 1" still persists (api/unabridged_api.rst: WARNING: document isn't included in any toctree) in the docs_exhale test, warning produced on second and subsequent builds. However, it seems I misdiagnosed this. If docs_exhale/index.rst is just

Testing With Exhale and Breathe
===============================

.. toctree::
   :maxdepth: 2

   api/library_root

The second and subsequent builds will warn about this. So perhaps it is better to open a separate issue since the core problem (breaking build failure) is resolved with #5354. The warning doesn't appear to inhibit the correct generation, and does not appear to be linked to anything related to the C++ cross references at all.

Is the sphinx-dev mailing list still a good place to ask about something like this? The .. include:: structure I originally conjured up was engineered specifically to avoid warnings from Sphinx about inclusion, and it seems the behavior has changed / I need to learn more.

Regardless, thanks for looking into / fixing this so quickly, totally epic man!

# TODO: enable assertion when we at some point find out how to do cleanup
# for now, just take the first one, it should work fine ... right?

hehehehe.

[jakobandersen]

Thanks! Merged.

WARNING: document isn't included in any toctree

This I am quite sure is a completely different problem that I don't know much about. Maybe #5281 fits, otherwise please open a new issue.
The mailing list doesn't seem to have much activity, so if it can be formulated as a Github issue that may be best.

"hehehehe"

There may be a way to break it, but it wouldn't be a regression, and I'm not sure the documented C++ would be valid :-) ("crash-driven development"?)

[svenevs]

Lol crash-driven-development rule 1: undefined behavior shall be randomly defined.

Ok I'll open a separate issue soon, this seems to be a separate issue / possible regression. I'll spend a little more time hunting down when this broke though ;)

Thanks again!