Sphinx: Sphinx build crashes intermittently when building C docs

[Lucide]

Describe the bug
Sphinx builds fail at most intermittently when building documentation (ok, fail, ok, fail, ...), sometimes less often. The culprits seem to be C functions. But I'm not sure if it's a Breathe issue. I've provided an example.
The original project uses CMake+Doxygen+Breathe+Sphinx with RTD Theme, it's a C library I'm writing documentation for.
I'm on Windows.

Log

# Sphinx version: 3.3.0
# Python version: 3.9.0 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.2
# Last messages:
#   Running Sphinx v3.3.0
#   loading pickled environment...
#   done
#   building [mo]: targets for 0 po files that are out of date
#   building [html]: targets for 0 source files that are out of date
#   updating environment:
#   0 added, 2 changed, 0 removed
#   reading sources... [ 50%] api-reference
# Loaded extensions:
#   sphinx.ext.mathjax (3.3.0) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\ext\mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinxcontrib\applehelp\__init__.py
#   sphinxcontrib.devhelp (1.0.2) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinxcontrib\devhelp\__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinxcontrib\htmlhelp\__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinxcontrib\serializinghtml\__init__.py
#   sphinxcontrib.qthelp (1.0.3) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinxcontrib\qthelp\__init__.py
#   alabaster (0.7.12) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\alabaster\__init__.py
#   breathe (4.23.0) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\__init__.py
#   recommonmark (0.6.0) from d:\users\nemo\desktop\test\venvwin\lib\site-packages\recommonmark\__init__.py
Traceback (most recent call last):
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\cmd\build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\application.py", line 352, in build
    self.builder.build_update()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\builders\__init__.py", line 297, in build_update
    self.build(to_build,
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\builders\__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\builders\__init__.py", line 418, in read
    self._read_serial(docnames)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\builders\__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\builders\__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\io.py", line 223, in read_doc
    pub.publish()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\core.py", line 219, in publish
    self.apply_transforms()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\core.py", line 200, in apply_transforms
    self.document.transformer.apply_transforms()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\transforms\__init__.py", line 87, in apply_transforms
    super().apply_transforms()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\transforms\__init__.py", line 171, in apply_transforms
    transform.apply(**kwargs)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\recommonmark\transform.py", line 333, in apply
    self.traverse(self.document)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\recommonmark\transform.py", line 313, in traverse
    self.traverse(child)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\recommonmark\transform.py", line 313, in traverse
    self.traverse(child)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\recommonmark\transform.py", line 303, in traverse
    newnode = self.find_replace(c)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\recommonmark\transform.py", line 285, in find_replace
    newnode = self.auto_code_block(node)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\recommonmark\transform.py", line 259, in auto_code_block
    parser.parse(newsource, new_doc)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\parsers\rst\__init__.py", line 191, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\parsers\rst\states.py", line 170, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\parsers\rst\states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\parsers\rst\states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\parsers\rst\states.py", line 2096, in directive
    return self.run_directive(
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\docutils\parsers\rst\states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\directives.py", line 137, in run
    return self.render(node_stack, project_info, filter_, target_handler, NullMaskFactory(),
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\directive\base.py", line 96, in render
    return object_renderer.render(node_stack[0], context)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\renderer\sphinxrenderer.py", line 1950, in render
    result = method(self, node)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\renderer\sphinxrenderer.py", line 1875, in dispatch_memberdef
    return self.visit_function(node)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\renderer\sphinxrenderer.py", line 1583, in visit_function
    nodes = self.handle_declaration(node, declaration)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\renderer\sphinxrenderer.py", line 608, in handle_declaration
    nodes_ = self.run_directive(obj_type, declaration, content_callback, options)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\breathe\renderer\sphinxrenderer.py", line 573, in run_directive
    nodes = directive.run()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 3225, in run
    return super().run()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\directives\__init__.py", line 190, in run
    name = self.handle_signature(sig, signode)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 3263, in handle_signature
    symbol = parentSymbol.add_declaration(
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 1983, in add_declaration
    res = self._add_symbols(nestedName, declaration, docname, line)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 1926, in _add_symbols
    symbol._fill_empty(declaration, docname, line)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 1538, in _fill_empty
    self._add_function_params()
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 1559, in _add_function_params
    self._add_symbols(nn, decl, self.docname, self.line)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 1926, in _add_symbols
    symbol._fill_empty(declaration, docname, line)
  File "d:\users\nemo\desktop\test\venvwin\lib\site-packages\sphinx\domains\c.py", line 1532, in _fill_empty
    assert line is not None
AssertionError

To Reproduce
Steps to reproduce the behavior:

$ git clone https://github.com/Lucide/sphinx-c-crash-test
$ cd sphinx-c-crash-test
$ pip install -r requirements.txt
$ doxygen ./doc/Doxyfile.VisualT_doxygen_generate
# configure absolute paths in doc/conf.py
$ sphinx-build -b html ./doc ./doc/html

Your project
https://github.com/Lucide/sphinx-c-crash-test

Environment info

• Windows 10
• Python version: 3.9.0 (CPython)
• Sphinx version: 3.3.0
• Sphinx extensions: [breathe, recommonmark]
• Extra tools: [Doxygen, RTD sphinx theme]

Additional context
I think it first occurred when I added pygments_style = "friendly" to my conf.py. The build failed so I removed the line, thinking it was conflicting with the theme. Later I realized it wasn't the direct cause... but I think I rememeber I had many successful builds with functions before the error first occurred.

[jakobandersen]

Thanks, this smells like it could be the issue @utzig experienced and fixed via #8364. If possible, can you try with the 3.3.x branch and see if the problem is fixed?
My guess for the reason why the problem is intermittent is that this method is used (more) during partial builds. If you clean all build files before each build I think you can avoid the bug for now.

[utzig]

Yeah, this is pretty much the issue I had, but a clean build would fail 100% of the time, and afterwards it fails somewhat intermittently. Current branch 3.3.x is working well.

[Lucide]

I admit I'm completely ignorant of how to build a Python project, and of Python in general. It surely looks like the same bug, but in my case, a clean build always succeeds, problems arises on the second, fourth, sixth, ...

[utzig]

I admit I'm completely ignorant of how to build a Python project, and of Python in general

The projects I have issues with are all low-level C software.

[Lucide]

Yes, I'm working with C too (not sure if low level), I was referring to the suggestion of trying out sphinx 3.3.x. I have no idea of how to build it.

[utzig]

Yes, I'm working with C too (not sure if low level), I was referring to the suggestion of trying out sphinx 3.3.x. I have no idea of how to build it.

Clone this repo, checkout branch 3.3.x, python setup.py sdist, there will be a tar.gz file at dist, install it with pip install dist/<the tar gz full name>. Otherwise wait a few days for 3.3.1! :-)

[Lucide]

The problem disappeared with 3.3.x branch, this issue can be closed (or left open till release, so it's easier to find).

[utzig]

My guess for the reason why the problem is intermittent is that this method is used (more) during partial builds. If you clean all build files before each build I think you can avoid the bug for now.

The project I have issues with consists of many smaller projects linked together with intersphinx. Two of them depend on each other, so I have to build one of them first only for the inventory, than build the second, then build the first one again. As the first project is built twice, it does actually match your "partial builds" comment!

[tk0miya]

Now I'm going to release 3.3.1. So closing this now. Thank you for reporting.