Sphinx: EOFError when using multiprocessing in 3.1.0

[robsan00]

Describe the bug
When I am using multiprocessing (option "-j auto") in my local repositories, at least under Linux I am getting pickle errors. I tried to reproduce this with a simple test repository (creating 20 dummy files), but that was not enough to reproduce the error. Maybe I need 100s of files or cross references or similar to get the error.

Anyway, the error:

Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py", line 91, in _process
    pipe.send((failed, collector.logs, ret))
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle traceback objects

Exception occurred:
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
The full traceback has been saved in /tmp/sphinx-err-4oh9kpix.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Process Process-6:
(...)

The content in the logfile:

# Sphinx version: 3.1.0
# Python version: 3.6.7 (CPython)
# Docutils version: 0.14 
# Jinja2 version: 2.10
# Last messages:
#   reading sources... [ 89%] pyprep/pyprep.mode .. test_pyframe/test_pyframe.test_sigop.test_siggen
#   
#   reading sources... [ 94%] test_pyframe/test_pyframe.test_sigop.test_sigproc .. tools/tools.device_setup.prep_device
#   
#   reading sources... [100%] tools/tools.errorcheck .. tools/tools.upgrade_pacmon
#   
#   
#   
#   waiting for workers...
#   
# Loaded extensions:
#   sphinx.ext.mathjax (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /home/robsan00/venv/python3/lib/python3.6/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.type_comment (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.viewcode (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/viewcode.py
#   sphinx.ext.intersphinx (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (3.1.0) from /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/ext/todo.py
Traceback (most recent call last):
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/application.py", line 348, in build
    self.builder.build_update()
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 416, in read
    self._read_parallel(docnames, nproc=self.app.parallel)
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 467, in _read_parallel
    tasks.join()
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py", line 107, in join
    self._join_one()
  File "/home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py", line 112, in _join_one
    exc, logs, result = pipe.recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

This works fine when using Sphinx 3.0.4 or "-j 1" instead of "-j auto".

Environment info

• OS: Linux
• Python version: 3.6.7
• Sphinx version: 3.1.0 (worked fine in 3.1.0)
• Sphinx extensions:
  sphinx-rtd-theme==0.4.3
  sphinxcontrib-applehelp==1.0.2
  sphinxcontrib-devhelp==1.0.2
  sphinxcontrib-htmlhelp==1.0.3
  sphinxcontrib-jsmath==1.0.1
  sphinxcontrib-qthelp==1.0.3
  sphinxcontrib-serializinghtml==1.1.4
  sphinxcontrib-websupport==1.1.0
• Extra tools: -

[tk0miya]

Sorry for the inconvenience. Could you add a debug log to sphinx.util/parallel.py to investigate the error, please?

--- a/sphinx/util/parallel.py
+++ b/sphinx/util/parallel.py
@@ -88,6 +88,7 @@ class ParallelTasks:
             errmsg = traceback.format_exception_only(err.__class__, err)[0].strip()
             ret = (errmsg, traceback.format_exc())
         logging.convert_serializable(collector.logs)
+        print(failed, collector.logs, ret)
         pipe.send((failed, collector.logs, ret))

     def add_task(self, task_func: Callable, arg: Any = None, result_func: Callable = None) -> None:  # NOQA

Please open /home/robsan00/venv/python3/lib/python3.6/site-packages/sphinx/util/parallel.py and enable the debug log.

[robsan00]

This gives me a 60 MByte file which (1) I am not sure if you can do anything with and (2) basically contains our source code, which I don't really to give out completely. The last things printed out before the whole thing goes down is something like this:

return settings\n\x94}\x94(\x8c\t_NicField\x94j\x02\x01\x00\x00K\x1eK,\x87\x94\x8c\x15create_mach_info_file\x94j=3\x01\x00K/Kp\x87\x94\x8c\x10get_nic_settings\x94j=3\x01\x00KsK\xaa\x87\x94u}\x94(\x8c\t_NicField\x94j)\x03\x00\x00\x8c\x15create_mach_info_file\x94j)\x03\x00\x00\x8c\x10get_nic_settings\x94j)\x03\x00\x00u\x8c\x0fpyprep.machinfo\x94t\x94uub.'

The next line is the error from above.

I also tried to just print in case of an exception:

try:
    pipe.send((failed, collector.logs, ret))
except:
    print(failed, collector.logs, ret)
    raise

This did not look better, though.

[webknjaz]

FTR here's another repo where it's reproducible: https://github.com/cherrypy/cheroot/runs/761272594#step:9:203 (Python 3.7).

Since it's a public repo, it should be possible to see the issue locally by cloning the repo and running tox -e build-docs (pip install tox first, of course). I hope having a public source will help you see it first-hands.

P.S. I confirm that w/o parallelism it works: https://readthedocs.org/projects/cheroot/builds/11233786/.

[tk0miya]

@webknjaz Thank you for info. I reproduced the error on my local!

[tk0miya]

I posted #7802 to fix this.

Note: I used this Dockerfile to reproduce the error.

FROM python:3.7-slim

RUN apt update; apt install -y git make build-essential vim
RUN git clone https://github.com/cherrypy/cheroot
WORKDIR /cheroot
RUN pip3 install -e .
RUN pip3 install -U Sphinx
RUN pip3 install tox
RUN apt install -y enchant
RUN tox -e build-docs --notest

[webknjaz]

s/#7802/#7818

[robsan00]

@webknjaz Thanks for reproducing this!

@tk0miya The fix works for us, thanks!

[webknjaz]

I also confirm that this is fixed for me. Great job, @tk0miya!