Sphinx: TypeError: add_source_parser() positional arguments error in Sphinx v3.0.0

[matthewfeickert]

Describe the bug
In pyhf's nightly CI we noticed that our build of our docs fails with the release of Sphinx v3.0.0

Exception occurred:
  File "/opt/hostedtoolcache/Python/3.7.6/x64/lib/python3.7/site-packages/sphinx/application.py", line 1069, in add_source_parser
    self.registry.add_source_parser(*args, **kwargs)
TypeError: add_source_parser() takes 2 positional arguments but 3 were given
The full traceback has been saved in /tmp/sphinx-err-gu866l6p.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

which when run locally the generated log is

$ cat /tmp/sphinx-err-blah.log
# Sphinx version: 3.0.0
# Python version: 3.7.5 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.1
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "/home/feickert/.venvs/pyhf-dev/lib/python3.7/site-packages/sphinx/cmd/build.py", line 279, in build_main
    args.tags, args.verbosity, args.jobs, args.keep_going)
  File "/home/feickert/.venvs/pyhf-dev/lib/python3.7/site-packages/sphinx/application.py", line 244, in __init__
    self.setup_extension(extension)
  File "/home/feickert/.venvs/pyhf-dev/lib/python3.7/site-packages/sphinx/application.py", line 398, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/feickert/.venvs/pyhf-dev/lib/python3.7/site-packages/sphinx/registry.py", line 414, in load_extension
    metadata = setup(app)
  File "/home/feickert/.venvs/pyhf-dev/lib/python3.7/site-packages/m2r.py", line 652, in setup
    app.add_source_parser('.md', M2RParser)
  File "/home/feickert/.venvs/pyhf-dev/lib/python3.7/site-packages/sphinx/application.py", line 1069, in add_source_parser
    self.registry.add_source_parser(*args, **kwargs)
TypeError: add_source_parser() takes 2 positional arguments but 3 were given

The exact same code passed the nightly build the night before with Sphinx v2.4.4.

To Reproduce
Steps to reproduce the behavior:

$ git clone https://github.com/scikit-hep/pyhf.git
$ cd pyhf
$ python -m pip install --ignore-installed -U -q --no-cache-dir -e .[docs,test]
$ cd docs
$ make html

For additional reference, our GitHub Actions workflow is here.

Expected behavior
The docs successfully build.

Your project

pyhf Issue 818

Environment info

• OS: Linux Ubuntu 19.10 (and whatever Linux the GitHub Actions ubuntu-latest is)
• Python version: 3.7.5
• Sphinx version: 3.0.0
• Sphinx extensions:

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.autosummary',
    'sphinx.ext.coverage',
    'sphinx.ext.mathjax',
    'sphinx.ext.ifconfig',
    'sphinx.ext.viewcode',
    'sphinx.ext.githubpages',
    'sphinxcontrib.bibtex',
    'sphinx.ext.napoleon',
    'sphinx_click.ext',
    'nbsphinx',
    'm2r',
    'sphinx_issues',
    'sphinx_copybutton',
    'xref',
]

Additional context
pyhf Issue 818

[CrossNox]

Having the same issue on gitlab-ci with

extensions = [
    'sphinx.ext.napoleon',
    'sphinx.ext.autodoc',
    'm2r'
]

[tk0miya]

Since 1.8.0, the suffix argument for app.add_source_parser() has been deprecated. And it is removed now.
https://github.com/sphinx-doc/sphinx/blob/713bbf5cafa3fc5e143ced59dafe56f4b802ef80/CHANGES#L1323-L1324

And it has also been reported at m2r project (https://github.com/miyakogi/m2r/issues/51). I think m2r should be updated to use new API.

Thanks,

[matthewfeickert]

@CrossNox Thanks for your PR to m2r. As the project hasn't had a release since 2018 though my guess is that it is in archive mode and your PR will never get reviewed and merged. If you find another suitable extension can you share it here?

[CrossNox]

@matthewfeickert sure, I'll have to search a replacement eventually. For now, we've pointed the dependency to my fork to test it. m2r's owner doesn't seem to have been active since last june :(

[matthewfeickert]

@CrossNox @halfak it might be worth (for all of our projects) checking how Black is able to use Makrdown but doesn't use m2r. Maybe if they have time (before I do to go step through it) @ambv, @willingc, or @cooperlees could comment RE: that.

[matthewfeickert]

checking how Black is able to use Markdown but doesn't use m2r.

From first glance it seems this is done by using recommonmark and then recommonmark.parser.CommonMarkParser to set

source_parsers = {".md": CommonMarkParser}

in docs/conf.py. Though the recommonmark README shows that

# for Sphinx-1.4 or newer
extensions = ['recommonmark']

I haven't tried this on pyhf yet, but this might be an easy solution(?). Though seems like for projects like pyhf Sphinx Issue 7000 is relevant.

[lwasser]

@matthewfeickert i'm running into the same issue right now. have you found a working fix? do you suggest trying this:

# for Sphinx-1.4 or newer
extensions = ['recommonmark']

So did you replace m2r with recommmark or is that an additional extension?
many thanks!

[matthewfeickert]

have you found a working fix?

No, until Sphinx Issue 7000 is resolved I don't think there is a clean solution (for Markdown READMEs that are in a directory above the docs). I would welcome being proved wrong.

So did you replace m2r with recommmark or is that an additional extension?

Yes. This would be part of the solution, but it isn't enough by itself. It seems that just switching m2r for recommmark takes care of everything else that is Markdown and further below.

[lwasser]

got it. thanks @matthewfeickert i guess we could also just go back to all rst. and maintain a separate readme ... or link to the readme. if you or anyone finds an elegant solution please let me know!

[CrossNox]

@matthewfeickert FWIW our build no longer fails with that hotfix on the m2r PR. Maybe it's worth it to tag a release on the fork until #7000 is developed?

[matthewfeickert]

FWIW our build no longer fails with that hotfix on the m2r PR

:+1: Glad to hear that's working for you.

Maybe it's worth it to tag a release on the fork until #7000 is developed?

Perhaps, though I think I'm more inclined to just move off of m2r as having our docs be dependent on (officially) unmainted software makes me feel uneasy.

i guess we could also just go back to all rst.

To that end we have pyhf PR 819 open. Ideally if we can find a solution in the next few days then we won't have to potentially revert this in the future.

[sheiksadique]

@CrossNox I have a build fail in my CI because of the same issue. Can you help me understand how you fetch the hotfix. Do you explicitly checkout the package from gitlab and install it manually or is there a more elegant way of going about it?

[CrossNox]

@sheiksadique we had 'm2r' in our setup.py and simply changed for 'm2r @ git+https://github.com/crossnox/m2r@dev#egg=m2r'

[kepler]

This simple and clever workaround did the trick: https://github.com/readthedocs/recommonmark/issues/191#issuecomment-622369992.

[CrossNox]

Since I'm still seeing lots of clones on my fork (which I created for the PR linked above) here goes a quick update: I've released it as m2r2 after changing some of the tooling for what I feel more comfortable with right now and building CD and testing pipelines.