Sphinx: 0 appended to reference id when called by automodule

[momargoh]

Subject: 0 appended to reference id when called by automodule

Problem

In my project root I have dirs for docs/, examples/ and src/. In docs/ I have an .rst file that uses automodule to print the docstring of a module in the examples directory. In this docstring is a citation of a reference as below.

This example goes over the analysis found in the author's paper [Author2018]_.

.. [Author2018] Author, T., Rest of the reference, 2018

When I compile my docs to .html the reference appears with an appended 0, ie:

This example goes over the analysis found in the author's paper [Author20180].
[Author20180] | Author, T., Rest of the reference, 2018

Procedure to reproduce the problem

I am using python3-sphinx: sudo apt install python3-sphinx and then simply running make html in docs/.

make html

Error logs / results

No error log is produced.

Expected results

The reference appears as in the docstring: without the appended 0.

Reproducible project / your project

I have attached a zip of a mock project demonstrating the problem.
example_project.zip

Environment info

• OS: Ubuntu 17.10 (same result on Ubuntu 16.04)
• Python version: 3.6.3
• Sphinx version: 1.5.6

[jfbu]

Thanks for reporting and the attached mock project. Same with Sphinx 1.6.6 and 1.7.2.
With provided https://github.com/sphinx-doc/sphinx/files/1846112/example_project.zip I can reproduce still after commenting out extensions in conf.py like that:

extensions = [
    'sphinx.ext.autodoc',
#    'sphinx.ext.doctest',
#    'sphinx.ext.intersphinx',
#    'sphinx.ext.todo',
#    'sphinx.ext.coverage',
#    'sphinx.ext.mathjax',
#    'sphinx.ext.ifconfig',
#    'sphinx.ext.viewcode',
#    'sphinx.ext.autosummary',
#    'sphinx.ext.githubpages',
    'numpydoc'
]

and the problem disappears when commenting out 'numpydoc'.

It appears numpydoc does some reference renaming (see thread at https://github.com/numpy/numpydoc/pull/136).

@jnothman can you please give a look? perhaps this is an issue known to numpydoc project or which should be reported there.

@momargoh thanks a lot for the mock project. I will examine later if blame is to put more on numpydoc or Sphinx ;-), awaiting @jnothman's advice.

[jnothman]

It's a known feature ...uhh... hack.
https://github.com/numpy/numpydoc/pull/136 should fix the rendering in
HTML, but makes a problem of TeX​, which is one reason it's not been merged.

[jnothman]

The problem that numpydoc is trying to solve is to have each reference scoped locally, to the docstring. This allows for repeated references, and for footnote notation to be used for references within a docstring. But then numpydoc mangles the references to make sure there are no conflicts. I think it used to do it less aggressively, but the current numpydoc release, which is the one I started hacking, is pretty aggressive about it.

[jfbu]

@jnothman thanks for these explanations. I need to go see how this is done at numpydoc currently. Regarding Sphinx + tex, it is planned that Sphinx will use in future normal \cite.

[momargoh]

@jnothman thanks a lot for the explanation, good to have that clarified.

[jnothman]

Currently we mangle the reST and destroy tables and all kinds of bad. The PR attempts to change the reference id but not the text.

[jfbu]

Adding this link to relevant numpydoc issue.

[jfbu]

side-note, I wanted to try out with sphinx latexpdf, there was an error. Turns out, to fix it I had to escape the underscore from LaTeX in document title. Here is the fix:

latex_documents = [(master_doc, 'example_project.tex',
                    'example\\_project Documentation', 'Your Name', 'manual'),
                   ]

cf the doc for "title" in LaTeX:

title: LaTeX document title. Can be empty to use the title of the startdoc. This is inserted as LaTeX markup, so special characters like a backslash or ampersand must be represented by the proper LaTeX commands if they are to be inserted literally.

[jfbu]

@jnothman could you please give a look at https://github.com/jfbu/numpydoc/tree/disambiguate-undecorate-refs. This is a follow-up to your https://github.com/numpy/numpydoc/pull/136. The idea is to keep some de-ambiguating in final result. In my brief testing on very minimal project this works fine for html and latex. Perhaps the -<number> postfix should be a letter.

I used the https://github.com/sphinx-doc/sphinx/files/1846112/example_project.zip provided by @momargoh. For latexpdf I first applied the above fix.

I had a problem that the docstring of example.py was not recognized as such, hence was skipped by your relabel_references(app, doc). I needed to comment out a conditional in it to force execution, and I have not investigated more (because I am no expert in Docutils internals). I added a few more docstrings where I used footnote style for citations (as seems to be the numpydoc way), to have duplicates and see how it came out. Probably you will have handy bigger test project.

I am commenting here and not at numpydoc because I feel I do not quite understand the numpydoc framework so it feels more cozy here.

[jnothman]

So that affects the display of the references when ambiguous, and also
fixes LaTeX? I suppose that's worthwhile. I might be happier if the display
is unaffected, but at least this would be a good step towards merge.

The code you commented out: I can't remember what it was for but presumably
you think it is unnecessary

[jfbu]

I don't have much experience with how numpydoc output is supposed to look like. I actually prefer your non-modified PR which does not affect the final display of references: the hyperlinks (and backlinks) both in html and latex work fine. With my addition footnote like citations [1], [2] will systematically get transformed to look like [1-1], [2-1], then [1-2], [1-3] etc... which (to my taste) are a bit weird in html. The only advantage is that the big PDF bibliography is completely unambiguous even when printed on paper.

I feel that this whole business of using footnote syntax to handle citations is a bit messy. And regarding bibilographic references with usual non-numeric markers, it would be expected that when the same [JohnDoe1980] is used multiple times it creates only one entry in the endnotes bibliography in LaTeX but this is not the case, a new occurence is handled like a new citation.

I had to comment out the code checking if something originated in a docstring because in my testing with the project of @momargoh it was failing to hangle the example.py docstring contents. I did not investigate more.

[jfbu]

@jnothman I tried building ScipPy documentation v1.0.0 for html and latex.

_Your code which I commented out is needed_ else assert len(prefix) == HASH_LEN + 1 raises an exception. (I will push a commit to my branch to de-comment it out; besides my test with SciPy showed me things do not behave as I naively thought. So consider my branch as awaiting total revision). This first happens with this citation node:

citation_node = <citation backrefs="id1" ids="ct65" names="ct65"><label>CT65</label><paragraph>Cooley, James W., and John W. Tukey, 1965, \u201cAn algorithm for the
machine calculation of complex Fourier series,\u201d <emphasis>Math. Comput.</emphasis>
19: 297-301.</paragraph></citation>
label_node = <label>CT65</label>
astext = CT65
prefix = CT65

Anyway, I feel that your https://github.com/numpy/numpydoc/pull/136 is very satisfying.

Regarding PDF builds, here is typical extract with numpydoc 0.7.0 from (PDF) bibliography showing the repetitions (the Abramowitz was highlighted when I took the screen shot).

with your PR it looks like this (from shorter reference tags, the page layout changed, so this does not cover exactly same text).

and latexmk complains

Latexmk: Summary of warnings:
  Latex found 638 multiply defined reference(s)
Latexmk: All targets (scipy-ref.pdf) are up-to-date

but these warnings can be ignored, because the links work fine.

Inside the auto-generated docs, one finds for example:

where the [1] links to one of the Milton Abramowitz and Irene A. Stegun in the bibliography.

My provisory conclusion is that:

• numpydoc should make it mandatory to user that citations are in a References section in each docstring, particularly when footnote syntax is used.

• then it would theoretically be easier for a Sphinx extension to use LaTeX bibunits package (not tested...) to incorporate the citations locally into the References section, for PDF build.

• and your PR would then give very good result not only for html but also latex

• and one does not care so much that LaTeX gives in PDF output also a bibliography with many similar looking citation keys at end of PDF.

SIDE NOTE: to build PDF doc of SciPy I needed two changes to their conf.py.

• latex_engine = 'xelatex'

• latex_elements = {'maxlistdepth': '12'}

I used Sphinx 1.6.6. Then the latex_preamble from SciPy's conf.py is silently ignored (since Sphinx 1.6). Which is for the better because it would provoke failure with xelatex engine. Usage of xelatex fixed the Unicode chars issues, starting with ≠ but then others too, I did not insist with pdflatex.

[jfbu]

@jnothman sorry for spam but I understand why my branch gives on scipy doc same result as yours. Turns out relabel_references() is called at each processed source file, not once globally. So my added code fails to detect reference duplications across the entirety of the SciPy doc, in particular the PDF output is not affected by it.

(using a global dict I can get the expected effect, regarding latex build)

[jnothman]

right. I don't mind the spam, but I don't have much capacity to deal with
anything resulting from it atm

On 28 Mar 2018 3:52 am, "Jean-François B." [e-mail] wrote:

@jnothman https://github.com/jnothman sorry for spam but I understand
why my branch gives on scipy doc same result as yours. Turns out
relabel_references() is called at each processed source file, not once
globally. So my added code fails to detect reference duplications across
the entirety of the SciPy doc, in particular the PDF output is not affected
by it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sphinx-doc/sphinx/issues/4782#issuecomment-376596628,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz61WzZtInIkmrmb8SwT5aOvwD1K36ks5tim5SgaJpZM4S6bgB
.

[jfbu]

@momargoh as @jnothman's https://github.com/numpy/numpydoc/pull/136 got merged, numpydoc 0.8.0 should solve your issue. I will keep this here open for a while, because the PDF builds will probably need for further thinking. They will be fine as hyperlinked documents but not so if printed on paper. On the side of Sphinx some refactoring of LaTeX versus citation references is planned, and I prefer to get this done first and then come back to examine impact on numpydoc.

[tk0miya]

@jfbu This issue was marked as 1.7.3 task. What is remaining task? Are there anything I can do for this?

[jfbu]

@tk0miya This is not directly a Sphinx issue, but a numpydoc one, which got solved at numpydoc 0.8.0. However the way it is done there is not completely satisfactory regarding LaTeX so I kept the ticket open. I can't work on this in immediate future, and it is not core Sphinx issue per se, thus closing now.