Sphinx: Non-Latin headings are not converted into proper anchor links

[shimizukawa]

If a heading includes non-Latin characters, they are ignored.

If none of the characters in a heading is Latin, an anchor link like "id1" is created.

It gets really bad if you have 2 headings with the same Latin word and the rest in non-Latin. e.g. "Инструкция по Searchanise" and "Модуль Searchanise"; this way both anchor links will be built upon the same word Searchanise, which really kills it.

You can check a live example here: http://searchanise-supporters-guide.readthedocs.org/ru/latest/

Cyrillic anchor names are valid and should be used (see Wikipedia for example: http://ru.wikipedia.org/wiki/Pantera#.D0.92.D0.BB.D0.B8.D1.8F.D0.BD.D0.B8.D0.B5_.D0.B8_.D1.82.D0.B5.D0.BD.D0.B4.D0.B5.D0.BD.D1.86.D0.B8.D0.B8).

Thanks!

---

• Bitbucket: https://bitbucket.org/birkenfeld/sphinx/issue/1449
• Originally reported by: Konstantin Molchanov
• Originally created at: 2014-04-11T07:24:47.880

[shimizukawa]

_From Takayuki Shimizukawa on 2014-04-11 06:01:26+00:00_

Thanks for reporting.
I can't find the issued location as you mentioned because the live example has many pages.. please give me a location information.

Technically, the behavior rely on the docutils implementation.
http://sourceforge.net/p/docutils/code/HEAD/tree/tags/docutils-0.11/docutils/nodes.py#l2081
Please let me know which docutils version do you use.
Thanks.

[shimizukawa]

_From Konstantin Molchanov on 2014-04-11 08:48:41+00:00_

Apologies for the misleading example. Here are some concrete links:

http://searchanise-supporters-guide.readthedocs.org/ru/latest/widget.html#id1
(heading—Бесплатный)

http://searchanise-supporters-guide.readthedocs.org/ru/latest/magento.html#searchanise
(heading—После установки расширения Searchanise админка недоступна)

http://searchanise-supporters-guide.readthedocs.org/ru/latest/admin.html#id2
(heading—Клиентская панель управления Searchanise; note that in this case, for some reason, even the word Searchanise is ignored).

I'm using docutils v. 0.11.

Should I redirect the issue to docutils then?

[shimizukawa]

_From Takayuki Shimizukawa on 2014-04-11 09:07:46+00:00_

Thanks. I'm looking for the example as you mentioned:

It gets really bad if you have 2 headings with the same Latin word and the rest in non-Latin. e.g. "Инструкция по Searchanise" and "Модуль Searchanise"; this way both anchor links will be built upon the same word Searchanise, which really kills it.

If this behavior is true, it's a bug. However, if these 2 headings generates 2 different ids (as "id1" and "id2"), it's a docutils' current specification I think.

Should I redirect the issue to docutils then?

Yeah, in either case bug or specification, I think it is faster/straight way.

[shimizukawa]

_From Konstantin Molchanov on 2014-04-11 20:34:08+00:00_

OK, I've posted it as a bug at docutils: https://sourceforge.net/p/docutils/bugs/254/

I think it is a bug rather than the desired behavior since many resources like Wikipedia use non-Latin anchor names and are OK with it.

UPD: I haven't gotten any response so far, and, according to how they handle other tickets, I won't get any soon. This is really sad since the feature is crucial for all users who use non-Latin alphabets.

[shimizukawa]

_From Gleb Goncharov on 2014-04-24 10:26:09+00:00_

Hi!

This issue is really annoying for me since I too use Russian headings.

The Docutils issue tracker on SourceForge appears to be kind of dead—the issues are not even reviewed.

Is there any chance this issue is fixed in Sphinx? I really don't think the Docutils people are going to do anything related anytime soon.

Thanks!

[shimizukawa]

_From Takayuki Shimizukawa on 2014-04-24 14:51:44+00:00_

I think it is too hard to override/substitute the ID generation function by Sphinx because the function is in the deep of docutils. If sphinx override it by monkey patch, I think it might be so fragile one.

[tk0miya]

I think it is very difficult to support multibyte IDs in many outputs. I know HTML5 allows them for its ID system. Do you know how HTMLHelp works? How about LaTeX? How about roff? Sphinx (and docutils) supports many kinds of output formats. And the ID system have to support all of them. If you have a good idea, please post it to the docutils group. I'll join the discussion.

[tony]

Edit: This may be unrelated to the issue, sorry!

I worked around this issue my prefixing the labels with letters

This may not be related, not sure, but I am having headers like this:

0.4.2 (2020-08-01)
--------------------

converted to id1, id2...

but if I wrote "melon", it'd work.

Is this related to this particular issue? Where in the sphinx doc code are these headers / label generated? Or is it in 100% in docutils?

Is there a config/tool/extension that works around this automatically?

I see mentions to set_id around the net:

• https://www.sphinx-doc.org/en/3.x/_modules/sphinx/addnodes.html
• https://stackoverflow.com/questions/13641592/specifying-anchor-names-in-rest
• https://github.com/gagern/Cindy3D/commit/c8b967276bd1dce093b410ac55eda1d6d993faad

When I did

libvcs 0.4.2 (2020-08-01)
---------------------------

Then I got good results: #libvcs-0-4-1-2020-08-01

[tk0miya]

@tony It is a behavior originated from docutils:
https://repo.or.cz/docutils.git/blob/HEAD:/docutils/docutils/nodes.py#l2196