Sphinx: Turkish smart quotes / docutils warnings

[Znni]

I updated Sphinx to version 1.6.1 and docutils to version 0.13.1. I'm building .tex documents in several languages, including Turkish. The Turkish build outputs first

[exec] WARNING: C:\foo\bar.rst:: (WARNING/2) language "tr" not supported: Docutils-generated text will be in English.

and then thousands of lines similar to

[exec] C:\foo\foobar.rst:28: WARNING: No smart quotes defined for language "tr".

I see there are some changes in latest Sphinx versions related to handling of smart quotes. I also see that docutils languages directory does not include Turkish language (tr.py). Is the problem simply that docutils does not support Turkish translation? In the end, my end results (PDF files) look fine and, on first glance, references etc. seem to work fine.

I did not get these warnings with Sphinx version 1.5.x. What can I do to get rid of the warnings?

[jfbu]

I do not understand the problem with 'tr', this is a Docutils generated warning, not a Sphinx one, (I am LaTeX consultant, not Python maintainer)

but for the smart quotes: the docutils/utils/smartquotes.py has been updated since 0.13.1 (http://docutils.sourceforge.net/HISTORY.html), in particular at https://sourceforge.net/p/docutils/code/8017 which adds smart quotes for Turkish. They are lacking from Docutils 0.13.1 release, and from the monkey-patch done at sphinx/util/smartypants.py.

I tested with adding this

from __future__ import absolute_import, unicode_literals

import re
from sphinx.util.smartypants import smartquotes
from sphinx.util.docutils import __version_info__ as docutils_version

if docutils_version < (0, 13, 2):
    quotes = smartquotes.smartchars.quotes
    quotes['tr'] = u'“”‘’'
    quotes['tr-x-altquot'] = u'«»‹›'

at top of conf.py, then I get the expected smart quotes applied in html.

I noticed a problem on make latex. It happens independently of the above, on second invocation and so on.

# Sphinx version: 1.6.2+
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6
# Last messages:
#   building [mo]: targets for 0 po files that are out of date
#   building [latex]: all documents
#   updating environment:
#   0 added, 0 changed, 0 removed
#   looking for now-outdated files...
#   none found
#   processing FOO.tex...
#   index
#   
#   resolving references...
# Loaded extensions:
#   alabaster (0.7.10) from /zzz/envs/py27/lib/python2.7/site-packages/alabaster/__init__.pyc
Traceback (most recent call last):
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/cmdline.py", line 306, in main
    app.build(opts.force_all, filenames)
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/application.py", line 338, in build
    self.builder.build_update()
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/builders/__init__.py", line 323, in build_update
    self.build(['__all__'], to_build)
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/builders/__init__.py", line 394, in build
    self.write(docnames, list(updated_docnames), method)
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/builders/latex.py", line 140, in write
    appendices=((docclass != 'howto') and self.config.latex_appendices or []))
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/builders/latex.py", line 190, in assemble_doctree
    self.env.resolve_references(largetree, indexfile, self)
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/environment/__init__.py", line 938, in resolve_references
    self.apply_post_transforms(doctree, fromdocname)
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/environment/__init__.py", line 951, in apply_post_transforms
    transformer.apply_transforms()
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/transforms/__init__.py", line 88, in apply_transforms
    Transformer.apply_transforms(self)
  File "/zzz/envs/py27/lib/python2.7/site-packages/docutils/transforms/__init__.py", line 170, in apply_transforms
    transform = transform_class(self.document, startnode=pending)
  File "/zzz/envs/py27/lib/python2.7/site-packages/docutils/transforms/__init__.py", line 56, in __init__
    document.settings.language_code, document.reporter)
  File "/zzz/envs/py27/lib/python2.7/site-packages/docutils/languages/__init__.py", line 45, in get_language
    'Docutils-generated text will be in English.')
  File "/zzz/envs/py27/lib/python2.7/site-packages/docutils/utils/__init__.py", line 220, in warning
    return self.system_message(self.WARNING_LEVEL, *args, **kwargs)
  File "/zzz/envs/py27/lib/python2.7/site-packages/docutils/utils/__init__.py", line 191, in system_message
    self.stream.write(msg.astext() + '\n')
  File "/zzz/envs/py27/lib/python2.7/site-packages/docutils/utils/error_reporting.py", line 203, in write
    self.encoding_errors))
  File "/Users/xxxxxx/yyyyyy/sphinx/sphinx/util/nodes.py", line 45, in write
    self.warnfunc(self._re.sub(r'\1:', text), None, '')
TypeError: 'NoneType' object is not callable

[jfbu]

ping @mitya57 what's your opinion on this issue ?

[mitya57]

I see two problems here:

1. thousands of lines similar to [exec] C:\foo\foobar.rst:28: WARNING: No smart quotes defined for language "tr".

   Turkish quotes support was added to docutils in revision 8017. However, Sphinx should not attempt to use smart quotes at all for older versions. The language in smartchars.quotes check in sphinx/environment/__init__.py should fail, and the default Docutils value (False) should be used. It would be nice if you could figure out why that does not work — maybe you have a custom docutils.conf somewhere?

2. TypeError: 'NoneType' object is not callable. This looks like a generic error related to messages reporting, nothing specific to smart quotes. Maybe some core Sphinx developer will look into it?

[jfbu]

@mitya57

1. I did not reproduce these messages indeed. However I noticed that Sphinx tries to apply the smart quotes with tr because if I add to conf.py the monkey-patch indicated in my comment where I copied over the quotes['tr'] from Docutils revision 8017 then it does work both for html and latex builders.

2. I wonder to what extent it has to do with the change in Sphinx 1.6.1 that some console output messages are localized. There was a report https://groups.google.com/d/msg/sphinx-users/B6PHvgm6fMM/s4h0eoEqCQAJ at google group sphinx-users. My Python literacy being weak I am mentioning this in passing only.

[Znni]

I got the TypeError: 'NoneType' object is not callable error also once before, but now I'm not able to reproduce it.

I'll try to see why the check in sphinx/environment/__init__.py fails...

[Znni]

I still tried to reproduce the issue and noticed the following:

• If I delete the old contents of my build directory and build Turkish latex with other languages (the builds are sequential, not parallel), I get the original warnings I reported, i.e. both WARNING: C:\foo\bar.rst:: (WARNING/2) language "tr" not supported: Docutils-generated text will be in English. and thousands of lines of WARNING: No smart quotes defined for language "tr"..

• If I delete the old contents of my build directory and build only Turkish latex, I get only WARNING: C:\foo\bar.rst:: (WARNING/2) language "tr" not supported: Docutils-generated text will be in English., but not the smart quote warnings.

• If I do not delete the old contents of my build directory and build only Turkish latex, I constantly get every other time only WARNING: C:\foo\bar.rst:: (WARNING/2) language "tr" not supported: Docutils-generated text will be in English. and every other time the error TypeError: 'NoneType' object is not callable.

I'll try to dig deeper into the python scripts when I have time...

[Znni]

One finding: The language in smartchars.quotes check in sphinx/environment/__init__.py works fine also in the build where I build Turkish latex with other languages. The check fails but regardless of that, I get thousands of WARNING: No smart quotes defined for language "tr". in this case.

And no, I don't have a custom docutils.conf file.

[mitya57]

Oh, so the setting is cached in the environment, and the environment is not reset when switching languages. A workaround would be to to use --fresh-env every time when you switch the language (or delete doctrees/environment.pickle manually).

I do not know how to fix this in Sphinx — can someone please enlighten me?

[jfbu]

I confirm your findings, and the fact that my suggested addition to conf.py in comment above removes the warnings about non availability of smart quotes for Turkish, but not the

 (WARNING/2) language "tr" not supported: Docutils-generated text will be in English.

I have a small project for which I generated via make gettext an index.pot file from which I generated via msgfmt English and Turkish translations in suitable locale/<language>/LC_MESSAGES/index.mo files, with locale_dirs = ['locale/'] in conf.py. I also have language='tr' in conf.py so make html uses by default the tr translated strings.

1. If I execute make -e SPHINXOPTS="-D language='en'" html, then simply make html I get something like

   WARNING: /Users/testtest/3788smartquotes/index.rst:: (WARNING/2) language "tr" not supported: Docutils-generated text will be in English.
   /Users/testtest/3788smartquotes/index.rst:7: WARNING: No smart quotes defined for language "tr".
   /Users/testtest/3788smartquotes/index.rst:10: WARNING: No smart quotes defined for language "tr".
   /Users/testtest/3788smartquotes/index.rst:12: WARNING: No smart quotes defined for language "tr".
   /Users/testtest/3788smartquotes/index.rst:14: WARNING: No smart quotes defined for language "tr".
   

2. If I then do rm _build/html/index.html and execute make html I get the

   Exception occurred:
    File "/Users/xxx/sphinx/sphinx/util/nodes.py", line 45, in write
      self.warnfunc(self._re.sub(r'\1:', text), None, '')
   TypeError: 'NoneType' object is not callable
   

But now if I add this to top of conf.py:

import re
from sphinx.util.smartypants import smartquotes
from sphinx.util.docutils import __version_info__ as docutils_version

if docutils_version < (0, 13, 2):
    # print("hack")
    quotes = smartquotes.smartchars.quotes
    quotes['tr'] = u'“”‘’'
    quotes['tr-x-altquot'] = u'«»‹›'

in case 1., I suppress all warnings except the

 (WARNING/2) language "tr" not supported: Docutils-generated text will be in English.

and in case 2., I do not reproduce anymore the

'NoneType' object is not callable

error. Can you try the above addition to conf.py and confirm it reduces the emitted warnings ?

[jfbu]

... but I still got the error on make latexpdf though.

[jfbu]

by the way @tk0miya the

__version_info__ = tuple(map(int, docutils.__version__.split('.')))

at https://github.com/sphinx-doc/sphinx/blob/stable/sphinx/util/docutils.py#L37 is a bit annoying because the int makes it impossible to test with a checkout of development version of docutils (link for git at http://docutils.sourceforge.net/docs/dev/repository.html) which has something like 0.13.2a as version string.

[Znni]

@jfbu I can confirm that by making the changes you proposed to conf.py removes the WARNING: No smart quotes defined for language "tr" warnings (in Turkish build) in building latex for all languages.

However, as you noticed, too, (WARNING/2) language "tr" not supported: Docutils-generated text will be in English. is still present and by running the Turkish latex build twice, 'NoneType' object is not callable is also still present.

[jfbu]

tidbit of information: the presence of the line https://github.com/sphinx-doc/sphinx/pull/3666/commits/edc7f30b9cc92f4d721eeef62ffa826f23a09a9f#diff-97700a3f4e77714edda29526a159758fR676 triggers at some point docutils.languages.get_language and this is the cause of the Docutils-generated text will be in English..

[jfbu]

It appears that with 1.5.6, when the read_doc() environment method is executed the docutils processing does not go via docutils.languages.get_language. If one adds simply the two lines

        language = (self.config.language or 'en').replace('_', '-')
        self.settings['language_code'] = language

in 1.5.6's class BuildEnvironment.read_doc() method one does trigger the execution of docutils.languages.get_language during environment doctree setting up. This then raises the Docutils warning as observed with 1.6.1.

[jfbu]

My above comment was not precise enough. I absurdly wrongly said docutils.languages.get_language was not executed during environment parsing and doctree construction with Sphinx 1.5.6. In fact it is, but with language_code set to 'en'. I added a print statement and a counter to the Docutils code (print(language_code, COMPTE)) and here is what I get on a test document (with language = 'tr') in conf.py:

(it has extra version info for debugging purposes)

Running Sphinx v1.5.6
# Sphinx version: 1.5.6
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6

loading translations [tr]... done
loading pickled environment... failed: build environment version not current
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
('en', 1)ources... [100%] index                                                 
('en', 2)
('en', 3)
('en', 4)
('en', 5)
('en', 6)
('en', 7)
('en', 8)
('en', 9)
('en', 10)
('en', 11)
('en', 12)
('en', 13)
('en', 14)
('en', 15)
('en', 16)
('en', 17)
('en', 18)
('en', 19)
('en', 20)
('en', 21)
('en', 22)
('en', 23)
('en', 24)
('en', 25)
('en', 26)
('en', 27)
('en', 28)
('en', 29)
('en', 30)
('en', 31)
('en', 32)

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
('en', 33)tput... [100%] index                                                  
('en', 34)
('en', 35)
('en', 36)
('en', 37)
('en', 38)
('en', 39)
('en', 40)
('en', 41)
('en', 42)
('en', 43)
('en', 44)
('en', 45)
('en', 46)
('en', 47)
('en', 48)
('en', 49)
('en', 50)
('en', 51)
('en', 52)
('en', 53)
('en', 54)
('en', 55)
('en', 56)
('en', 57)
('en', 58)

generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in Turkish (code: tr) ... done
dumping object inventory... done
build succeeded.

Now the same with 1.6.1, but I commented out the Docutils generation of a warning in docutils.languages.get_language():

Running Sphinx v1.6.1
# Sphinx version: 1.6.1
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6

loading translations [tr]... done
loading pickled environment... failed: build environment version not current
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
('tr', 1)ources... [100%] index                                                 
('tr', 2)
('tr', 3)
('tr', 4)
('tr', 5)
('tr', 6)
('tr', 7)
('tr', 8)
('tr', 9)
('tr', 10)
('tr', 11)
('tr', 12)
('tr', 13)
('tr', 14)
('tr', 15)
('tr', 16)
('tr', 17)
('tr', 18)
('tr', 19)
('tr', 20)
('tr', 21)
('tr', 22)
('tr', 23)
('tr', 24)
('tr', 25)
('tr', 26)
('tr', 27)
('tr', 28)
('tr', 29)
('tr', 30)
('tr', 31)
('tr', 32)
('tr', 33)

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
('tr', 34)tput... [100%] index                                                  
('tr', 35)
('tr', 36)
('tr', 37)
('tr', 38)
('en', 39)
('en', 40)
('en', 41)
('en', 42)
('en', 43)
('en', 44)
('en', 45)
('en', 46)
('tr', 47)
('tr', 48)
('en', 49)
('en', 50)
('en', 51)
('en', 52)
('en', 53)
('en', 54)
('en', 55)
('en', 56)
('en', 57)
('en', 58)
('en', 59)
('en', 60)
('en', 61)
('en', 62)
('en', 63)
('en', 64)
('en', 65)

generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in Turkish (code: tr) ... done
dumping object inventory... done
build succeeded.

The difference is that language_code is now 'tr' on first calls to docutils.languages.get_language().

The change at https://github.com/sphinx-doc/sphinx/commit/edc7f30b9cc92f4d721eeef62ffa826f23a09a9f#diff-97700a3f4e77714edda29526a159758fR676 explains the language_code. There seems to be a timing problem with when language_code should be set.

[jfbu]

For the time being, I can propose this fix to get rid of the Docutils warning. In my brief testing with the additions proposed below to conf.py, the smart quotes are processed ok for languages which support them, and ignored for languages which do not support them (there is still the issue when making successive builds with different languages, such as 'en' then 'tr' I reproduce earlier fix at bottom).

In your conf.py add at top:

from docutils.languages import get_language
from docutils.languages import en as du_en_module

_languages = {'en': du_en_module}

and at bottom:

def patched_get_language(language_code, reporter=None):
    """Original code returned module with language localizations.

    It was always called with language_code='en' by Sphinx < 1.6
    The language_code is now set earlier in sphinx.environment.read_doc()
    in order to be taken into account by Docutils for smart quotes.
    But no warning should be issued by docutils.languages.get_language().
    """
    return _languages['en']

def setup(app):
    docutils.languages.get_language = patched_get_language

I hope this does the right thing (I am a bit shaky about Python import), but beware I am not core Python developer...

This resolves in my testing the issue with language "tr" not supported: Docutils-generated text will be in English. with seemingly no nefarious consequences for internationaliztion (at least for html and latexpdf)

For the other problem with building documents with English, then with Turkish and getting

WARNING: No smart quotes defined for language "tr".

you can get rid of it with this further addition at top of conf.py, as I mentioned in earlier comment:

from sphinx.util.smartypants import smartquotes
from sphinx.util.docutils import __version_info__ as docutils_version

and at bottom, additionally to the above:

def setup(app):
    docutils.languages.get_language = patched_get_language
    if docutils_version < (0, 13, 2):
        quotes = smartquotes.smartchars.quotes
        quotes['tr'] = u'“”‘’'
        quotes['tr-x-altquot'] = u'«»‹›'

This should temporarily fix your issues. But definitely a core fix at 1.6.2 is to be hoped...
(@tk0miya ...)

@mitya57 do you have any comments ;-) ? I have tested the above additions to conf.py with 1.6.1.

[tk0miya]

I believe #3808 might resolve the warning. Could you try it please?
Is there anything else?

Note:

• The TypeError is fixed at ffb84cb
• __version_info__ will be fixed in another commit (I will do it later)

[jfbu]

In my testing it does resolve the warning. But if one does

make clean
make -e SPHINXOPTS="-D language='en'" html
make -e SPHINXOPTS="-D language='tr'" html

one gets the English smart quotes applied to Turkish build, from previous environment. Under

make clean
make -e SPHINXOPTS="-D language='tr'" html

no smart quotes are applied because 'tr' is not recognized as "supported" by the #3808 fix.

The secondary thing is pointed out at https://github.com/sphinx-doc/sphinx/pull/3808#pullrequestreview-40531550: the Docutils seems to have support for smart quotes for more languages than those recognized as being supported by the docutils.languages.get_language(). So it seems we lose opportunity to get the smart quotes for those languages. But perhaps this Docutils issue will evolve.

side note: I tested also with latexpdf but then there is unrelated pure LaTeX problem that LaTeX babel complains when one switches languages without having cleaned auxiliary files. Thus one should make -C _build/latex clean to test effect of a make latexpdf with a new language (keeping previous _buid/doctrees for testing).

[tk0miya]

Thanks, I did not noticed the cached behavior. I also fixed it on #3808.
Could you try it again?

[jfbu]

The cached behavior is fixed in my testing with html, and switching between 'en', 'fr', 'tr', and 'et' languages. The latter two are not supported by Docutils get_language() but 'et' has smart quotes in 0.13.1. But they are not applied of course with #3808.

[jfbu]

about

and then thousands of lines similar to

[exec] C:\foo\foobar.rst:28: WARNING: No smart quotes defined for language "tr".

there should have been only one such line. This is an issue in Docutils 0.13.1 which I have reported upstream on their dev mailing list.

This being said, the fix to this issue will make sure no such warning are emitted. As per tr next release of Docutils has the smart quotes for it.

[tk0miya]

Fixed at #3813 by @jfbu (Thanks!)

[mitya57]

I have tested the latest version in stable branch and it works fine for me. @jfbu @tk0miya thanks for your work (and sorry for not replying earlier).

[jfbu]

Thanks @mitya57 ! but we have another issue with smart quotes see PR #3816

[Znni]

Thanks everybody, with Sphinx 1.6.2 all my builds finish without warnings or errors!