Sphinx: Issue with intersphinx with latest developer version of Sphinx

[astrofrog]

With the latest developer version of Sphinx (and 1.5) I see the following warnings/errors:

Running Sphinx v1.6
loading pickled environment... failed: build environment version not current
loading intersphinx inventory from http://docs.astropy.org/en/stable/objects.inv...
WARNING: intersphinx inventory 'http://docs.astropy.org/en/stable/objects.inv' not readable due to ValueError: unknown or unsupported inventory version
loading intersphinx inventory from http://matplotlib.org/objects.inv...
WARNING: intersphinx inventory 'http://matplotlib.org/objects.inv' not readable due to ValueError: unknown or unsupported inventory version
loading intersphinx inventory from https://docs.scipy.org/doc/numpy/objects.inv...
loading intersphinx inventory from http://www.sphinx-doc.org/en/latest/objects.inv...
WARNING: intersphinx inventory 'http://www.sphinx-doc.org/en/latest/objects.inv' not readable due to ValueError: unknown or unsupported inventory version
loading intersphinx inventory from https://docs.python.org/2.7/objects.inv...

This works fine with 1.4.8. Unfortunately many of these intersphinx objects.inv files are beyond my control, so I cannot simply re-generate them all with Sphinx 1.5. This seems like a pretty big backward-incompatibility. Would it be possible to restore backward-compatibility in 1.5.1?

[tk0miya]

At my local, I can't reproduce the problem.

With sphinx-1.5:

sphinx-build -b html -d _build/doctrees  -T . _build/html
Running Sphinx v1.5
making output directory...
loading translations [ja]... done
loading pickled environment... not yet created
loading intersphinx inventory from http://www.sphinx-doc.org/en/latest/objects.inv...
loading intersphinx inventory from http://docs.astropy.org/en/stable/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/numpy/objects.inv...
loading intersphinx inventory from http://matplotlib.org/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 4 source files that are out of date
updating environment: 4 added, 0 changed, 0 removed
reading sources... [100%] qux
looking for now-outdated files... none found
pickling environment... done
checking consistency... /Users/tkomiya/work/tmp/doc/qux.rst:: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] qux
generating indices... genindex
writing additional pages... search
copying images... [100%] logo.jpg
copying static files... done
copying extra files... done
dumping search index in Japanese (code: ja) ... done
dumping object inventory... done
build succeeded, 1 warning.

Build finished. The HTML pages are in _build/html.

With master HEAD:

sphinx-build -b html -d _build/doctrees  -T . _build/html
Running Sphinx v1.6
making output directory...
loading translations [ja]... done
loading pickled environment... not yet created
loading intersphinx inventory from http://www.sphinx-doc.org/en/latest/objects.inv...
loading intersphinx inventory from http://docs.astropy.org/en/stable/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/numpy/objects.inv...
loading intersphinx inventory from http://matplotlib.org/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 4 source files that are out of date
updating environment: 4 added, 0 changed, 0 removed
reading sources... [100%] qux
looking for now-outdated files... none found
pickling environment... done
checking consistency... /Users/tkomiya/work/tmp/doc/qux.rst:: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] qux
generating indices... genindex
writing additional pages... search
copying images... [100%] logo.jpg
copying static files... done
copying extra files... done
dumping search index in Japanese (code: ja) ... done
dumping object inventory... done
build succeeded, 1 warning.

Build finished. The HTML pages are in _build/html.

Note: my settings

intersphinx_mapping = {
    'astropy': ('http://docs.astropy.org/en/stable/', None),
    'matplotlib': ('http://matplotlib.org/', None),
    'numpy': ('https://docs.scipy.org/doc/numpy/', None),
    'python': ('http://www.sphinx-doc.org/en/latest/', None),
}

[tk0miya]

Note:
Since 1.4.8, the format of inventory is not changed. So I believe this is not a compatibility problem.

The reported warnings came from some ValueError on read_inventory() even if the exception is not related with inventory version.

            try:
                join = localuri and path.join or posixpath.join
                invdata = read_inventory(f, uri, join)
            except ValueError:
                raise ValueError('unknown or unsupported inventory version')

To investigate more deeply, we have to know what exception did read_inventory() throw.

[tk0miya]

Finally, I updated error message to investigate the error at 4047329.
This will report the error more detail.

@astrofrog Could you try this please?

[AlexanderS]

I have the same problem:

SphinxWarning: WARNING: intersphinx inventory 'http://www.voidspace.org.uk/python/mock/objects.inv' not readable due to ValueError: unknown or unsupported inventory version: UnicodeDecodeError('utf8', '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x00', 1, 2, 'invalid start byte')

Seems like requests does not handle gzip decoding when using .raw: https://github.com/kennethreitz/requests/issues/2155

Using the suggested wrapper for read seems to fix this issue:

--- ext/intersphinx.py  2017-01-10 19:59:20.559467922 +0100
+++ ext/intersphinx.py  2017-01-10 19:59:55.008071840 +0100
@@ -32,6 +32,7 @@
 import posixpath
 from os import path
 import re
+from functools import partial

 from six import iteritems, string_types
 from six.moves.urllib.parse import urlsplit, urlunsplit
@@ -166,6 +167,7 @@
     r = requests.get(url, stream=True, config=config, timeout=config.intersphinx_timeout)
     r.raise_for_status()
     r.raw.url = r.url
+    r.raw.read = partial(r.raw.read, decode_content=True)
     return r.raw

[tk0miya]

Fixed at cea72dd.
@AlexanderS Thank you for advice!

[thijstriemstra]

I'm still seeing this problem with 1.5.3 and python 3.5.3:

$ python -m sphinx.ext.intersphinx _build/html/objects.inv 
intersphinx inventory '_build/html/objects.inv' not readable due to ValueError: unknown or unsupported inventory version: ValueError()

And the error message is not really helpful..

[thijstriemstra]

When I add a print on line 127 it shows something with zlib and also the bare value error:

    if 'zlib' not in line:
        print('zlib')
        raise ValueError

$ python -m sphinx.ext.intersphinx _build/html/objects.inv 
zlib
intersphinx inventory '_build/html/objects.inv' not readable due to ValueError: unknown or unsupported inventory version: ValueError()

Commenting out the check completely gives a new error:

intersphinx inventory '_build/html/objects.inv' not readable due to error: Error -3 while decompressing data: incorrect header check

Any idea @tk0miya?

[tk0miya]

Hmm, more investigation is needed.
Can you share me the problematic objects.inv?

In addition, could you file a new issue?
This is already closed. so this is not good at for management.

[thijstriemstra]

thanks for the feedback @tk0miya, I opened #3488.