Nbconvert: No Pygments lexer found for "ipython3"

[ResidentMario]

I'm not sure whether to raise this here or over at sphinx. Here goes.

I'm writing documentation for a library. The tutorials for the documentation I am writing in a handful of Jupyter notebooks, which I then emit to the rest of the documentation using nbconvert. Since this is tedious to do manually, I've written a small script that does it for me. The core of it is just jupyter nbconvert with a --to rst argument.

Everything was working until I updated some things today and discovered that my notebook code cells were no longer displaying.

Here is an example of a page which should have multiple code blocks on it . As you can see, the output of each cell displays just fine, but the input has disappeared.

To verify this, click on the Source button on the UI; this will bring you to here, which shows that the code is still embedded in the RST representation:

.. code:: ipython3

    boroughs.crs




.. parsed-literal::

    {'init': 'epsg:4326'}

But that it disappears on the HTML source, which gets translated to:

<code class="docutils literal"><span class="pre">crs</span></code> attribute:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s1">&#39;init&#39;</span><span class="p">:</span> <span class="s1">&#39;epsg:4326&#39;</span><span class="p">}</span>
</pre></div>
</div>

Checking the console output from when I run make html to build the docs, I see that indeed I am getting numerous warnings to the effect of:

No Pygments lexer found for "ipython3"

Note that all of this worked circa a month ago.

My best guess is that the sphinx pygments lexar has dropped support for ipython3 due to IPython 3.x being comparatively ancient technology at this point. However, nbconvert has not been updated to reflect this fact.

Perhaps changing the export prompt to ipython4 or jupyter (?) would work.

[ResidentMario]

OK, a quick test shows that neither ipython4 nor jupyter work as prompts. Ideas?

[ResidentMario]

@mgeier FYI wrt nbsphinx#24.

[takluyver]

ipython3 is the correct name - it means Python 3 + IPython's special syntax. We started using the name around IPython 0.12, when it seemed like IPython version 3.0 was a very long way off.

Pygments, which does the syntax highlighting, tries to look up a lexer that is provided by IPython.

My guess would be that you have IPython and Sphinx installed in two different Pythons, or Python environments, so the one running Sphinx cannot find IPython. If that is the problem, then the fix is to either install Sphinx into the Python where you use IPython, or vice versa.

[mgeier]

Multiple users of nbsphinx had problems with that, see https://github.com/spatialaudio/nbsphinx/issues/24 and https://github.com/spatialaudio/nbsphinx/issues/27.

I don't know what's going on, since I didn't experience those problems myself. It also always worked on readthedocs.org.

A common work-around is to add 'IPython.sphinxext.ipython_console_highlighting' to extensions in conf.py.
I think this works for all of the people who had that problem, but it cannot seriously be the solution.

@takluyver I have the feeling that it is more subtle than inadvertently installing IPython and Sphinx for different Python interpreters. It probably has to do with the order of installation or something?

[takluyver]

Order of installation shouldn't matter. I suspect it might be quite common to have Sphinx on Python 2, for instance if you've installed it from apt.

Pygments also needs pkg_resources to find it, but that should be available in most Pythons.

@ResidentMario can you run this in IPython, and put it in conf.py for Sphinx to run:

```python
from pygments.plugin import find_plugin_lexers
print(list(find_plugin_lexers()))
````

[takluyver]

Also, just to check a couple of things:

import pkg_resources  # See if it fails with an ImportError
import sys
print(sys.executable)

[ResidentMario]

With regards to ipython and sphinx being in different environments, nope, this isn't the issue. Here's the relevant bit of conda list:

ipython                   5.1.0                    py35_1  
ipython_genutils          0.1.0                    py35_0
Sphinx                    1.5                       <pip>
sphinx-bootstrap-theme    0.4.12                    <pip>

With regards to the 'IPython.sphinxext.ipython_console_highlighting' workaround: I actually did try that. Didn't seem to help.

Running...

from pygments.plugin import find_plugin_lexers
print(list(find_plugin_lexers()))

...in the IPython console returned an empty list ([]).

Inserting...

import pkg_resources  # See if it fails with an ImportError
import sys
print(sys.executable)

...into conf.py and then running make html succeeds, with the following printout to console (geoplot is the name of my conda environment):

/Users/Honors/anaconda/envs/geoplot/bin/python

[takluyver]

I think I see the issue. Can you try installing the IPython built by conda-forge:

conda update ipython -c conda-forge

[ResidentMario]

This is, indeed, the fix.

FYI, the pygments I have is (still) one installed on the default channel:

pygments                  2.1.3                    py35_0

IPython is now:

ipython                   5.2.2                    py35_0    conda-forge

So (ignoring issues about the interplay between the default channel and conda-forge, which really ought to be better documented), is the issue that this version of pygments is compatible with the most recent conda-forge build, but retroactively breaks when used with a recent default build?

[takluyver]

I don't think it's an issue with pygments, but the IPython package from the default channel appears to be built in a way that doesn't install the entry points (which are how we register lexers for pygments). In conda-forge, where we control the ipython package, it is built using pip, which works.

You may want to file an issue on anaconda-issues, pointing them to the recipe on conda-forge.

[ResidentMario]

Done.