Notebook: Python 3.6 f-strings break notebook syntax highlighting

[bede]

Using Notebook v4.3.1, use of f-strings (PEP 498, Python 3.6) breaks syntax highlighting for the remaining code in a given cell.

Is Jupyter curently unable to handle this syntax? I would imagine that this issue would have been raised by others were this the case.

Using OS X, brewed Python 3.6.0, Firefox 50.

$ python3 --version
Python 3.6.0
$ jupyter --version
4.2.1
$ jupyter-notebook --version
4.3.1
$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.12.2
BuildVersion:   16C68

[takluyver]

It's an issue with Codemirror, the editor component. It has been fixed (codemirror/CodeMirror#4182), but we had to stick with an old version of Codemirror to work around an issue with Safari (#2004).

@gnestor @minrk what's the plan for getting back to a new version of Codemirror? Do we know if 5.22.1 fixes it?

[bede]

Thanks, at least the fix is trivial.

Could we at least patch the existing Codemirror version? F-strings are the big ticket item in Python 3.6 after all.

[takluyver]

The JS files are built into one big concatenated file. We don't have a good way to carry patches against upstream JS projects, as far as I know.

[bede]

Ah that's a pity, but understandable @takluyver. I've just grep'ed all js files in site-packages for rbu (safe from minification since it's a string?) and can't find this concatenated JS you mention... For the sake of myself and others patching it, please could tell me its path?

[takluyver]

Should be notebook/static/notebook/js/main.min.js, I think

[minrk]

@gnestor the CodeMirror issue was closed, I think due to a comment that the bug wasn't reproducible outside the notebook. Does that mean it's due to our CSS somewhere?

[bede]

Patch

Temporary solution tested on notebook version 4.3.1

Two patterns inside site-packages/notebook/static/notebook/js/main.min.js need replacing: [rub] with [rubf] and "rub" with "rubf"

With sed:

sed -i 's/'"rub"'/'"rubf"'/g' -i 's/\[rub\]/\[rubf\]/g' /usr/local/lib/python3.6/site-packages/notebook/static/notebook/js/main.min.js

[gnestor]

@minrk @takluyver I just tested 5.22.2 in Safari and the same issue as before. @jasongrout fixed this in lab by updating the CSS. I tried changing .CodeMirror-lines from padding: 0.4em to padding: 5px but no luck...

@bede I can't find a[rub] or "rub" in the source...

[bede]

@gnestor Lines 65, 73 and 188 of https://github.com/codemirror/CodeMirror/blob/97eb5221f05c09cfeafefa45b52da022fbcc46af/mode/python/python.js

rubf and rbuf are the relevant updated strings

[takluyver]

I've marked this as 5.0 - I'd really like to get it fixed soon, because syntax highlighting is currently broken with the new Python 3.6 strings.

Are we aware of any fix for the Safari issue in CodeMirror that would let us update it again? If not, can we work out a way to use a patched CodeMirror?

[gnestor]

@takluyver Agreed! I've spent some time investigating the Safari issue and what I've concluded is that it was introduced in CodeMirror 5.17. To quote from https://github.com/codemirror/CodeMirror/issues/4454#issue-196746612:

Here is a list commits included in 5.17: https://github.com/codemirror/CodeMirror/commits/master?after=Y3Vyc29yOjiTKo4zLKAvc9OV%2BJnP36dpIuEyKzIwOQ%3D%3D

I have a feeling that this commit might be the culprit although I haven't been able to resolve this by simply reversing the changes in this commit: https://github.com/codemirror/CodeMirror/commit/2c913e5d2914fdc64df80031544a384940006feb

I will keep trying to isolate the issue. If it's not resolved by 5.0 release time, let's try patching CodeMirror.

[oldgeeksguide]

Thanks for the detailed info on the patch!

[gnestor]

Ok, I've isolated this bug to this commit: https://github.com/codemirror/CodeMirror/commit/e60f4b7dd88b71e227f79200932c028ca047d78e

We'll see if we can fix it on the CM side... @takluyver Any idea why this commit is causing this behavior?

[ruxi]

whats the recommended workaround for this?

[gnestor]

You can try @bede's patch: https://github.com/jupyter/notebook/issues/2037#issuecomment-272466046

[gnestor]

Notebook 4.3.2 was just published 👍