Jupyter_contrib_nbextensions: python-markdown does not escape string content

[Krastanov]

The following two examples:

a = '<strong>blah</strong>'
b = ["one", "two", "three"]

do not seem to work properly.

The first one: {{a}} results in

blah _instead of_

The second one: {{b}} results in

one _instead of_ ["one", "two", "three"]

[Krastanov]

Also, this works b = '["test", "test"]' while this does not work b = "['test', 'test']"

[jcb91]

I'm afraid that the python markdown nbextension uses regex replacement rather than an extension to the markdown parser, so this is to some extent expected. Could you clarify what you mean by

this does not work b = "['test', 'test']"

please? What do you expect to happen, and what actually happens?

[Krastanov]

"['test', 'test']" results in test

[Krastanov]

I would expect this nb_extension to pass the str representation through something like an html or xml escaping encoder before submitting it to the rest of the pipeline. Is this escaping pass unreasonable?

[jcb91]

Not completely unreasonable, just not what is currently done. The main issue I can see is in deciding what should be escaped, and what shouldn't. After all, the HTML only works because it's part of the markdown. So should we also escape markdown control sequences? We could perhaps add an optional escaped/non-escaped distinction with different tags, but then this starts to get complicated. I'd be reluctant to escape everything all the time, as it can be useful to allow direct markdown/HTML output to allow people to create things like lists and tables. Any suggestions for an escaping method you'd like to see used?

[Krastanov]

I would have said "escape everything", because I view this extension as having only the job of showing the content of a variable. I would use "IPython.display" for everything else. I do believe this is a good distinction (i.e. never using this extension if IPython.display is sufficient). But this is just my view.

[jcb91]

Well, the alternative would be to implement both behaviours, and have them controlled by a configuration setting, I guess. For a quick fix for now, you can probably fix most things by just wrapping your variable in markdown code tags, e.g.

`{{a}}`

[Krastanov]

That wrapping still does not work for b = ["one", "two", "three"]. As in {{["one", "two", "three"]}} results in one

[jcb91]

Ok, the list thing, I guess is some side effect of the markdown parsing, though I don't know without reviewing the extension code, (which I can't easily do right now as I'm on a phone only, in the middle of reinstalling OS). I'll let you know if anything occurs to me...

[juhasch]

You are right, there is no proper escaping. I was only expecting simple repl outputs (including html, graphics, etc). I can't promise to fix this very soon.

[jcb91]

having looked at this again, I believe it's a consequence of attempted escaping going wrong, rather than no escaping being done - see the regex on python-markdown/main.js#L95 which, as it's lazy, ends up matching after just the _first_ string-like section in the list.

It's not easy to un-escape strings in all cases, so I think the best solution is not to try, and then if people want the string as it would be printed, they can wrap it in a print statement, as {{print(my_string)}}, since the printed output gets captured too, without the quotes. As for whether to run markdown over things, I guess it might make sense to have a second template syntax (e.g. like jinja's {% if foo %} type construct) for things which should _not_ be run through the marked stack. Thoughts?

[juhasch]

I think simply improving the quote stripping would help in this case.
If you remove the lines

var q = html.match(/'([\s\S]*?)'/); // strip quotes from strings
if (q !== null) html = q[1]

The above examples work. Only now you have quotes sorrounding a string, and this is ugly if you want to things like `How are you {{ yourname }}'.

[jcb91]

I think simply improving the quote stripping would help in this case.

It would, yes

The above examples work. Only now you have quotes sorrounding a string, and this is ugly if you want to things like `How are you {{ yourname }}'.

Not really, because you can do How are you {{print(yourname)}} (for python, would be different in other kernels)

[juhasch]

I know, it's just that I am lazy...

[jcb91]

haha! ok, well we could always make it configurable, too :stuck_out_tongue_winking_eye:

[juhasch]

I think improving the regex on only stripping outer quotes will be good for most cases. Not sure if anyone would ever bother to configure this...
I'll think about this tomorrow.

[Krastanov]

Is it not simpler to just do HTML escaping with import html; html.escape (cgi instead of html for older pythons)? No configuration necessary, no regex, and it works always. If anybody wants something fancier (like purposefully placing tags in their output in order to format it) they would (and should) just use import IPython.display, which would work much better for that task anyway.

One simple good tool per task, instead of multiple complicated configurable tools that do the same is part of the pythonic way, right ;)

[jfbercher]

@Krastanov Certainly, but not easily here since the nbextensions are written in javascript, not in python, and are also, usually, intended to be language agnostic, ie work for any kernel (this is the case with python-markdown where the python is for "historical" reasons).

[jcb91]

I see this as essentially two competing paradigms for what the desired functionality is:

1. @juhasch's: a way of adding variable text into my markdown, to be interpreted as richly as possible by adding in images, interpreting markdown, and all the rest of the usual processing
2. @Krastanov's:
   > I view this extension as having only the job of showing the content of a variable

@Krastanov I don't really follow how this usage would offer advantages over just using regular kernel code in a cell to output the string representation, but perhaps you could elaborate?

[Krastanov]

It is about showing the content of a variable in the flow of text, not just showing it on its own in a big cell dedicated only to it. This is the only reason I would use this extension - for anything fancier I would just use IPython.display. I have nothing against adding more features on top of that, as described by @juhasch , but right now (due to the lack of escaping) this base-level feature does not always work.

[Krastanov]

And even with those fancier features discussed above, the need for escaping is still there - when you build a full templating engine in this extension you will still need to address bugs where a user wants to show something containing quotes or other special characters.

[lordascor]

Hi, I think my problem might be related to this issue.

I found out that the escaping issue has variable behaviour:

• if you have only one \' in the string everything works fine

'here\'s a test' --> here's a test

• if you have several \' in the string, it's chaos (it only displays what's between the first two \')

'here\'s a test, and here\'s another test' --> s a test, and here

Have you found a workaround or how to fix this since last February ?