Highlight.js: Infinite loop for illegal tokens when `ignore_illegals` or `highlightAuto`

[wooorm]

On 10.0.1, the following XML:

<?xml version="1.0"?>
<response value="ok" xml:lang="en">
  <text>Ok</text>
  <comment html_allowed="true"/>
  <ns1:description><![CDATA[
  CDATA is <not> magical.
  ]]></ns1:description>
  <a></a> <a/>
</response>

When passed through highlightAuto or highlight with ignore_illegals turned on, will cause an infinite loop, specifically, because of '$' as an illegal in CSS selector attributes:

https://github.com/highlightjs/highlight.js/blob/e8624ccb65cfd859a0273b58ec3897afd16a5498/src/languages/css.js#L72

[joshgoebel]

Oh yikes, that's quite a find.

[joshgoebel]

@wooorm Please review fix. Do you think my catch-all is overkill? I really think we've covered everything now with being/end 0 width detection plus illegal 0 width detection. Am I just being paranoid adding a third check?

I just think freezes like this a VERY VERY bad things.

@egor-rogov

[wooorm]

Lgtm. Loop check is very heavy, but may be good indeed.

Weirdly enough, this used to work in 9.16 (or at least in the lowlight clone since when it started). I’m not sure what changed: i looked a bit through git blame and the ‘m’ flag isn’t new, and I didn’t see anything else that was weird 🤷‍♂️

[joshgoebel]

Loop check is very heavy, but may be good indeed.

Heavy handed perhaps. I'm not sure it's that CPU heavy. I run a few test iterations and couldn't see a measurable difference with the typical marge of changes I usually see.

Oh it could probably be faster if we just started iterations with a negative # and then compared without the multiplication... but now I think i'm micro-optimizing without a real problem. :)

[joshgoebel]

Weirdly enough, this used to work in 9.16

Yeah, not worth it to figure out why... something prolly changed, but 0 width matches with illegal is definitely an edge case that needs to be handled to make sure the cursor advances.

[wooorm]

Totes! I wasn’t worried about the performance, more code wise heavy!