Hermes: Unicode line terminators not supported in string literals

[djschilling]

There are 4 Unicode caracters that i know of that are not supported by hermes:

• https://www.compart.com/de/unicode/U+2028
• https://www.compart.com/de/unicode/U+2029
• https://www.compart.com/de/unicode/U+1680
• https://www.compart.com/de/unicode/U+2000

There are probably more, but these i had problems with:

Executing this with hermes leads to these errors:
var foo = "
";

test.js:1:12: error: non-terminated string
var foo = "
";
test.js:1:11: note: string started here
var foo = "
";
test.js:1:17: error: non-terminated string
var foo = "
";
test.js:1:15: note: string started here
var foo = "
";

The unicode Character used in this example is U+2028.

The problem occurred to me using this lib: https://github.com/nodeca/unhomoglyph/blob/master/data.json#L160

[tmikov]

Hermes doesn't try to interpret most Unicode characters, so there shouldn't be anything to support. But there could be bugs, of course.

What is the encoding of the input? Can you post the contents of a small example as binary or upload it somewhere?

[djschilling]

Here is a file that produces the error described above: https://gist.github.com/djschilling/12605071bdfafd5d817ece1d12911d8c

In the browser it seems like its an empty string, but actually there is the Unicode Character U+2028 in the string.

[puzrin]

Do you load js file to hermes directly or it passes server/browser? In second case problem may be related to bad server encoding (or missed charset meta). If related to encoding, there are 2 possibilities:

• fix encoding (preferred)
• force app builder use ASCII (usually all minifiers have such option)

[tmikov]

This is a subtle one. I didn't realize this immediately, but we consider \u2028 and \u2029 to be line terminators, that's why we are reporting a non-terminated string error. Here is a link to the ES9 spec describing our current behavior:
https://www.ecma-international.org/ecma-262/9.0/index.html#sec-literals-string-literals

This was changed in ES10, so \u2028 and \u2029 are now considered valid in a string, precisely for compatibility with JSON. We have not caught up with ES10 yet, but I think we can easily implement this specifically. Thanks for reporting it!

About the other characters from your list: what errors are you getting with them? When I tried them, they worked fine.

[djschilling]

@puzrin

I execute it like that: ./bin/hermes unicode-hermes.js

@tmikov You are right only \u2028 and \u2029 are causing the issue. Thanks for investigating. I'm looking forward to a new version where it will be fixed.

[tmikov]

@djschilling the fix landed and will be included in 0.6.0 as well as in the next patch release 0.5.1.