Corefxlab: Encoding from UTF-32 to UTF-8 does not correctly handle unpaired surrogates

[ahsonkhan]

From: https://github.com/dotnet/corefxlab/pull/1508

The surrogate area, i.e. codepoint values between D800 and DFFF are invalid codepoints, unless they are paired.
https://en.wikibooks.org/wiki/Unicode/Character_reference/D000-DFFF#endnote_SURROGATE

Our implementation does not return false for incorrect code points.
The test EncodeIntoUtf8Tests.InputBufferEndsTooEarlyAndRestart fails for UTF-32 because of this reason.

cc @KrzysztofCwalina, @shiftylogic

This sample app shows where we should be returning false (range D800-DFFF ~and some of FFF0–FFFF):
https://en.wikipedia.org/wiki/Specials_(Unicode_block)~
C# public static void MyTable() { Encoding enc = new UTF32Encoding(false, true, true); for (int i = 0; i < 0x110000; i++) { byte[] intBytes = BitConverter.GetBytes(i); try { var x = Encoding.Convert(Encoding.UTF32, Encoding.UTF8, intBytes); if (x.Length == 3) { // Fall back - 239 191 189 (i.e. EF BF BD) if (x[0] == 0xEF && x[1] == 0xBF && x[2] == 0xBD) Console.WriteLine("Index: " + i); } } catch(Exception ex) { Console.WriteLine("Index: " + i + " : " + ex.Message); return; } } }
Edit: if (x[0] == 0xEF && x[1] == 0xBF && x[2] >= 0xB9 && x[2] <= 0xBF) (Fall back - 239 191 190 (i.e. EF BF BE)) was incorrect, the fall back character is EF BF BD, i.e. if (x[0] == 0xEF && x[1] == 0xBF && x[2] == 0xBD)

[ahsonkhan]

For reference:
Encoding code points within the range 0xD800 to 0xDFFF to UTF-8 should be invalid and return false.

Example:
ED A0 80 (1110 1101 1010 0000 1000 0000, i.e. 0xD800) is technically valid UTF-8 sequence, but it should be treated as invalid since it is in the surrogate range.

https://en.wikipedia.org/wiki/UTF-8#Invalid_code_points
Since RFC 3629 (November 2003), the high and low surrogate halves used by UTF-16 (U+D800 through U+DFFF) and code points not encodable by UTF-16 (those after U+10FFFF) are not legal Unicode values, and their UTF-8 encoding must be treated as an invalid byte sequence.
Not decoding surrogate halves makes it impossible to store invalid UTF-16, such as Windows filenames, as UTF-8. Therefore, detecting these as errors is often not implemented and there are attempts to define this behavior formally (see WTF-8 and CESU below).

https://tools.ietf.org/html/rfc3629

The definition of UTF-8 prohibits encoding character numbers between
U+D800 and U+DFFF, which are reserved for use with the UTF-16
encoding form (as surrogate pairs) and do not directly represent
characters. When encoding in UTF-8 from UTF-16 data, it is necessary
to first decode the UTF-16 data to obtain character numbers, which
are then encoded in UTF-8 as described above.

[shiftylogic]

The new implementation is correct. We now return InvalidData status for this scenario.

[ahsonkhan]

Do we have a test case that covers this scenario now?

[shiftylogic]

This test should cover it.