Runtime: TextEncoder misses last BMP character after surrogate pair

Created on 30 Jul 2017  路  3Comments  路  Source: dotnet/runtime

Version: 2.0.0-preview2-final
```C#
string url = System.Text.Encodings.Web.UrlEncoder.Default.Encode("饜尯X");
string html = System.Text.Encodings.Web.HtmlEncoder.Default.Encode("饜尯X");
string js = System.Text.Encodings.Web.JavaScriptEncoder.Default.Encode("饜尯X");

**Expected:**

url = "%F0%90%8C%BAX"
html = "𐌺X"
js = "\uD800\uDF3AX"

**Actual:**

url = "%F0%90%8C%BA"
html = "𐌺"
js = "\uD800\uDF3A"
```

area-System.Text.Encoding bug

Most helpful comment

@svick Because I haven't done that before, thought issues should be discussed first, suspected that declaring the iteration variable out of the loop might not be the preferred solution, didn't have the build environment set up, expected I would be asked to write tests for that which I don't have any experience with either and feared it would take me much more time than someone doing this on daily basis.

I do care about this issue being fixed though, so here is my attempt. Thanks for prompting me.

All 3 comments

I would find this blocking for ASP.NET Core release.

In EncodeIntoBuffer the condition of if (!wasSurrogatePair) final block does not make sense, it's irrelevant whether the previous character was a surrogate pair or not, this block is meant to process remaining odd character, so it should test for that instead, e.g.:
```C#
private unsafe int EncodeIntoBuffer(char* buffer, int bufferLength, char* value,
int valueLength, int firstCharacterToEncode)
{
...
// this loop processes character pairs (in case they are surrogates).
// there is an if block below to process single last character.
int secondCharIndex;
for (secondCharIndex = valueIndex + 1; secondCharIndex < valueLength; secondCharIndex++)
{
...
}

if (secondCharIndex == valueLength) // was if (!wasSurrogatePair)
{
    ...
}

return totalWritten;

}

@miloush If you know how to fix it, why don't you create a PR?

@svick Because I haven't done that before, thought issues should be discussed first, suspected that declaring the iteration variable out of the loop might not be the preferred solution, didn't have the build environment set up, expected I would be asked to write tests for that which I don't have any experience with either and feared it would take me much more time than someone doing this on daily basis.

I do care about this issue being fixed though, so here is my attempt. Thanks for prompting me.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

jzabroski picture jzabroski  路  3Comments

EgorBo picture EgorBo  路  3Comments

nalywa picture nalywa  路  3Comments

jamesqo picture jamesqo  路  3Comments

GitAntoinee picture GitAntoinee  路  3Comments