Runtime: Decoder.GetCharCount() returns incorrect result when multi-byte characters are split

[Timovzl]

When Decoder.GetCharCount() is called on two halves of a byte sequence, splitting a multi-byte character, it returns an incorrect count.

• The first call correctly ignores the partial character.
• The second half returns a count that is too great. Perhaps it is ignoring the unprocessed byte from the previous call.
• In contrast, Decoder.GetChars() seems to work correctly.
• Tested with UTF-8, splitting a 4-byte 💩 "pile of poo" after the first byte. _(How's that for getting your hands dirty!)_
• This bug seems to affect both .NET Core 2.1 and .NET Framework.

var str = "a💩a";
var bytes = Encoding.UTF8.GetBytes(str); // 'a', 4 bytes that make up the pile of poo, 'a'
var decoder = Encoding.UTF8.GetDecoder();

// Getting char count at once works as expected
var totalCharCount = decoder.GetCharCount(bytes, 0, bytes.Length); // 4
decoder.Reset();

// Getting chars at once works as expected
var chars1 = new char[4];
var expectedResult = decoder.GetChars(bytes, 0, bytes.Length, chars1, 0); // 4: chars 'a', 2 chars that make up the pile of poo, 'a'
decoder.Reset();

// Gettings chars in two goes works as expected
var chars2 = new char[4];
var chars2Position = decoder.GetChars(bytes, 0, 2, chars2, 0, flush: false); // Including 1 byte from pile of poo
decoder.GetChars(bytes, 2, bytes.Length - 2, chars2, chars2Position, flush: true); // The remaining bytes
var getCharsInTwoGoesIsCorrect = chars2.SequenceEqual(chars1); // True
decoder.Reset();

// Getting char count in two goes returns INCORRECT result
var leftCount = decoder.GetCharCount(bytes, 0, 2, flush: false); // 1, correctly ignoring the first byte of pile of poo
var rightCount = decoder.GetCharCount(bytes, 2, bytes.Length - 2, flush: true); // 4! incorrectly treating the additional 3 bytes of pile of poo as 3 characters
var getCharCountInTwoGoesIsCorrect = leftCount + rightCount == totalCharCount; // False!

[jkotas]

The behavior you are seeing is expected. GetCharCount does not update the state of the decoder. The remarks section in https://docs.microsoft.com/en-us/dotnet/api/system.text.decoder.getcharcount?view=netcore-2.1 tries to explain it.

[juliusfriedman]

@jkotas, perhaps StringInfo should be referenced in the docs in some way?

[Timovzl]

@jkotas Thanks for clearing that up. It does not do a very good job at explaining it, in my opinion, as it misleads you in the very next sentence.

This method does not affect the state of the decoder.

To calculate the exact array size that GetChars requires to store the resulting characters, the application should use GetCharCount.

The latter is exactly what I am trying to do. How can we determine the exact size correctly before allocating anything, in the case where the data is split up (such as in a ReadOnlySequence<byte>)?

[jkotas]

How can we determine the exact size correctly before allocating anything, in the case where the data is split up

I do not think you can do it with the current Decoder APIs.

[SystemExtensions]

I had the same problem and I thought it was a bug

          // 3 chars 9 bytes
          var bytes = Encoding.UTF8.GetBytes("胡正媛");
          var decoder = Encoding.UTF8.GetDecoder();
            //incorrectly 
            var charCount = decoder.GetCharCount(bytes, 0, 1, false);
            Console.WriteLine(charCount);//0
            var charCount1 = decoder.GetCharCount(bytes, 1, 8, false);
            Console.WriteLine(charCount1);//4
            //correctly
            var charCount2 = decoder.GetCharCount(bytes, 0, 2, false);
            Console.WriteLine(charCount2);//0
            var charCount3 = decoder.GetCharCount(bytes, 2, 7, false);
            Console.WriteLine(charCount3);//3

[tarekgh]

closing this per @jkotas answers. feel free to send any more questions if you you have more.