Runtime: The UTF-8 BOM within a file should be ignored when calling JsonSerializer.ReadAsync

[ahsonkhan]

Similar to other APIs within the JSON stack that accept a stream (such as JsonDocument.ParsAsync), the JsonSerializer.ReadAsync should ignore the UTF-8 BOM at the beginning of the stream (if one exists).

Currently, we throw an exception since that is invalid JSON:

System.Text.Json.Serialization.Tests.StreamTests.ReadLargeFile [FAIL]
        System.Text.Json.JsonException : '0xEF' is an invalid start of a value. Path: $ | LineNumber: 0 | BytePositionInLine: 0.
        ---- System.Text.Json.JsonReaderException : '0xEF' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
        Stack Trace:
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\ThrowHelper.Serialization.cs(105,0): at System.Text.Json.ThrowHelper.ReThrowWithPath(JsonException exception, String path)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.cs(118,0): at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(211,0): at System.Text.Json.JsonSerializer.ReadCore(JsonReaderState& readerState, Boolean isFinalBlock, Span`1 buffer, JsonSerializerOptions options, ReadStack& readStack)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.Stream.cs(144,0): at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
          /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/ValueTask.cs(759,0): at System.Threading.Tasks.ValueTask`1.get_Result()
          E:\GitHub\Fork\corefx\src\System.Text.Json\tests\Serialization\Stream.ReadTests.cs(30,0): at System.Text.Json.Serialization.Tests.StreamTests.ReadLargeFile()
          --- End of stack trace from previous location where exception was thrown ---
          ----- Inner Stack Trace -----
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\ThrowHelper.cs(233,0): at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(1086,0): at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(954,0): at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first)
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(869,0): at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Reader\Utf8JsonReader.cs(258,0): at System.Text.Json.Utf8JsonReader.Read()
          E:\GitHub\Fork\corefx\src\System.Text.Json\src\System\Text\Json\Serialization\JsonSerializer.Read.cs(33,0): at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack)

For example:
https://github.com/dotnet/corefx/blob/dae07b7c0ef23d528baf60ce4f84e6b61c22ceb6/src/System.Text.Json/src/System/Text/Json/Document/JsonDocument.Parse.cs#L689

cc @steveharter, @bartonjs

[am11]

Given:

$ cat file.json
{}

$ hexdump -C  file.json
00000000  ef bb bf 7b 7d                                    |...{}|
00000005

```c#
JsonDocument.Parse(File.ReadAllBytes("file.json"))

throws:

System.Text.Json.JsonReaderException: '0xEF' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0.
at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan1 bytes) at System.Text.Json.Utf8JsonReader.ConsumeValue(Byte marker) at System.Text.Json.Utf8JsonReader.ReadFirstToken(Byte first) at System.Text.Json.Utf8JsonReader.ReadSingleSegment() at System.Text.Json.Utf8JsonReader.Read() at System.Text.Json.JsonDocument.Parse(ReadOnlySpan1 utf8JsonSpan, Utf8JsonReader reader, MetadataDb& database, StackRowStack& stack)
at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedBytes) at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory1 utf8Json, JsonDocumentOptions options)
```

is this intentional?

[bartonjs]

@am11 Yes. The ReadOnlyMemory<byte> parse method says it is utf8Json. It's supposed to just be "text" at that point (just like Utf8JsonReader), so the BOM is not permitted. If you're using it from a file then the Stream parse is the better way to go.

It's documented, see the Remarks section at https://docs.microsoft.com/en-us/dotnet/api/system.text.json.jsondocument.parse?view=netcore-3.0#System_Text_Json_JsonDocument_Parse_System_ReadOnlyMemory_System_Byte__System_Text_Json_JsonDocumentOptions_

[am11]

Thanks @bartonjs, I was writing an adaptor for JSONTestSuite. Using stream API fixed one failing test. For rest of the failing tests (mostly due to unsupported encoding), I have opened https://github.com/dotnet/corefx/issues/40482.

[alexsandro-xpt]

How to fix it? I have same problem with dotnet 3.1

[am11]

@alexsandro-xpt, if you are referring to JsonDocument.Parse, I ended up using the Stream overload of JsonDocument.Parse in dotnet core 3.0 app: https://github.com/nst/JSONTestSuite/blob/9f23c68/parsers/test_dotnet_system_text_json/Program.cs#L14-L19.

[alexsandro-xpt]

@am11 It's not about JsonDocument.Parse it's about JsonSerializer.Deserialize<T>:

I solve it with MemoryStream.

````csharp
var jsonString = File.ReadAllText(@"C:\mysonfile.json", new UTF8Encoding());
var jsonByte = Encoding.UTF8.GetBytes(jsonString);

using (Stream stream = new MemoryStream(jsonByte))
{
var jsonModel = await JsonSerializer.DeserializeAsync(stream, options);

Console.Title = jsonModel.Tabs.Count.ToString();

foreach (var item in jsonModel.Tabs)
{
    System.Console.WriteLine(item.FavIconUrl);
    System.Console.WriteLine(item.Title);
    System.Console.WriteLine(item.Url);
}

}
````

[am11]

You can simplify the usage without MemoryStream:

c# using var stream = File.Open(@"C:\mysonfile.json", FileMode.Open, FileAccess.Read); var jsonModel = await JsonSerializer.DeserializeAsync<Window>(stream, s_options); Console.Title = ...

[ahsonkhan]

It's not about JsonDocument.Parse it's about JsonSerializer.Deserialize<T>:

@alexsandro-xpt - the stream APIs on JsonSerializer handle (and ignore) the UTF-8 BOM for you. You can pass in Stream to those APIs directly, as @am11 mentioned.

If you have utf-8 text (i.e. a byte[], span, etc.), then it can't contain the UTF-8 BOM, just like a utf-16 string.

Also, if you need to use File.ReadAllText, then the BOM has already been removed. You can just pass in the jsonbyte directly to the synchronous Deserialize method. Creating a stream around the byte[] (particularly MemoryStream) doesn't help (especially if you need the IO to be async).

[alexsandro-xpt]

You can simplify the usage without MemoryStream:

using var stream = File.Open(@"C:\mysonfile.json", FileMode.Open, FileAccess.Read);
var jsonModel = await JsonSerializer.DeserializeAsync<Window>(stream, s_options);
Console.Title = ...

@am11 scary... this throw the same error System.Text.Json.JsonReaderException: '0xEF' is an invalid start of a value.

@ahsonkhan This was not working, I guess with NewtownSoft JSON should work, I not test yet.

[ahsonkhan]

scary... this throw the same error System.Text.Json.JsonReaderException: '0xEF' is an invalid start of a value.

Can you share a standalone project with the repro, including the json file you are using? What version of .NET Core are you on? Also, what is the entire exception message?