Aspnetcore: AspNetCore 2.1-preview2 - System.Text.DecoderFallbackException

[adnan-kamili]

Hi,

I recently upgraded my code from asp.net core 2.0 to asp.net core 2.1-preview2. Since we upgraded we are encountering a crash in the production (for some requests) which never occurred before for the same code:

System.Text.DecoderFallbackException · Unable to translate bytes [C7] at index 829 from specified code page to Unicode.
System.Text.DecoderExceptionFallbackBuffer.Throw(byte[] bytesUnknown, int index)    
System.Text.DecoderExceptionFallbackBuffer.Fallback(byte[] bytesUnknown, int index) 
System.Text.DecoderFallbackBuffer.InternalFallback(byte[] bytes, Byte* pBytes, ref Char* chars) 
System.Text.UTF8Encoding.GetChars(Byte* bytes, int byteCount, Char* chars, int charCount, DecoderNLS baseDecoder)   
System.Text.DecoderNLS.GetChars(byte[] bytes, int byteIndex, int byteCount, char[] chars, int charIndex, bool flush)    
System.Text.DecoderNLS.GetChars(byte[] bytes, int byteIndex, int byteCount, char[] chars, int charIndex)    
Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.ReadIntoBuffer()  
Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.Read(char[] buffer, int index, int count) 
Newtonsoft.Json.JsonTextReader.ReadData(bool append, int charsRequired) 
Newtonsoft.Json.JsonTextReader.ParseValue() 
Newtonsoft.Json.JsonTextReader.Read()   
Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, bool hasConverter)    
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, bool checkAdditionalContent) 
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter+<ReadRequestBodyAsync>d__17.MoveNext()   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
System.Runtime.CompilerServices.TaskAwaiter<TResult>.GetResult()    
Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder+<BindModelAsync>d__7.MoveNext()   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder+<BindModelAsync>d__11.MoveNext()  
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
System.Runtime.CompilerServices.TaskAwaiter<TResult>.GetResult()    
Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider+<>c__DisplayClass0_0+<<CreateBinderDelegate>g__Bind|0>d.MoveNext()   
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__13.MoveNext()  
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextExceptionFilterAsync>d__24.MoveNext()

It seems request parsing is failing which wasn't happening before. Can't even log the request model as crash occurs before the controller action is called.

[kichalla]

Based on the callstack, I wonder if this could be reproduced outside asp.net core too. You could try with the same content as in your failed request's body and in a standalone .net core console app and using json.net (version 11.0.2, used in asp.net core 2.1.0-preview2-final). If you can still repro it, then you should probably file a bug here: https://github.com/dotnet/corefx/issues

[adnan-kamili]

@kichalla, unfortunately, I don't have the exact request body which is causing it to crash. I need to log it before the json is parsed and I don't know whether I can do that in any filter or middleware.

[kichalla]

Try this: Write an exception filter and decorate it on the action which fails and in the filter log the request body

A quick example:
```c#
public class TestExceptionFilterAttribute : Attribute, IExceptionFilter
{
public void OnException(ExceptionContext context)
{
var requestBody = context.HttpContext.Request.Body;
if (requestBody.CanSeek)
{
// rewind it as modelbinding step would have read the body already
requestBody.Seek(0L, SeekOrigin.Begin);

            //you need not close this stream as underlying layer (which created it) would it for you.
            var streamReader = new StreamReader(requestBody);
            var content = streamReader.ReadToEnd();
            var logger = context.HttpContext.RequestServices.GetRequiredService<ILogger<TestExceptionFilterAttribute>>();
            logger.LogError(content);
        }
    }
}

`` By default,JsonInputFormatterbuffers the request body before deserliazation it a instance of the model type to bind to, unless you explicitly setMvcOptions'sSuppressInputFormatterBuffering` to a different value.

[adnan-kamili]

@kichalla Thanks, I tried your code snipped and following key value pair seems to be causing the issue:

...
"hostname":"��������-PC",
...

[kichalla]

@adnan-kamili The characters in your earlier post are messed up. Is it possible for you to share the whole request data so that we could reproduce at it at our end?

[Tratcher]

E.g. a wireshark trace

[kichalla]

@karelz Any ideas on why this would be happening?

[adnan-kamili]

@kichalla I can't provide a wireshark trace because it's happening on machines of our customers. However, I can share how I fixed the issue:

Our C++ HTTP client gets hostname of machine using GetComputerNameA function.

Some machines seem to have non-ASCII characters in the hostname, so I changed it to GetComputerName which is the utf-16 version of the function, and before http post I converted the utf-16hostname string to utf-8 hostname string. This fixed the issue at my end.

[Tratcher]

So it sounds like everything here was behaving appropriately when presented with invalid bytes from the client.

[karelz]

Any ideas on why this would be happening?

@kichalla No idea, I think we need isolated repro. Latest reply hints it may be machine name encoding problem? The stack doesn't suggest anything like that though.

[kichalla]

Thanks @karelz
Regarding

The stack doesn't suggest anything like that though.

I am not sure about your question here, but the hostname (with invalid characters) is present in the body of the request and decoding it is failing.

[kichalla]

Closing this issue as there's no action item on our side.

[karelz]

@kichalla

but the hostname (with invalid characters) is present in the body of the request and decoding it is failing.

Ah, that was not 100% clear to me.
How did the invalid characters get there? Should they be encoded? Is it a problem on server or on client? Anything we can/should fix in CoreFX?

[kichalla]

Based on https://github.com/aspnet/Home/issues/3105#issuecomment-385252992, https://github.com/aspnet/Home/issues/3105#issuecomment-385492323 and the callstack, the scenario is that there's a C++ client posting json data(having invalid characters) to asp.net core app and we failed to decode these characters in the server.
AFAIK, we need not fix anything at our end.

[adnan-kamili]

@kichalla We received a new error in production, can the following be due to the code we added for logging request body by reading it from the stream:

Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate. 
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate. 
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.PumpAsync() 
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException() 
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) 
   at System.IO.Pipelines.Pipe.GetReadAsyncResult() 
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding) 
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext) 
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value) 
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext() 
--- End of stack trace from previous location where exception was thrown --- 
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync() 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync() 
[12:17:20 ERR] Connection id "0HLDG3CS3KJNF", Request id "0HLDG3CS3KJNF:00000001": An unhandled exception was thrown by the application. 
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate. 
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.PumpAsync() 
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException() 
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) 
   at System.IO.Pipelines.Pipe.GetReadAsyncResult() 
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding) 
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext) 
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value) 
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext() 
--- End of stack trace from previous location where exception was thrown --- 
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync() 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync() 
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException() 
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) 
   at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token) 
   at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 count) 
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.Read(Byte[] buffer, Int32 offset, Int32 count) 
   at System.IO.StreamReader.ReadBuffer() 
   at System.IO.StreamReader.ReadToEnd() 
   at ProdWebApi.Filters.CustomExceptionFilterAttribute.OnException(ExceptionContext context) in /app/Filters/CustomExceptionFilterAttribute.cs:line 48 
   at Microsoft.AspNetCore.Mvc.Filters.ExceptionFilterAttribute.OnExceptionAsync(ExceptionContext context) 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter() 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context) 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync() 
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync() 
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext) 
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context) 
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context) 
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) 
   at AspNetCoreRateLimit.IpRateLimitMiddleware.Invoke(HttpContext httpContext) 
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context) 
   at ProdWebApi.Middlewares.CustomResponseHeadersMiddleware.Invoke(HttpContext context) in /app/Middlewares/CustomResponseHeadersMiddleware.cs:line 38 
   at ProdWebApi.Middlewares.ForceHttpsMiddleware.Invoke(HttpContext context) in /app/Middlewares/ForceHttpsMiddleware.cs:line 34 
   at Bugsnag.AspNet.Core.Middleware.Invoke(HttpContext context, IClient client) 
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) 

[kichalla]

Based on the call stack, the original exception is happening when the JsonInputFormatter is trying to buffer the request body.
Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.

This exception is being caught in the exception filter but it's still trying to read the request body (from the request stream and not the buffered one) since CanSeek is always true.

@Tratcher can CanSeek be set to True only when all the request body is buffered successfully into FileBufferingReadStream?

[Tratcher]

@kichalla how would that help here? I see DrainAsync throwing. Where else is the request body read?

[kichalla]

Its being read in the exception filter here: https://github.com/aspnet/Home/issues/3105#issuecomment-385250402

[adnan-kamili]

@kichalla
Is it possible to change the default fallback strategy if decoding fails?

https://stackoverflow.com/questions/50948947/change-character-encoding-default-fallback-strategy#50948947