Aws-lambda-dotnet: 'Unable to expand length of this stream beyond its capacity.' when using DefaultLambdaJsonSerializer

[timhill-iress]

Description

On migrating from Amazon.Lambda.Serialization.Json.JsonSerializer to Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer there is a significant increase in the size of the serialised response.

I'm generating JSON and returning it in the body of APIGatewayProxyResponse. Previously the quotes in the JSON where encoded as \" ( 2 characters), now they are encoded as \u0022 ( 6 characters). I could previously return more than 5MB of JSON before hitting the Lambda Response Payload limitation of 6MB, now I can only return around 3.5MB of JSON

The error I get is:

Error converting the response object of type Amazon.Lambda.APIGatewayEvents.APIGatewayProxyResponse from the Lambda function to JSON: Unable to expand length of this stream beyond its capacity.

I'm raising this as a bug, as it's not clear that there is a significant impact from moving to the new serialiser and there doesn't seem to be an easy way to override the behaviour to make it produce JSON in the way it used to be produced.

Reproduction Steps

Sample code showing the difference:
``` c#
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Mime;
using System.Text.Json;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Amazon.Lambda.Serialization.SystemTextJson;
using Microsoft.Net.Http.Headers;

namespace testserialiser
{
class Program
{
static void Main(string[] args)
{
var o = new
{
a = "test",
b = 27,
c = new
{
a = "foobar"
}
};
string body = System.Text.Json.JsonSerializer.Serialize(o);
int qc = body.Count(x => x == '\"');
Console.WriteLine($"Source:{body} Length:{body.Length}, QuoteCount:{qc}");
//Source:{"a":"test","b":27,"c":{"a":"foobar"}} Length:38, QuoteCount:12

        var response = new APIGatewayProxyResponse
        {
            StatusCode = (int)HttpStatusCode.OK,
            Headers = new Dictionary<string, string> { { HeaderNames.ContentType, MediaTypeNames.Application.Json } },
            Body = body
        };

        //Use DefaultLambdaJsonSerializer
        var ms = new MemoryStream(new byte[6 * 1000 * 1000]);
        ILambdaSerializer ser = new DefaultLambdaJsonSerializer();
        ser.Serialize(response, ms);
        LogMS(ms);
        //Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\u0022a\u0022:\u0022test\u0022,\u0022b\u0022:27,\u0022c\u0022:{\u0022a\u0022:\u0022foobar\u0022}}","isBase64Encoded":false} Length:196

        //Use DefaultLambdaJsonSerializer, attempt to override JsonEscaping
        ms.Position = 0;
        JsonSerializerOptions options = null;
        ser = new DefaultLambdaJsonSerializer(o =>
        {
            options = o;
            o.Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping;
            o.WriteIndented = false;
        });
        ser.Serialize(response, ms);
        LogMS(ms);
        //Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\u0022a\u0022:\u0022test\u0022,\u0022b\u0022:27,\u0022c\u0022:{\u0022a\u0022:\u0022foobar\u0022}}","isBase64Encoded":false} Length:196


        //Plain Serialize using same options as DefaultLambdaJsonSerializer
        var js = JsonSerializer.Serialize(response, options);
        Console.WriteLine($"Expected:{js} Length:{js.Length}");
        //Expected:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\"a\":\"test\",\"b\":27,\"c\":{\"a\":\"foobar\"}}","isBase64Encoded":false} Length:148

        //Serialize using same options as DefaultLambdaJsonSerializer and Utf8writer
        ms.Position = 0;
        using (var writer = new Utf8JsonWriter(ms))
        {
            JsonSerializer.Serialize(writer, response, options);
        }
        LogMS(ms);
        //Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\u0022a\u0022:\u0022test\u0022,\u0022b\u0022:27,\u0022c\u0022:{\u0022a\u0022:\u0022foobar\u0022}}","isBase64Encoded":false} Length:196

        //Serialize using old JsonSerializer
        ms.Position = 0;
        ser = new Amazon.Lambda.Serialization.Json.JsonSerializer();
        ser.Serialize(response, ms);
        LogMS(ms);
        //Actual:{"statusCode":200,"headers":{"Content-Type":"application/json"},"multiValueHeaders":null,"body":"{\"a\":\"test\",\"b\":27,\"c\":{\"a\":\"foobar\"}}","isBase64Encoded":false} Length:173

    }

    private static void LogMS(MemoryStream ms)
    {
        long len = ms.Position;
        ms.SetLength(len);
        ms.Position = 0;
        var sr = new StreamReader(ms);
        var j = sr.ReadToEnd();
        Console.WriteLine($"Actual:{j} Length:{j.Length}");
        ms.Position = 0;
    }
}

}

```

Logs

Error converting the response object of type Amazon.Lambda.APIGatewayEvents.APIGatewayProxyResponse from the Lambda function to JSON: Unable to expand length of this stream beyond its capacity.: JsonSerializerException
at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.SerializeT

at System.IO.UnmanagedMemoryStream.WriteCore(ReadOnlySpan1 buffer) at System.IO.UnmanagedMemoryStream.Write(ReadOnlySpan1 buffer)
at System.Text.Json.Utf8JsonWriter.Flush()
at System.Text.Json.JsonSerializer.WriteCore(Utf8JsonWriter writer, Object value, Type type, JsonSerializerOptions options)
at System.Text.Json.JsonSerializer.WriteValueCore(Utf8JsonWriter writer, Object value, Type type, JsonSerializerOptions options)
at System.Text.Json.JsonSerializer.SerializeTValue
at Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer.SerializeT

Environment

• Build Version: 2.0.1
• OS Info: Ubuntu
• Build Environment: VSCode
• Targeted .NET Platform:dotnetcore3.1

Resolution

---

This is a :bug: bug-report

[martincostello]

Have you tried using some settings where you've raised the value of JsonSerializerOptions.DefaultBufferSize? Maybe that might resolve/workaround the issue?

[timhill-iress]

Thanks for the suggestion, but really my test code above is really focused on showing that there is a significant increase in the size of the serialised JSON. It's that combined with the hard AWS limit of 6MB (https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html) that causes a problem for me.
I can workaround by reverting to the old Serailiser Amazon.Lambda.Serialization.Json.JsonSerializer, but I understood there the recommendation was to use the new one as it improves the cold start performance. I didn't understand the downside of the new Serialiser.

[NGL321]

Hey @timhill-iress,

Thank you for reporting this. I am going to look into it further, but on initial review it does seem like an unintended problem introduced with the change.
It seems that you have a good workaround (although I understand the concern).
I am going to give this one more confirmation, and then pass it along for resolution to the dev team.

😸 😷

[ashishdhingra]

Hi @timhill-iress,

I was able to confirm that the new library that the new library does produces a longer serialized JSON. Unfortunately, due to time constraints, it could be a period of time until someone looks into this issue. If you would like, feel free to submit a fix to it, else we will update the issue when someone works on it.

Thanks,
Ashish

[AnushaReddy1998]

Hi,
I got the same issue..I'm using AWS server less web API using runtime .net core.i'm deploying API's into gateway by using AWS Toolkit.from past one year i'm using this but recently i got the this exception and taken aws support and posted the response size returned by API is nearly 3.2 MB.The team said this is because of migrating lambda JsonSerializer. Our application is used by clients, they are unable to see the data(response from API)in website. Please work on it

[srinadh-k]

Guys, Since for couple of days we have been facing same issue. any update when it will be fixed?

[ozziepeeps]

+1

[hvanbakel]

Ok, so this issue is a whole list of issues. First of all the documentation here:
https://github.com/aws/aws-lambda-dotnet/tree/master/Libraries/src/Amazon.Lambda.Serialization.SystemTextJson

Mentions the assembly level attribute, this however no longer works because of this code:
https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.AspNetCoreServer/AbstractAspNetCoreFunction.cs#L426-L430
As the method level serializer takes precedence, you can only override it at the method level now.

Then there’s the serializer doing the additional encoding. The way the code stands right now there is no way to fix it other than copying the serializer code and putting it in your solution with a few modifications. You then need to override the serializer on the method like above.

So what changes are needed on the serializer?
First of all you want to set the encoder, which you could do through the constructor that takes an action.
You probably want to set this to JavaScriptEncoder.UnsafeRelaxedJsonEscaping. The unsafe refers to this allowing characters that could lead to exploits in html but this is probably not the typical use case. You can read the caution here though.

All of this can still be done without your own implementation but unfortunately the instantiation of the writer here, should use the overload that takes jsonwriteroptions, to again set the same encoder.

I’ve tested the above and it gives me the same response sizes as I was seeing on 2.1 with newtonsoft. I might be able to write this up as a formal PR over the weekend but this should at least unblock the other people on the thread.

[philasmar]

This issue has been fixed via the following release:
Release 2020-09-16
Amazon.Lambda.Serialization.SystemTextJson (2.0.2)
Added default JsonWriterOptions to change serialization of quotation marks from ascii representation to an escaped quote
Amazon.Lambda.AspNetCoreServer (5.1.5)
Updated to version 2.0.2 of Amazon.Lambda.Serialization.SystemTextJson

If you are still facing issues, feel free to reopen the ticket.