Containers-roadmap: [Fargate] [bug]: Logs getting corrupted on fargate 1.4.0

[ghost]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

Sample malformed json log:

c0]"}
{"traceId":"c5f566ec44264f64","spanId":"c5f566ec44264f64",...
...

Confirmed by AWS support, adding it here for community:

I understand you are facing issues with CloudWatch Logs where the JSON log message were malformed while using Fargate Platform version 1.4.0, 

Firstly, thank you for sending in the detailed message and screenshots. It really helped understanding the issue.

I see there is an ongoing issue with CloudWatch logs in Fargate platform version 1.4.0 where the log messages sometimes are truncated or corrupted. This issue is being identified by the Fargate Service team and they are actively working on it to improve this moving forward.

...

As you mentioned correctly, the logs are correctly populated when using Fargate Platform version 1.3.0. As a workaround I would suggest you to rollback the Tasks to Fargate platform version 1.3.0.

Which service(s) is this request for?
Fargate

Are you currently working around this issue?
Only affects services high volume logs (revert to 1.3.0 for those cases)

Additional context
Anything else we should know?

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[uherberg]

Is there any ETA on this? This makes Fargate 1.4.0 unusable for us, as many Java exceptions are pretty long and makes the Cloudwatch logs unreadable in Fargate 1.4.0

[aaithal]

Hello, thanks for reporting this issue. We recently rolled out changes that should fix this issue when using awslogs log driver with the default blocking mode on platform version 1.4.0. Please let us know if you continue running into this issue.

[ghost]

@aaithal Is this fixed only for blocking mode or for non-blocking mode as well?

[aaithal]

We're currently working on fixing this for non-blocking mode. The fix for blocking mode has already been released.

[uherberg]

@aaithal Any updates on this? I am concerned that once AWS switches the LATEST label of the Fargate Platform Version to 1.4.0, this bug will make Java logs unreadable for many teams.

[aaithal]

Hello, the fix for non-blocking mode has also been rolled out to platform version 1.4.0. Please let us know if you continue running into this.

[uherberg]

Thanks, @aaithal. This is looking good.

[kumailh]

This issue has been resolved.