Amazon-ecs-agent: How to move custom container logs in cloudwatch log group

[rajasekaran07]

Hi team,

Using awslog driver we can send general container logs in cloudwatch log group.
I am using python application to generate custom logs and send to logfile.log, Can someone suggest how to send my custom in cloudwatch log group.

Regards,
Raja

[samuelkarp]

The awslogs driver is like the other Docker logging drivers in that it only captures messages sent to stdout/stderr instead of files. If you want to log to a file and have that sent to CloudWatch Logs, we've published an example for how to do that here.

[nodanaonlyzuul]

Hi. Sorry to chime in on a very old ticket....

From what I can tell - messages sent to stdout & stderr in fargate tasks go straight to CloudWatch but....

I have an app that logs to stdout and it's messages only get sent to CloudWatch once the container dies. (e.g. upon redeployment).

This is similar to how docker run works on your localhost (without specifying the -t option).
Messages only get printed to your OS's terminal after the container goes down.

Sorry if I'm asking something that's a well understood issue...I've done some documentation reading and haven't seen anything on the subject. Does the behavior I describe above sound familiar?

[rturam]

It takes just 3 steps to send docker logs to cloudwatch.
Check out this article.

[cado1982]

@nodanaonlyzuul I'm seeing the same thing. I have .net core apps running in docker containers on ecs fargate. The logs sent to stdout are then sent to cloudwatch using the awslogs docker logger. However, as you mentioned the logs are only visible in cloudwatch once the fargate task is restarted or dies.

Can you remember if you found a solution? There's an aws forum post that mentions that it was an application configuration problem but doesn't go into details (https://forums.aws.amazon.com/thread.jspa?messageID=868933).

[nodanaonlyzuul]

Hi @cado1982

We _did_ solve this problem.
Our apps are written in Ruby and Ruby buffers the output to STDOUT and STDERR to avoid many, small writes.

So....our messages were stuck in that buffer purgatory until freed up by the gentile stopping of a the container.

This StackOverflow ticket put us on the right path. Maybe .net has a similar feature?

[cado1982]

@nodanaonlyzuul I think I experienced something similar. I switched to using a logging library called Serilog for .net and the problem resolved itself. Thanks for the reply.