Amazon-ecs-agent: Allow newer awslogs driver options

[jhovell]

I am guessing this is blocked on the ECS service because I can't even create task definitions with newer options such as aws-datetime-format, I get the error

aws ecs register-task-definition --family some-family --cli-input-json file://taskdef.json

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Log driver awslogs disallows options: awslogs-datetime-format

taskdef:

{
  "containerDefinitions": [

    {
      "memory": 1000,
      "name": "some-ecs-task",
      "cpu": 256,
      "image": "xxx.dkr.ecr.us-west-2.amazonaws.com/some-task:foo",
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/my/log-group",
          "awslogs-region": "us-west-2",
          "awslogs-stream-prefix": "1.0",
          "awslogs-datetime-format": "%Y-%m-%dT%H:%M:%S,%f"
        }
      }
    }
  ],
  "family": "my-family"
}

.. but opening this issue for visibility on when it might be implemented.

For example (key for using awslogs driver for things like Java stack traces)

https://docs.docker.com/engine/admin/logging/awslogs/#aws-multiline-pattern

Also curious if there are workarounds. I had considered running a separate container running the Linux aws logs agent (which does support this functionality) but I'm not sure that would guarantee that all logs have been flushed to CWL (my use case is for a ECS task, not a service) when the essential task exits.

Only other workaround I can think of is to modify application to encode multi-line log messages as a single line, but then encoded messages are stored in CWL and I need another mechanism to decode messages for reading messages unencoded.

[adnxn]

hey @jhovell - thanks for bringing this to our attention.

I am guessing this is blocked on the ECS service

yes it is! but we're tracking this request and it is on our roadmap.

[miensol]

Would you mind sharing if there is any particular reason to block options that are directly passed "as is" to docker invocation?

[kensaggy]

Could please share when in the roadmap can we expect this? This will greatly help with readability in CloudWatch Logs!

[jhovell]

@kensaggy as a workaround you can write logs to disk or a docker volume and run the CWL agent as a sidecar container. That's how we're working around this. Could probably take the time to publish a Dockerfile / task definition template if you want.

[adnxn]

Could please share when in the roadmap can we expect this? This will greatly help with readability in CloudWatch Logs!

@kensaggy - thanks for your sharing feedback and calling out readability in CloudWatch logs. Getting specific use cases and pain points is very helpful for prioritizing our feature requests. =]

Also - Sorry, we currently have no public roadmap.

[adnxn]

Would you mind sharing if there is any particular reason to block options that are directly passed "as is" to docker invocation?

@miensol - good question! We are actively collecting uses cases and evaluating ways to add support for this feature in the agent. Your feedback is appreciated and has been noted internally. To answer your question in the most direct way possible I need to describe how the agent calls Docker today and share our current thoughts.

The ECS Agent creates/starts containers by interacting with the docker socket file by constructing API parameters and using a client library and doesn't really invoke docker create or docker run using the CLI flags. In order to do this, the agent parses the message sent from the ECS backend that includes the container definitions in a JSON format and then merges it with more configurations that are generated by the agent (example: container names for linking, data only containers etc). The ECS backend in turn parses the Task Definition JSON to generate these container definitions. The agent will also validate and fill in necessary default values as constrained by Docker version, we do this so your task will have a higher chance of successes.

So, support must be added in the ECS Task Definition specification to support arbitrary docker flags, where by arbitrary JSON fields can be specified in the Task Definition and eventually merged into container Create/Start configs. While we recognize that this is a possible solution to use non-Task Definition container flags, it's also tricky to get right as an end-user because erroneous configs could lead to asynchronous failures when starting Tasks. But as I was saying, we're still collecting feedback from users and evaluating the best way to move forward. So please feel free to let us know what you think! We welcome suggestions from users.

[kensaggy]

@kensaggy - thanks for your sharing feedback and calling out readability in CloudWatch logs. Getting specific use cases and pain points is very helpful for prioritizing our feature requests. =]
Also - Sorry, we currently have no public roadmap.

Could please share when in the roadmap can we expect this? This will greatly help with readability in CloudWatch Logs!

@adnxn Well the use-case is pretty general... if you follow the 12-factor app principals for example (or some of them) you'll be outputting to stdout... even the simplest console.log/print of an error with its stack will create multiple lines which are ingested as separate events in cloudwatch, each one prepended with the timestamp - makes it very very hard to read stack traces that are logged to cloudwatch (not to mention they sometimes arrive out of order for some reason) - same goes to logging anything else that's multiline.

If we can set the multiline prefix (as the timestamp in our case) all these lines are expected to collapse to a single event which will make it so much easier to look at.

To be honest we're actively looking to replace CloudWatch now because of readability and usability issues (most of which aren't related to ECS or the ECS agent of course).

Thanks

[rodmaz]

+1

[kevinprice41]

+1

[ababushkin]

Any update on this one? Would be great to see it made available rather than using workarounds

[johnatdoshii]

I agree with above, this feature is important. This impacts downstream tools like SumoLogic as well. Multi-line log support is important for readability and usability.

It would be great to get an idea of when this is planned to be implemented.

[angelamancini]

The ability to collect multi-line logs is a necessity. My use case: I have several tasks (rails or sinatra apps) running on one ECS cluster that all log to cloudwatch log groups. Those log groups stream data to a lambda that then feeds into SumoLogic for searching. It works great, except stack traces don't work and come in as separate lines or worse separate lines and out of order.

I need to be able to define awslogs-multiline-pattern or awslogs-datetime-format. These options are available in the awslog driver, I don't see why I can't set them in an ECS task.

[kensaggy]

@angelamancini Yes, we're seeing the same thing (no SumoLogic though) - getting stack traces as separate lines, sometimes out of order...

[Trandel]

Any updates on this issue?

[LukaszTheProgrammer]

I have similar issue. If there is no solution available so far Is there any alternative to see multiline log entries as a single event in cloud watch ?

[jhovell]

@LukaszTheProgrammer the workaround is to run a separate container for the Linux CWL agent, and write your app logs to disk. You may need to also add some delays on shutdown to increase the likelihood the logs get flushed through the agent. Still not sure this would guarantee some log messages might not get lost.

AWS Labs has a docker image for this, but you probably need to modify it, if for no better reason than to configure the CWL agent file to parse the application log file.

https://github.com/awslabs/ecs-cloudwatch-logs

I could publish our Dockerfile or image for it, but it's dependent on being configured through envconsul. Guessing a more generic version could just use environment variables. Offered before but could create that if it's useful to someone.

Task def would look roughly like this.

{
  "containerDefinitions": [

    {
      "name": "my-app",
      "cpu": 1234,
      "mountPoints": [
        {
          "containerPath": "/log",
          "sourceVolume": "log"
        }
      ]
    },
    {
      "name": "cloudwatchlogs",
      "image": "your-repo/ecs-cloudwatch-logs:latest",
      "cpu": 50,
      "portMappings": [],
      "memory": 64,
      "essential": true,
      "volumesFrom": [
        {
          "sourceContainer": "my-app",
          "readOnly": true
        }
      ]
    }
  ],
  "volumes": [
    {
      "host": {},
      "name": "log"
    }
  ],
  "family": "my-app"
}

[bjornbak]

+1

[mlovretovich]

+1

Seems ridiculous to not support flags supported by both docker and the aws log driver.

[danieljimenez]

Guys, while this is supported by Docker, it's not until 17.06.0. ECS is still running 17.03. Give AWS a chance to catch up to the latest docker before assuming they don't support it.

[dmreiland]

They just released a new ECS AMI that runs 17.06.2-ce. I imagine support for multiline isn't far off.

[rmmeans]

Saw the same announcement for ECS AMI last night. Was going to try it out today as I've been waiting on this in ECS since the original PR got merged in to docker itself to add these options to the docker log agent for AWS CloudWatch logs.

Any chance that the new version of the agent that was also shipped last night has this ability? (v.1.15.0)? ECS is JVM space without this is a big pain, makes it harder to sell the service to colleagues etc, and I hate hacking around this stuff where it should just be a simple argument I specify on the task def.

[boatmisser]

Great to see this being actioned.
Does https://github.com/aws/amazon-ecs-agent/pull/1050 actually implement the awslogs changes, or is it just a prerequisite. The statement at the top "required for awslogs multiline support as per #872" could be taken either way.

Will the new awslogs options be available in 1.16.0 (milestone) and if so is there a rough ETA on it's release.

[adnxn]

The statement at the top "required for awslogs multiline support as per #872" could be taken either way.

@boatmisser You're right, the statement was bit unclear 😅

1050 is a prerequisite. It's basically just the changes that lets the ECS service know that "hey the agent on this instance can support tasks that require docker api version 1.30 (Docker 17.06.x)!". The main changes to enable awslogs-multiline-pattern are actually on the backend where the task registration logic is implemented. Otherwise, I don't have a rough ETA for you at this time, but I'll update this issue as soon as we do.

[SleeperSmith]

+1 guys, we need this.

[mlovretovich]

+1 please

[adnxn]

update: the agent now supports awslogs-multiline-pattern and awslogs-datetime-format flags on linux. More specifically, ECS service now allows for task definitions that include awslogs-multiline-pattern and awslogs-datetime-format flags. Example below.

...
    "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
            "awslogs-group": "awslogs-multiline-validate",
            "awslogs-region": "us-east-1",
            "awslogs-multiline-pattern":"^INFO"
        }
    }
...

...
    "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
            "awslogs-group": "awslogs-datetime-validate",
            "awslogs-region": "us-east-1",
            "awslogs-datetime-format":"%b %d, %Y %H:%M:%S"
        }
    }
...

[boatmisser]

That's great thanks!!!!
Do we have an ETA on a release?

[adnxn]

@boatmisser, awslogs-datetime-format and awslogs-multiline-pattern have already been released with the 1.16.0 agent, and ECS service side changes are out now. The awslogs-create-group flag is scheduled for the 1.17.0 release.

[JacobASeverson]

I'm seeing an issue placing tasks when trying to pass awslogs-multiline-pattern on the 1.16.0 agent.

unable to place a task because no container instance met all of its requirements. The closest matching ... is missing an attribute required by your task.

Task places fine once I remove it.

[adnxn]

@JacobASeverson, Do you mind opening a separate issue to track this? Can you verify that you are running Docker version 17.06, as that is when this flag was introduced in Docker.

[JacobASeverson]

Ahh yes a Docker version upgrade is needed, thanks @adnxn!

[sharanyad]

This feature is now available as part of the new release v.1.17.0 and 2017.09.h ECS Optimized AMI.