Amazon-ecs-agent: TaskARN --> Container ID

[pedrorjbr]

It very difficult to track the taskARN to the execution docker container ID. These two number have no relation. How could I find which container(s) the task started/run?

[samuelkarp]

If you're logged into the EC2 instance, you can use the introspection API to retrieve this information. Are you asking for this to be returned in the ECS API?

[pedrorjbr]

I saw in the documentation that I can use it to get the container Id.

But look my use case:
I am using the ECS infrastructure to run dockerized batch jobs. I am using Airflow workflow (from Airbnb) to schedule these tasks and Syslog to centralize all my logs.
When a task failed I need to "link" the task arn with the container Id, because in the Syslog I only see the container Id.
If I use cloudwatch to centralize all my logs, I imagine that I will face the same problem.

That's why I need to link the task arn with my container's Id. If I have a way to pass the task arn an get all container Id, it solve my problem. What do you think?

[samuelkarp]

Thanks for the details. I'll mark this as a feature request for now.

[ruudderooij]

I ran into the same issue (for a very similar reason, logging to CloudWatch Logs in my case). My workaround so far is to run a small script on each ECS instance that essentially listens to the output of "docker events" and for each started container, records the com.amazonaws.ecs.task-arn label and the docker container id to a place where I can later correlate the two IDs again.

Like the original reporter, I'd love to have this available as a first-class citizen in the ECS API so I don't need my clunky script anymore.

[aldarund]

+1

[wjimenez5271]

Ditto, I'm using the fluentd logging driver for docker in my ECS cluster, however the driver only picks up the docker container id, which is unrelated to anything the ECS API seems to provide, meaning a user cannot correlate their ECS tasks to logs unless the log onto the instance that runs their contianer and lookup the ID.

It seems to me an acceptable work around would be for the ECS agent to capture that docker container ID and present it through the API so a user could correlate that to what they see in their logging system.

[hridyeshpant]

@wjimenez5271
you can pass tag option in your taskdefinition to related log with docker image name
https://docs.docker.com/engine/reference/logging/log_tags/

[wjimenez5271]

@hridyeshpant Thanks for the suggestion. Am I understanding correctly however that this same tag would propagate to all tasks/containers launched with that task definition? Meaning a user wouldn't be able to distinguish once instance of a task/container vs another?

If so this might not help much then because we already have a way to tell the image name in the logs which is essentially a similar abstraction as task defintion, and the missing piece is being able to identify logs from a specific container.

[natefox]

What is the best way to 'vote' for this feature? I too would like to be able to look at the containerArn and get the container ID, but for very different reasons.

Should those who are interested in this simply hit subscribe?

[wjimenez5271]

@natefox I would subscribe and add a +1 comment...more comments on an issue typically gets people's attention.

[gabadi]

+1

[MFProduction]

+1

[wjimenez5271]

In case its helpful for anyone else in the meantime, I ended working around this limitation by writing a tool to collect the container id and task id: https://github.com/CheggEng/ecs_id_mapper

[mattcallanan]

+1

[bshelton229]

I'm also using cloudwatch logs and need to correlate a task run to a log stream. We allow developers to launch tasks into certain clusters and need the ability to stream the logs for each task back to them in an interface. We're so very close to this, we're just missing the ability to easily link the task back to the key used for the log stream using the API.

[mrdziuban]

I was able to work around this by exposing the Docker remote API. I exposed the API by editing the OPTIONS line in /etc/sysconfig/docker to look like this:

OPTIONS="--default-ulimit nofile=1024:4096 -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"

Then when making a RunTask or StartTask call to the ECS API, I used the containerOverrides option to pass a unique environment variable to the Docker container. I was then able to find the container that had the same environment variable using the Docker remote API.

With that being said, it would be really nice if ECS provided this.

[ghost]

+1

[rjpereira]

+1

[MFProduction]

+1, I am also using awslogs for streaming log files, and it would be really nice to have this functionality.

[admgre]

+1 awslogs

[forwardinc]

+1 for this, I am working on an app that needs to stream the logs for each container, via cloudwatch requests using the JS SDK.

Very unfortunate that no thought was given to being able to programmatically correlate the logs to the tasks, please add this soon!

[shanmugakarna]

+1

Need this to find what log streams are active and what are dead (dead tasks vs active tasks)

[nickholl]

+1

[jimmysoho]

Trying to use awslogs to get the docker logs to end up in CloudWatch Logs. And therefor I too need to be able to correlate the container ARN to the container ID, because the container ID is used by default as the Log Stream Name in CloudWatch Logs.

See this documentation:

https://docs.docker.com/engine/admin/logging/awslogs/

It mentions the option awslogs-stream can be used. That would be helpful as then we could assign a unique ID to it ourselves and then we don't need a mapping from container ARN to container ID anymore. Unfortunately it appears the option is not accepted when used in "logConfiguration" in a task definition:

Log driver awslogs disallows options: awslogs-stream

@samuelkarp I think that message is coming from the ECS internals. If the ECS team could add allowing that option that would be great.

[kiranmeduri]

Today we launched support for awslogs-stream-prefix, which lets you name your log stream that includes the taskID so that it's easy to go from a task to the log stream and from the log stream to the task. See the documentation for more information. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html

[joelangeway]

+1

[rhunwicks]

I still think we need the ability to link the Docker Container Id to the ContainerARN. Ideally, it would be included in the response to describe_tasks

[bacoboy]

I too was looking at that containerArn and wondering what the heck that represented.

"containerArn": "arn:aws:ecs:us-east-1:1234567890:container/e24e1af0-d975-42b0-87aa-ee232ac255a7",

Is it just an internally generated unique thingy to represent the running docker container instance?

I'm not sure there is another layer of indirection/mapping as @rhunwicks indicated

Feels like the containerArn should just use the docker container id? Granted it isn't an Arn in that it isn't an AWS resource. Perhaps it shouldn't be an Arn at all but the docker id as originally requested.

Really in the end I don't care how AWS exposes this linkage/mapping, but seems like it should be built-in and not something every user of ecs needs to hack together. Guess in the meantime I'll have to checkout some of the hackery in the comments people have suggested...

Waiting patiently :)

[doubledgedboard]

@samuelkarp There definitely seems to be a gap for figuring out exactly which docker id + task id you are, from inside a container.

For example, as a container I can query the instance metadata to figure out the host IP, but I can't figure out which dynamic port I've been mapped to at the host. I could install the aws cli and describe tasks, but how would I know which task I am out of the list that may be there?

My use case is an executable that wants the current host\port it's going to be listening on (its 'advertisement address') as an argument or environment variable, when it runs. For my 'CMD' I could block on an aws call to describe the task and once it's running get the mapped port, then store that in an env variable, and then run my exe.

However, right now I can't figure out a way to associate which instance of the task I am while inside the container.

[jwconway]

I managed to find a workaround for this that works for us.

We need to know the mapped dynamic port of our zuul instances for registering our services with eureka so we can aggregate the hystrix streams with turbine. This is possible with the describe-tasks AWS API if you know which task you're in.

We use the $HOSTNAME environment variable from within the container which, it turns out, maps to the short container id. Once we have this we call the ECS Container Agent Introspection API to list all tasks that are running on the agent (using the AWS metadata API to retrieve the local IP address of the agent).

Each task has a ContainerId. This ContainerId is the long container id. We compare the short id against the long id and use this to choose the Task ARN we need. The long id will start with the short id. This is a slight leap of faith as I understand there is a slight chance of a collision here.

Once we know the Task ARN of the task were in we can easily extract the dynamically mapped port using the describe-tasks API mentioned above.

Its worth knowing as well that this only works when the task is in the RUNNING status.

[doubledgedboard]

@jwconway oh wow, that's an excellent tip!

Out of curiosity have you tried looking into the events stream as well? I was thinking of seeing if I can pull the event from the task getting placed back from the event stream, from within the container, and see if the full mapping exists there as well.

Edit: also, are you able to hit the introspection API from inside the container? I assumed it might only respond to requests from localhost i.e. the host itself. (e.g. if i have a container that's NOT running in 'host' networking mode...)

[AjitDas]

@jwconway James - can you please share any snippets you have I did something similar for hazelcast discovery where I need to know the dynamic host port mapped so that I can publish to consul for the service discovery.

I am using these but it's not working for me, inside the docker container I run this before starting my catalina.sh (am using a webapp using tomcat base image).

From EC2 machines I can query all these ECS introspection API by putting localhost, but inside docker container if I use this with private ip of EC2 instance it doesn't get any response. Any pointers what am missing ?

FYI I have all the packages pre-installed in my docker images (aws cli, jq etc ...)

HTTP_STATUS=$(curl -s --connect-timeout 10 -o /dev/null -w '%{http_code}' http://169.254.169.254/latest/meta-data/local-ipv4)
if [ "$HTTP_STATUS" == "200" ]; then
    export AWS_PRIVATE_IP=$(curl --connect-timeout 10 http://169.254.169.254/latest/meta-data/local-ipv4)
fi

# get task ARN from ECS introspection APIs
TASKS_JSON=$(curl  -s --connect-timeout 10 http://$AWS_PRIVATE_IP:51678/v1/tasks 2>/dev/null)
echo $TASKS_JSON
TASK_ARN=$(echo $TASKS_JSON | /usr/bin/jq -r ".Tasks[] | select(.Containers[].DockerId | test(\"^$HOSTNAME\")) | .Arn")

# get cluster name from ECS introspection APIs
METADATA_JSON=$(curl  -s --connect-timeout 10 http://$AWS_PRIVATE_IP:51678/v1/metadata 2>/dev/null)
echo $METADATA_JSON
CLUSTER_NAME=$(echo $METADATA_JSON | /usr/bin/jq -r ".Cluster")

log "-- AWS ECS Metadata ::: TASK_ARN         = $TASK_ARN --"
log "-- AWS ECS Metadata ::: CLUSTER          = $CLUSTER_NAME --"

#TASK_COMMAND='aws ecs describe-tasks --cluster '$CLUSTER_NAME' --tasks '$TASK_ARN' --region '$AWS_REGION
TASK_COMMAND='aws ecs describe-tasks --cluster '$CLUSTER_NAME' --tasks '$TASK_ARN
log "-- AWS ECS Metadata ::: DESCRIBE_TASK_CMD      = '$TASK_COMMAND' --"

# get hazelcast hostport mapping from networkbinging
TASK_JSON=$($TASK_COMMAND 2>/dev/null)
echo $TASK_JSON
HZ_PORT=$(echo $TASK_JSON | /usr/bin/jq '(.tasks[].containers[].networkBindings[] | select(.containerPort==5701)) | .hostPort')

export PORT_5701=$HZ_PORT

[mhurne]

@AjitDas , we're solving the same problem I believe (Hazelcast member discovery on ECS), and we've had success reaching the agent introspection API at http://172.17.0.1:51678 from within containers.

[pecigonzalo]

2yr have gone, any progress from AWS? @samuelkarp

[jonathannaguin]

@mhurne seems like the best solution, based on the assumption we can always reach the introspection API on http://172.17.0.1:51678.

Is that something documented in AWS ECS?

[bndw]

@jonathannaguin The Container Agent Introspection API is documented here. It runs on all Container Instances on port 51678.

172.17.0.1 is the Docker bridge network that all containers are connected to by default, see here. By making a request to the Docker bridge on port 51678, you're being routed to the local ecs-agent on your Container Instance.

[bacoboy]

While these hacks work, I think the intent for the issue is for the agent to report the docker container specifics back to the ECS mothership so it can be inspected using the SDK/CLI/Console.

[bndw]

@bacoboy that's clearly the intent, but these various features have been requested for ~2 years.

If you're in ECS and require these missing features, you must resort to hacks until there is a metadata service.

[IngmarStein]

For anyone looking how to get the dynamically allocated host port (for example for service discovery registration), here's a gist that illustrates @jwconway's workaround: https://gist.github.com/IngmarStein/c2495962534ba4a6cc8791bb25635607

Instead of relying on the shortened hostname, the code above reads the docker container id from the /proc filesystem.

[TheLastWilson]

I really can't believe this hasn't been resolved. awslogs-stream-prefix is great and all if you are using awslogs. I'm trying to use syslog and it seems impossible to correlate the hostname/docker id to the ecs container id

[DougHaber]

On the logging side, this would be a lot easier if the ECS provided the task id via template markup for the logging tag. For example, Docker provides these:

https://docs.docker.com/engine/admin/logging/log_tags/

If there was a {{.TaskID}}, then we could set the tag to something like {{.TaskId}}/{{.Name}}. If using syslog, we can then extract it on receipt and set the file path based on that information. As it is now, I don't see a way to group containers by task with the syslog logging driver. Without being able to separate out containers by task, it is hard to debug multiple instances of a task that has more than one container. This makes the syslog logging driver (and probably others) not very useful on ECS.

[lucabotti]

@IngmarStein your gist fails at line 108 - not really into GO but...

[IngmarStein]

@lucabotti you can comment on the gist, no need to spam the thread here :) I've added the missing import, thanks for letting me know!

The gist itself is just illustrative and shows how to solve the problem. It's not complete, i.e. you need to do something with the gathered information like passing it to your task.

[cederberg]

Summarizing the hacks mentioned above, all variants seem to rely on the metadata API. Which can be called from within a container as such:

curl "http://172.17.0.1:51678/v1/tasks?dockerid=${HOSTNAME}"

Which results in:

{
    "Arn":"arn:aws:ecs:us-east-1:****:task/****-7732-416c-b8fd-28ad7f628bd2",
    "DesiredStatus":"RUNNING",
    "KnownStatus":"RUNNING",
    "Family":"my-app",
    "Version":"152",
    "Containers":[
        {
            "DockerId":"65f111da430c7d61a6822c50bb9f50150f653f34f4f84335243c647e81d4776d",
            "DockerName":"ecs-my-app-152-memcached-94f0b0e5d289f2ac3200",
            "Name":"memcached"
        },
        {
            "DockerId":"c29e6495a44599778d766e79c6969002d4d364df7932ae784cbec8fe3ce99f2a",
            "DockerName":"ecs-my-app-152-gunicorn-e4948af880b1ccab9d01",
            "Name":"gunicorn"
        },
        {
            "DockerId":"44f9c83df2c45f154dd34640723f4828b49eb0ec18a86b150908d172884c3b0c",
            "DockerName":"ecs-my-app-152-nginx-d6d8d2e3fca1beb73f00",
            "Name":"nginx"
        }
    ]
}

The Arn above corresponds to the taskArn from the aws command-line.

But... I can't seem to find the containerArn in any shape or form... Nor can I correlate the id at the end of the DockerName to anything useful. So the only way to match the aws ecs describe-tasks output is via the taskArn and name fields combined it seems.

It isn't immediately obvious why the DockerId and DockerName fields from the metadata API above hasn't been added to the ECS API container datatype (http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Container.html) already. Which would've solved this whole issue.

[CpuID]

Pretty sure this is now exposed through container metadata: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-metadata.html#metadata-file-format

[bndw]

That's correct- container metadata was introduced in v1.15.0

[samuelkarp]

Container metadata solves one half of this request: determining the task ARN from within the container. I'm leaving this open to track the other half: determining the Docker ID from the ECS API.

[geerlingguy]

The only way I can currently get a running Docker container ID for a task arn (using the ECS Container Agent Introspection is:

1. Log into the ECS instance running the task (convoluted mess to get to this point using automation).
2. Run curl -s 'http://127.0.0.1:51678/v1/tasks?taskarn=$TASK_ARN' | jq -r '.Containers[0].DockerId'

It works for now, but I really wish there were a way for the AWS CLI and/or ECS API to return the docker ID alongside the Task ARN (because outside of a few ECS operations, the Task ARN is not really helpful).

[tomelliff]

I was hoping I might be able to get the container ID/name from the task ARN outside of the container so I could remotely exec into the container without SSHing into the instance. For some context I've been toying with the idea of creating an ECS Gitlab Runner executor to be able to run builds as one off tasks on our ECS clusters.

Ideally for this purpose you would just be able to exec into a running container via the ECS API (there's some requests for this elsewhere) but I'm sure that comes with a whole host of problems so I was considering simply exposing the Docker daemon to allow the Gitlab Runner to remote exec containers that ECS had scheduled on the container instances. The best I can think of right now is to generate a task definition family with a unique ID in it because the container name at least seems to contain the task definition but this seems ugly.

[diegoacuna]

For anyone interested, I wrote a small python script following the steps provided by @geerlingguy to automate the two steps: https://github.com/diegoacuna/ecs-ctl

[nickholl]

FWIW we're just going to move to EKS instead of worrying about ECS any more, this sort of limitation has been around for just too long in ECS

[bradacina]

Wow, it's 2018!

Increment these for me will you:
years_sitting_in_unresolved++;
num_developers_looking_for_this_functionality++;

[rsworden59]

Another vote for this. I'm using the AWS CLI to document various things, and need to get from the ContainerInstance to the Container so I can work on tracking deployment dates, container versions etc. So I'm outside the container.

[adnxn]

It very difficult to track the taskARN to the execution docker container ID. These two number have no relation. How could I find which container(s) the task started/run?

Catching up on this thread - I think we can best track progress on this by splitting it out into a seperate issue.

The original request was -

It very difficult to track the taskARN to the execution docker container ID. These two number have no relation. How could I find which container(s) the task started/run?

The container metadata file provides that this metadata from within the container. Additionally, we've also discussed exposing this through the ECS API.

It works for now, but I really wish there were a way for the AWS CLI and/or ECS API to return the docker ID alongside the Task ARN

Container metadata solves one half of this request: determining the task ARN from within the container. I'm leaving this open to track the other half: determining the Docker ID from the ECS API.

To reiterate @samuelkarp's message, this only solves half the problem. We are also exploring the best way to expose Docker IDs through the ECS API. I'm closing this issue for now and opening https://github.com/aws/amazon-ecs-agent/issues/1662 to track the progress on changes to ECS API. Let's move any additional discussion wrt exposing this metadata through the API to the new thread.

[pavneeta]

Hi Everyone,
Today, August 30th , Amazon ECS launched the capability for customers to lookup the runtime container IDs of containers running as part of an ECS task. Now, the describe-tasks and stop-task APIs return the runtime container id for all the containers within an ECS task. - https://aws.amazon.com/about-aws/whats-new/2019/08/amazon-ecs-now-exposes-runtime-containerids-to-apis-and-ecs-console/