Amazon-ecs-agent: Splunk Log Driver
Hi Guys,
First, thanks for including the Splunk log driver in ECS. I am running agent v. 1.11.0 with Docker v. 1.11.2 on the ami-241bd844.
If I try to run the task definition below, I am getting an error. Is there something else I need to specify in order for the Splunk log driver to begin being used? I am able to run the same docker run command with no issues on the same host.
Error:
service example-rest-splunk-dev was unable to place a task because no container instance met all of its requirements. The closest matching container-instance 4792b206-19a3-47e6-822b-70ef075cf3d2 is missing an attribute required by your task. For more information, see the Troubleshooting section.
Task Definition:
{
"taskDefinition": {
"status": "ACTIVE",
"family": "splunk_log_driver_test-dev",
"requiresAttributes": [
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.splunk"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
}
],
"volumes": [],
"taskDefinitionArn": "arn:aws:ecs:us-west-2:539783510382:task-definition/splunk_log_driver_test-dev:4",
"containerDefinitions": [
{
"environment": [
{
"name": "spring_profiles_active",
"value": "dev"
}
],
"name": "example-rest",
"mountPoints": [],
"image": <MY_IMAGE>,
"cpu": 128,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 80,
"hostPort": 80
}
],
"logConfiguration": {
"logDriver": "splunk",
"options": {
"splunk-token": <MY_TOKEN>,
"splunk-url": <MY_URL>,
"splunk-index": <MY_INDEX>
}
},
"memory": 4096,
"essential": true,
"volumesFrom": []
},
{
"environment": [],
"name": "nginx",
"mountPoints": [],
"image": "nginx",
"cpu": 128,
"portMappings": [
{
"protocol": "tcp",
"containerPort": 80,
"hostPort": 8000
}
],
"memory": 128,
"essential": true,
"volumesFrom": []
}
],
"revision": 4
}
}
smugcloud
All 13 comments
@smugcloud can you please paste the output of ecs describe-container-instances --cluster <your-cluster-name> --container-instance <container-instance-iarn>? It looks like you're missing an attribute on your container instance.
Thanks,
Anirudh
aaithal
on 14 Jul 2016
I was wondering how to get all that data @aaithal :D It does indeed look like Splunk is missing from here. Is this supposed to be a part of the default agent?
{
"failures": [],
"containerInstances": [
{
"status": "ACTIVE",
"registeredResources": [
{
"integerValue": 2048,
"longValue": 0,
"type": "INTEGER",
"name": "CPU",
"doubleValue": 0.0
},
{
"integerValue": 7986,
"longValue": 0,
"type": "INTEGER",
"name": "MEMORY",
"doubleValue": 0.0
},
{
"name": "PORTS",
"longValue": 0,
"doubleValue": 0.0,
"stringSetValue": [
"22",
"2376",
"2375",
"51678",
"51679"
],
"type": "STRINGSET",
"integerValue": 0
},
{
"name": "PORTS_UDP",
"longValue": 0,
"doubleValue": 0.0,
"stringSetValue": [],
"type": "STRINGSET",
"integerValue": 0
}
],
"ec2InstanceId": "i-dbbf2306",
"agentConnected": true,
"containerInstanceArn": "arn:aws:ecs:us-west-2:539783510382:container-instance/1b52f3d8-e038-49cb-8cfe-023a1b5c4c0c",
"pendingTasksCount": 0,
"remainingResources": [
{
"integerValue": 2048,
"longValue": 0,
"type": "INTEGER",
"name": "CPU",
"doubleValue": 0.0
},
{
"integerValue": 7986,
"longValue": 0,
"type": "INTEGER",
"name": "MEMORY",
"doubleValue": 0.0
},
{
"name": "PORTS",
"longValue": 0,
"doubleValue": 0.0,
"stringSetValue": [
"22",
"2376",
"2375",
"51678",
"51679"
],
"type": "STRINGSET",
"integerValue": 0
},
{
"name": "PORTS_UDP",
"longValue": 0,
"doubleValue": 0.0,
"stringSetValue": [],
"type": "STRINGSET",
"integerValue": 0
}
],
"runningTasksCount": 0,
"attributes": [
{
"name": "com.amazonaws.ecs.capability.privileged-container"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.20"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.22"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.json-file"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.syslog"
},
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role"
}
],
"versionInfo": {
"agentVersion": "1.11.0",
"agentHash": "c9aefeb",
"dockerVersion": "DockerVersion: 1.11.2"
}
}
]
}
Yeah, that does seem to be the issue. You'd have to override the ECS Agent config to add splunk logging driver using the ECS_AVAILABLE_LOGGING_DRIVERS environment variable config. More details on doing that can be found here.
For example, the following user-data registers the instance with splunk and awslogs logging drivers:
#!/bin/bash
echo ECS_AVAILABLE_LOGGING_DRIVERS='["splunk","awslogs"]' >> /etc/ecs/ecs.config
Thanks @aaithal . For any AWS employees reading this, it would be wonderful if this was included in the recommended AMI's so we don't need to account for this as well.
smugcloud
on 15 Jul 2016
@smugcloud Is there an easy way where we can test the functionality of the splunk logging driver? If we can write a functional test for the splunk log driver (for example, awslogs driver is being tested here), we can look at modifying ecs-init to register the agent with splunk logging driver on startup, by default. Please let us know.
Thanks,
Anirudh
aaithal
on 21 Jul 2016
@aaithal can we just add that ECS_AVAILABLE_LOGGING_DRIVERS line to the /etc/ecs/ecs.config after the fact? We dont want to have to rebuild the cluster.
MaerF0x0
on 21 Jul 2016
@aaithal That test could be modified to validate the existence of the Splunk log driver in the base config. It shouldn't need to be added out of band, if that functionality is provided by default. Just like fluentd, gelf, etc. are exposed as available options in the UI, Splunk should be too.
@MaerF0x0 yes, you can update the ecs.config file on a running cluster.
smugcloud
on 21 Jul 2016
@MaerF0x0 as per @smugcloud's comment, you can just update the /etc/ecs/ecs.config file, then restart ECS Agent by running sudo stop ecs && sudo start ecs on an existing container instance.
@smugcloud we are working on enabling that in the UI. My question was more about writing an end-to-end test in the ECS Agent for splunk logging driver. If you look at the test that I mentioned earlier, we start a task (with container using awslogs driver) and validate that container's logs can be read from the Cloudwatch Logstream. We were wondering if we could something similar with splunk.
aaithal
on 21 Jul 2016
@aaithal I don't have a great suggestion for that. Validating that all of the available drivers are active for the agent is a valuable test as it would have saved me some time, and prevent regressions in the future.
Outside of that, I supposed you'd have to do something similar and ensure calls are being made to the HTTP endpoint.
smugcloud
on 22 Jul 2016
@aaithal We are having the same issue until the env variable is manually configured in agent. Since we are able to specify the log driver in task definition, should the agent take care of the env variable without it being explicitly specified?
ericwush
on 26 Aug 2016
@ericwush Since the logging drivers could be streaming logs to an external endpoint, which could require additional configuration on the instance, we chose not to enable all logging drivers by default on the ECS Agent as there isn't a straight forward/generic scheme for the ECS Agent to validate such configurations, which could lead to task start failures.
aaithal
on 20 Sep 2016
Hi All, we are closing this issue as the only missing item here is enabling the Splunk logging driver by default. Since it's hard for the ECS Agent to verify the driver configuration, we are not in favor of this being enabled by default. You can still override the environment variables and use the Splunk logging driver in your application. Please let us know if you have any additional comments.
Thanks,
Anirudh
aaithal
on 5 May 2017
I am also facing the same issue with my setup. Is there a permanent fix provided by AWS or do we have to run command : echo ECS_AVAILABLE_LOGGING_DRIVERS='["splunk","awslogs"]' >> /etc/ecs/ecs.config on all the ECS nodes?
Looking forward for a response.
sahil-zymr
on 8 Feb 2019
Related issues
leonblueconic
路
3Comments
pspanchal
路
3Comments
sparrc
路
4Comments
taktakpeops
路
4Comments
cjbottaro
路
4Comments
Most helpful comment
@aaithal We are having the same issue until the env variable is manually configured in agent. Since we are able to specify the log driver in task definition, should the agent take care of the env variable without it being explicitly specified?