Containers-roadmap: ecs update force-new-update does not trigger a deployment
Summary
When certain conditions are met, the aws ecs update-service --cluster CLUSTER --service SERVICE --force-new-deployment does not update the service (nothing happens, also not in the logs).
Description
The conditions I observed while having this issue are:
- task definition not modified
- desired_count = 1
- deployment_minimum_healthy_percent = 50
- deployment_maximum_percent = 100
- launch_type = EC2
- docker plugin Rexray installed and used by a container
This behavior happens with all of my ECS services meeting these conditions, but I will focus on prometheus as example.
Expected Behavior
After running aws ecs update-service --cluster prometheus --service prometheus --force-new-deployment, the running task is stopped and a new task using the newest matching ECR image.
Observed Behavior
The deployment "order" is properly registered:
Deployment Id Status Desired count Pending count Running count Created time Updated time
ecs-svc/9223370479462856384 PRIMARY 1 0 0 2019-05-09 10:51:59 +0200 2019-05-09 10:51:59 +0200
ecs-svc/9223370479464906040 ACTIVE 1 0 0 2019-05-09 10:17:49 +0200 2019-05-09 10:51:59 +0200
ecs-svc/9223370479466921099 ACTIVE 1 0 1 2019-05-09 09:44:14 +0200 2019-05-09 10:07:45 +0200
The ecs agent does not stop the running task.
Environment Details
- task definition
{
"ipcMode": null,
"executionRoleArn": "arn:aws:iam::1234567890:role/prometheus-ecs_task_execution",
"containerDefinitions": [
{
"dnsSearchDomains": null,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "prometheus",
"awslogs-region": "eu-west-1",
"awslogs-stream-prefix": "master"
}
},
"entryPoint": null,
"portMappings": [
{
"hostPort": 9090,
"protocol": "tcp",
"containerPort": 9090
}
],
"command": null,
"linuxParameters": null,
"cpu": 717,
"environment": [
{
"name": "AWS_DEFAULT_REGION",
"value": "eu-west-1"
}
],
"resourceRequirements": null,
"ulimits": null,
"dnsServers": null,
"mountPoints": [
{
"readOnly": null,
"containerPath": "/prometheus",
"sourceVolume": "rexray-prometheus"
}
],
"workingDirectory": null,
"secrets": [],
"dockerSecurityOptions": null,
"memory": null,
"memoryReservation": 717,
"volumesFrom": [],
"stopTimeout": null,
"image": "1234567890.dkr.ecr.eu-west-1.amazonaws.com/prometheus:current",
"startTimeout": null,
"dependsOn": null,
"disableNetworking": null,
"interactive": null,
"healthCheck": {
"retries": 3,
"command": [
"CMD-SHELL",
"wget -qO - http://localhost:9090/-/healthy || exit 1"
],
"timeout": 5,
"interval": 10,
"startPeriod": 60
},
"essential": true,
"links": null,
"hostname": null,
"extraHosts": null,
"pseudoTerminal": null,
"user": null,
"readonlyRootFilesystem": null,
"dockerLabels": null,
"systemControls": null,
"privileged": null,
"name": "prometheus"
},
{
"dnsSearchDomains": null,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "prometheus",
"awslogs-region": "eu-west-1",
"awslogs-stream-prefix": "pushgateway"
}
},
"entryPoint": null,
"portMappings": [
{
"hostPort": 9091,
"protocol": "tcp",
"containerPort": 9091
}
],
"command": null,
"linuxParameters": null,
"cpu": 205,
"environment": [],
"resourceRequirements": null,
"ulimits": null,
"dnsServers": null,
"mountPoints": [],
"workingDirectory": null,
"secrets": null,
"dockerSecurityOptions": null,
"memory": null,
"memoryReservation": 205,
"volumesFrom": [],
"stopTimeout": null,
"image": "1234567890.dkr.ecr.eu-west-1.amazonaws.com/prometheus-pushgateway:current",
"startTimeout": null,
"dependsOn": null,
"disableNetworking": null,
"interactive": null,
"healthCheck": {
"retries": 3,
"command": [
"CMD-SHELL",
"wget -qO - http://localhost:9091/-/healthy || exit 1"
],
"timeout": 5,
"interval": 10,
"startPeriod": 60
},
"essential": true,
"links": null,
"hostname": null,
"extraHosts": null,
"pseudoTerminal": null,
"user": null,
"readonlyRootFilesystem": null,
"dockerLabels": null,
"systemControls": null,
"privileged": null,
"name": "pushgateway"
}
],
"placementConstraints": [],
"memory": "1024",
"taskRoleArn": "arn:aws:iam::1234567890:role/prometheus-ecs_task_execution",
"compatibilities": [
"EC2"
],
"taskDefinitionArn": "arn:aws:ecs:eu-west-1:1234567890:task-definition/prometheus:25",
"family": "prometheus",
"requiresAttributes": [
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.task-eni"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.docker-plugin.rexray/ebs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.container-health-check"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.secrets.ssm.environment-variables"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
}
],
"pidMode": null,
"requiresCompatibilities": [
"EC2"
],
"networkMode": "awsvpc",
"cpu": "1024",
"revision": 25,
"status": "ACTIVE",
"proxyConfiguration": null,
"volumes": [
{
"name": "rexray-prometheus",
"host": null,
"dockerVolumeConfiguration": {
"autoprovision": false,
"labels": null,
"scope": "shared",
"driver": "rexray/ebs",
"driverOpts": null
}
}
]
}
- output of
ecs update --force-new-deploymentcommand
{
"service": {
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-***********",
"subnet-***********"
],
"securityGroups": [
"sg-************"
],
"assignPublicIp": "DISABLED"
}
},
"launchType": "EC2",
"enableECSManagedTags": false,
"loadBalancers": [],
"desiredCount": 1,
"clusterArn": "arn:aws:ecs:eu-west-1:1234567890:cluster/prometheus",
"serviceArn": "arn:aws:ecs:eu-west-1:1234567890:service/prometheus",
"deploymentConfiguration": {
"maximumPercent": 100,
"minimumHealthyPercent": 50
},
"createdAt": 1551780397.866,
"schedulingStrategy": "REPLICA",
"placementConstraints": [],
"deployments": [
{
"status": "PRIMARY",
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-***********",
"subnet-***********"
],
"securityGroups": [
"sg-***********"
],
"assignPublicIp": "DISABLED"
}
},
"pendingCount": 0,
"launchType": "EC2",
"createdAt": 1557391919.423,
"desiredCount": 1,
"taskDefinition": "arn:aws:ecs:eu-west-1:1234567890:task-definition/prometheus:25",
"updatedAt": 1557391919.423,
"id": "ecs-svc/9223370479462856384",
"runningCount": 0
},
{
"status": "ACTIVE",
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-***********",
"subnet-***********"
],
"securityGroups": [
"sg-***********"
],
"assignPublicIp": "DISABLED"
}
},
"pendingCount": 0,
"launchType": "EC2",
"createdAt": 1557389869.767,
"desiredCount": 1,
"taskDefinition": "arn:aws:ecs:eu-west-1:1234567890:task-definition/prometheus:25",
"updatedAt": 1557389869.767,
"id": "ecs-svc/9223370479464906040",
"runningCount": 0
},
{
"status": "ACTIVE",
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-***********",
"subnet-***********"
],
"securityGroups": [
"sg-***********"
],
"assignPublicIp": "DISABLED"
}
},
"pendingCount": 0,
"launchType": "EC2",
"createdAt": 1557387854.708,
"desiredCount": 1,
"taskDefinition": "arn:aws:ecs:eu-west-1:1234567890:task-definition/prometheus:25",
"updatedAt": 1557389265.734,
"id": "ecs-svc/9223370479466921099",
"runningCount": 1
}
],
"serviceName": "prometheus",
"events": [
{
"message": "(service prometheus) has reached a steady state.",
"id": "a1f46f4d-0a75-4661-bc48-ee105b94efde",
"createdAt": 1557388788.178
},
{
"message": "(service prometheus) has started 1 tasks: (task 4befb18c-0bf4-477f-af13-69cc626d0ca3).",
"id": "6df75672-0d99-4e34-b06b-3b46c6a9601d",
"createdAt": 1557388726.234
},
{
"message": "(service prometheus) was unable to place a task because no container instance met all of its requirements. The closest matching (container-instance fb427c19-2ae4-4d23-a02c-ba0091fe19c4) doesn't have the agent connected. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.",
"id": "ffe6c2c6-67b3-475f-8140-f018285d73d0",
"createdAt": 1557388376.62
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "8816e6da-1c96-478c-b008-73298a3440bf",
"createdAt": 1557363488.647
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "fe7c9f0f-fbec-4d7d-a42a-b2939329a1ca",
"createdAt": 1557341867.69
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "b4a7735f-c312-4ce9-b03f-25cd09474e74",
"createdAt": 1557320248.342
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "26113fb6-f836-45c6-a6a7-6fe79fdd3c54",
"createdAt": 1557298623.831
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "ada78c19-6415-4abc-bc01-6e6d48b96900",
"createdAt": 1557277023.094
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "732b2966-5fa2-4626-bfba-162a6db6b002",
"createdAt": 1557255395.87
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "1f1de15e-f556-4fe2-aee0-68cb9e3801fe",
"createdAt": 1557233767.862
},
{
"message": "(service prometheus) has started 1 tasks: (task a793cd2a-d6b8-4558-b4cf-773736ea9241).",
"id": "778a4cc6-4c14-4248-af8b-86f5294d0d37",
"createdAt": 1557233709.834
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "d0d8c374-407d-4413-8086-f285e831fb3a",
"createdAt": 1557214162.219
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "2eb561ff-b13a-4a55-8bcb-cfda99e498d3",
"createdAt": 1557192560.685
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "bdbbd14a-18c6-4ae5-becb-a467ffa9331b",
"createdAt": 1557170940.149
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "f8412ea3-f420-40c1-b6fb-90b7ea56fc97",
"createdAt": 1557149335.516
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "eb663604-b88c-4914-8d18-4564240ad88d",
"createdAt": 1557127733.251
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "73aabe44-aa89-4b71-a1d2-82313b63a313",
"createdAt": 1557106120.083
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "1575686b-5011-4cc7-a8f6-d1f0e32f6823",
"createdAt": 1557084490.431
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "1a610f7f-eec3-4ff2-9296-6655e9840e42",
"createdAt": 1557062858.563
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "4f4f3647-1606-4dde-970e-76d208c13d15",
"createdAt": 1557041240.351
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "18f29481-c809-4a6d-a958-81d37239b3e5",
"createdAt": 1557019632.217
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "752c5d9e-301f-4a5b-99ee-c8a07ac3d5e8",
"createdAt": 1556998002.338
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "6fdae7cc-3524-4ae6-aec7-9096ecaaef3a",
"createdAt": 1556976382.412
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "e9424f72-6ef6-44d2-bfee-9dc077fd73b2",
"createdAt": 1556954769.724
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "41a3dc66-1d53-4366-898a-179596f169cc",
"createdAt": 1556933165.216
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "d56a599a-7570-4f0d-9758-727e1e5cb0dc",
"createdAt": 1556911537.473
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "dfe3050c-9fa0-45bd-89ce-c44dda68c991",
"createdAt": 1556889934.575
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "419ee773-32ba-457e-899e-458ee0116195",
"createdAt": 1556868315.829
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "696b8b06-8c29-4473-9936-316c5209c4ce",
"createdAt": 1556846687.002
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "ddf73bad-9472-4be4-ae39-251044c899c9",
"createdAt": 1556825078.643
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "178bb6a7-b1f1-46e3-b5fc-e901e44cc399",
"createdAt": 1556803465.511
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "decc3534-6e3c-4260-8c51-4e7dc845762a",
"createdAt": 1556781863.833
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "814048f5-d2f2-40df-beeb-354c5b99f623",
"createdAt": 1556760262.945
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "c13dce42-6c88-4694-90fc-b2012e67f402",
"createdAt": 1556738635.648
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "973ef9f5-85eb-49f1-a9e4-ba748540073c",
"createdAt": 1556717034.08
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "1d395762-246b-45bb-a88d-571d6829a001",
"createdAt": 1556695428.031
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "4e630a96-1fa2-4f50-b87d-17e5629f164e",
"createdAt": 1556673818.455
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "cd44543d-795b-4717-a65f-599d64bf90eb",
"createdAt": 1556652202.768
},
{
"message": "(service prometheus) operations are being throttled on service discovery. Will try again later.",
"id": "45f18508-b4e8-46f0-8ad6-92d2d5f9074b",
"createdAt": 1556652168.722
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "ad43b278-6f0d-4109-933c-54f2e3a5e8e1",
"createdAt": 1556651963.434
},
{
"message": "(service prometheus) operations are being throttled on service discovery. Will try again later.",
"id": "45ba9435-b941-492d-92bc-ab7a8dd501d6",
"createdAt": 1556651930.169
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "cdbeca41-e779-4722-9cb4-af7cbae94a8b",
"createdAt": 1556632571.113
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "608abddf-0d1f-4bae-b0cc-4d522d409f9c",
"createdAt": 1556610940.076
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "9348e35e-ea0b-41db-942a-e13223099fbe",
"createdAt": 1556589308.018
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "562eae56-0bbc-4e00-9c4d-8fb6a1c2e020",
"createdAt": 1556567693.847
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "9fc30deb-2a5e-4d73-95a7-7707e6d55e80",
"createdAt": 1556546069.755
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "df349e3b-fa03-4665-a7c3-2ce4900be5a7",
"createdAt": 1556524445.912
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "303f4116-7802-4e1f-a7ba-1069667998dd",
"createdAt": 1556502819.965
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "56852ae9-ad23-4025-b852-37fe9dc5dc6c",
"createdAt": 1556481191.483
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "646db482-0a80-4a11-bd2f-9d9908e21821",
"createdAt": 1556459583.098
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "bc21c3dc-a9b5-4c0b-8e04-66402fd86820",
"createdAt": 1556437971.043
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "2a383121-5741-401a-bea6-0fc360d48aa0",
"createdAt": 1556416357.789
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "c1755478-e7bd-4b1c-8d7d-f56e2caaaff2",
"createdAt": 1556394740.688
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "534953cc-7200-4c32-aff9-c1ef27918215",
"createdAt": 1556373115.39
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "ba93ee1d-63db-45f1-9265-b2361b07874b",
"createdAt": 1556351499.992
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "19bea8ab-f205-4574-b4dd-236bdc495f42",
"createdAt": 1556329876.304
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "b1b348fc-2376-4962-a36f-987ff23fc620",
"createdAt": 1556308253.927
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "33460848-da9d-4f33-a1fc-c5afb9e54e25",
"createdAt": 1556286625.817
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "c4439aee-96b3-4f60-86b4-2dd1625b5ba6",
"createdAt": 1556265010.511
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "f2a08536-cbf7-4f4f-8641-25557cee486b",
"createdAt": 1556243378.482
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "cc0d04f2-06cf-42c0-9152-9a628949948e",
"createdAt": 1556221749.612
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "954b7034-ef48-4d76-899a-d6ace4a80343",
"createdAt": 1556200121.231
},
{
"message": "(service prometheus) has started 1 tasks: (task 5ed3be04-09bc-4a26-a1e2-5d7d19a4f51e).",
"id": "f5fc3aba-a7d4-4772-87a8-60255f9eae70",
"createdAt": 1556199945.688
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "8265769e-ff26-41fb-b087-b68596892a25",
"createdAt": 1556198973.36
},
{
"message": "(service prometheus) has started 1 tasks: (task 584585f5-a189-40cf-9a31-9efd88b03147).",
"id": "90ff7057-60e8-4874-8efb-d78a06cb9073",
"createdAt": 1556198916.538
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "4621833c-e49d-4ede-9977-fd60666fbc29",
"createdAt": 1556198016.777
},
{
"message": "(service prometheus) has started 1 tasks: (task 316f93da-84dc-4564-af02-6a8cefb9bab8).",
"id": "323357d6-b76a-4a2a-bf7c-5379b6c216b8",
"createdAt": 1556197960.535
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "dd097c66-4e7e-4f18-9c90-8d456eee79c8",
"createdAt": 1556192515.103
},
{
"message": "(service prometheus) has started 1 tasks: (task f1c40dd9-e7ef-4b99-b9d4-5143fe1c9970).",
"id": "bfe5ab1e-34c8-48cb-bec6-344a2fd380b0",
"createdAt": 1556192354.431
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "a51a8a70-e941-4dc7-bb46-50ec6e80cd58",
"createdAt": 1556180425.551
},
{
"message": "(service prometheus) has started 1 tasks: (task 539816ad-24eb-40ea-be9f-44910edbbc76).",
"id": "41d568d1-33aa-4072-a19d-b9ae74f1238a",
"createdAt": 1556180308.388
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "bf1021bc-66a6-4ae0-8c1d-b76355ba8c33",
"createdAt": 1556161382.064
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "49c1af8a-a194-4731-bddb-d4948c823953",
"createdAt": 1556139761.506
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "862e7162-3522-42e8-baca-1ebdaca91b48",
"createdAt": 1556118115.784
},
{
"message": "(service prometheus) has started 1 tasks: (task b717090e-09bd-4ff9-9035-4277ff9a317e).",
"id": "ca008d05-8a4e-467e-8149-69bba71c32fb",
"createdAt": 1556118071.591
},
{
"message": "(service prometheus) has started 1 tasks: (task e19b59af-9b8d-4726-adf4-c54423da6a0e).",
"id": "4aaaa55c-f6c5-43c3-97f3-c1f963f941ad",
"createdAt": 1556117741.791
},
{
"message": "(service prometheus) has started 1 tasks: (task db78a2c2-718e-490e-ad43-72bea7503ea9).",
"id": "910ee1e7-ac49-4e73-a0de-e401e3c5a5df",
"createdAt": 1556117685.718
},
{
"message": "(service prometheus) has started 1 tasks: (task 7026baab-528d-401c-a33d-dc3374f0dd87).",
"id": "cd921c4e-b57e-4967-a36d-7639d0e4fc45",
"createdAt": 1556117635.436
},
{
"message": "(service prometheus) was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.",
"id": "125c58a7-064c-4b56-acbd-07cdce81ab0b",
"createdAt": 1556117338.598
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "c110e963-a263-4da0-acff-2ffc80797762",
"createdAt": 1556096127.952
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "f26449f5-3812-49a4-afcb-5b075613f91a",
"createdAt": 1556074497.88
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "67f60c0e-96b6-437d-b260-a34d3c1e64cc",
"createdAt": 1556052877.65
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "a900ae4c-f23f-4dbf-80c4-528dd590846e",
"createdAt": 1556031268.713
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "2c4c050a-be13-4c50-9246-c7d5bdde7884",
"createdAt": 1556009643.772
},
{
"message": "(service prometheus) has started 1 tasks: (task 3d61799d-c0cf-4714-91ee-39cf22e010ab).",
"id": "8325de33-f1b0-407e-881a-bd4723996406",
"createdAt": 1556009553.167
},
{
"message": "(service prometheus) has stopped 1 running tasks: (task ab17c23b-e254-4817-b05f-a0e69b2d9a14).",
"id": "5bdc9ebb-3f69-4948-a743-1390b7eba0f3",
"createdAt": 1556009530.683
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "f83f1241-53e4-447f-8fe0-3b81eefb1dd9",
"createdAt": 1555991675.569
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "106e9d2a-5c25-4fd8-9d42-455858b93bf5",
"createdAt": 1555970053.498
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "f36a49e0-99dd-4c36-a8c0-5ff424c6aae3",
"createdAt": 1555948440.223
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "441999e8-913a-4e8a-83f9-fe877d027b1f",
"createdAt": 1555926815.408
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "8ed65911-db5f-428b-83c5-a03d0d4596c0",
"createdAt": 1555905191.637
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "48e60b36-8efd-44cc-865b-86ed393e3cde",
"createdAt": 1555883568.191
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "b80d029c-5fac-42f9-b2ef-364fa6f97c00",
"createdAt": 1555861949.217
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "f54155ee-50c6-4102-a6de-cdfbd1e16b50",
"createdAt": 1555840339.788
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "5c6f687c-7c92-41fa-9146-71e7b17222f5",
"createdAt": 1555818723.972
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "b9a22477-c1f5-42bc-8d46-54074f5ff9cf",
"createdAt": 1555797097.707
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "d92a04c0-9ed6-4459-8ad0-099349d05834",
"createdAt": 1555775493.981
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "e72f2ecf-fd9b-458b-8c8c-84fa888e405e",
"createdAt": 1555753867.31
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "a46ac223-5fb7-4b15-9a20-568da01fcf22",
"createdAt": 1555732237.142
},
{
"message": "(service prometheus) has reached a steady state.",
"id": "a012b4c0-1d8b-4507-b3c1-5a97a16e1216",
"createdAt": 1555710620.939
}
],
"runningCount": 1,
"status": "ACTIVE",
"serviceRegistries": [
{
"registryArn": "arn:aws:servicediscovery:eu-west-1:1234567890:service/srv-***********"
}
],
"pendingCount": 0,
"placementStrategy": [],
"propagateTags": "NONE",
"roleArn": "arn:aws:iam::1234567890:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS",
"taskDefinition": "arn:aws:ecs:eu-west-1:1234567890:task-definition/prometheus:25"
}
}
- docker info
Containers: 7
Running: 5
Paused: 0
Stopped: 2
Images: 6
Server Version: 18.06.1-ce
Storage Driver: devicemapper
Pool Name: docker-docker--pool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: ext4
Udev Sync Supported: true
Data Space Used: 522.2MB
Data Space Total: 23.33GB
Data Space Available: 22.81GB
Metadata Space Used: 3.06MB
Metadata Space Total: 25.17MB
Metadata Space Available: 22.11MB
Thin Pool Minimum Free Space: 2.333GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.106-79.86.amzn1.x86_64
Operating System: Amazon Linux AMI 2018.03
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.912GiB
Name: ip-10-1-50-149
ID: WQOC:WITR:7H6R:2CPX:H4MS:KKAI:M7RO:CYK2:FCKO:CAKO:O4L4:CN3H
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
- curl http://localhost:51678/v1/metadata
{"Cluster":"prometheus","ContainerInstanceArn":"arn:aws:ecs:eu-west-1:1234567890:container-instance/4e660ed5-5ad5-4118-aa9d-f206ca7d9a84","Version":"Amazon ECS Agent - v1.27.0 (aabe65ee)"}
- df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 970M 100K 970M 1% /dev
tmpfs 979M 0 979M 0% /dev/shm
/dev/nvme0n1p1 9,8G 897M 8,9G 10% /
Supporting Log Snippets
- ecs-agent.log
2019-05-09T06:40:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:40:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. prometheus to RUNNING, but already RUNNING
2019-05-09T06:40:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:40:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. ~internal~ecs~pause to RUNNING, but already RESOURCES_PROVISIONED
2019-05-09T06:40:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:45:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:45:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. pushgateway to RUNNING, but already RUNNING
2019-05-09T06:45:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:45:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. prometheus to RUNNING, but already RUNNING
2019-05-09T06:45:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:45:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. ~internal~ecs~pause to RUNNING, but already RESOURCES_PROVISIONED
2019-05-09T06:45:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:46:05Z [INFO] TCS Websocket connection closed for a valid reason
2019-05-09T06:46:05Z [INFO] Establishing a Websocket connection to https://ecs-t-2.eu-west-1.amazonaws.com/ws?cluster=prometheus&containerInstance=arn%3Aaws%3Aecs%3Aeu-west-1%1234567890%3Acontainer-instance%2Ffb427c19-2ae4-4d23-a02c-ba0091fe19c4
2019-05-09T06:46:05Z [INFO] Connected to TCS endpoint
2019-05-09T06:50:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:50:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. pushgateway to RUNNING, but already RUNNING
2019-05-09T06:50:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:50:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. prometheus to RUNNING, but already RUNNING
2019-05-09T06:50:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:50:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. ~internal~ecs~pause to RUNNING, but already RESOURCES_PROVISIONED
2019-05-09T06:50:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:51:53Z [INFO] Begin building map of eligible unused images for deletion
2019-05-09T06:51:53Z [INFO] Image excluded from deletion: [Image: [ImageID: sha256:e9688394b472c654fb3f49b90796af35ef2c0f84848af73912cd4253e5f1541b; Names: amazon/amazon-ecs-pause:0.1.0] referenced by 1 containers; PulledAt: 2019-05-07 12:51:53.418218013 +0000 UTC m=+0.739529031; LastUsedAt: 2019-05-07 12:51:53.418219043 +0000 UTC m=+0.739530011; PullSucceeded: false]
2019-05-09T06:51:53Z [INFO] No eligible images for deletion for this cleanup cycle
2019-05-09T06:51:53Z [INFO] End of eligible images for deletion: No more eligible images for deletion; Still have 3 image states being managed
2019-05-09T06:55:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:55:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. pushgateway to RUNNING, but already RUNNING
2019-05-09T06:55:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:55:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. prometheus to RUNNING, but already RUNNING
2019-05-09T06:55:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:55:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: redundant container state change. ~internal~ecs~pause to RUNNING, but already RESOURCES_PROVISIONED
2019-05-09T06:55:53Z [INFO] Managed task [arn:aws:ecs:eu-west-1:1234567890:task/a793cd2a-d6b8-4558-b4cf-773736ea9241]: task at steady state: RUNNING
2019-05-09T06:57:08Z [INFO] ACS Websocket connection closed for a valid reason
2019-05-09T06:57:08Z [INFO] Establishing a Websocket connection to https://ecs-a-2.eu-west-1.amazonaws.com/ws?agentHash=aabe65ee&agentVersion=1.27.0&clusterArn=prometheus&containerInstanceArn=arn%3Aaws%3Aecs%3Aeu-west-1%3A1234567890%3Acontainer-instance%2Ffb427c19-2ae4-4d23-a02c-ba0091fe19c4&dockerVersion=DockerVersion%3A+18.06.1-ce&sendCredentials=false&seqNum=1
2019-05-09T06:57:08Z [INFO] Connected to ACS endpoint
ldormoy
All 4 comments
We always have a lot of issues with desired_count=1. I think it's because ECS won't allow there to be no tasks running with deployment_minimum_healthy_percent = 50. I forgot if the behavior differs between a normal deployment and a force-update but as a workaround I suggest changing minimum_health_percent to 0 or setting desired_count to 2.
Schweigi
on 16 May 2019
👍5
👎1
desired_count=2 isn't really an option, I need a single container running at all times.
deployment_minimum_healthy_percent=0 did the trick, the service now properly stops the running task and starts a new one.
Thanks!
ldormoy
on 16 May 2019
👍1
I am not sure if this is relevant, but if your desired=1 and your minimumPercent=50%, then make sure your maximumPercent=200%, so ECS is able to temporarily have two replicas.
whereisaaron
on 21 Jun 2019
👍10
Thanks @whereisaaron - pointed me in the right direction.
kyleian
on 14 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
MartinDevillers
路
3Comments
ORESoftware
路
3Comments
tabern
路
3Comments
mineiro
路
3Comments
abby-fuller
路
3Comments
Most helpful comment
I am not sure if this is relevant, but if your
desired=1and yourminimumPercent=50%, then make sure yourmaximumPercent=200%, so ECS is able to temporarily have two replicas.