Terraform-provider-aws: ECS service_registries not sticking

[adamgotterer]

Terraform Version

Terraform v0.11.7

• provider.aws v1.16.0
• provider.template v1.0.0

Also tried provider.aws v1.165.0

Affected Resource(s)

• aws_service_discovery_service
• aws_ecs_service

Terraform Configuration Files

resource "aws_service_discovery_service" "headless-chrome" {
  name = "headless-chrome"
  dns_config {
    namespace_id = "${aws_service_discovery_private_dns_namespace.service.id}"
    dns_records {
      ttl = 10
      type = "SRV"
    }
    routing_policy = "MULTIVALUE"
  }

  health_check_custom_config {
    failure_threshold = 2
  }
}

data "template_file" "ecs_task_svc_headless_chrome" {
  template = "${file("task_definitions/svc-headless-chrome.json")}"
}

resource "aws_ecs_task_definition" "svc-headless-chrome" {
  family = "svc-headless-chrome"
  container_definitions = "${data.template_file.ecs_task_svc_headless_chrome.rendered}"
  task_role_arn = "${module.ecs_private.task_iam_profile_id}"
  network_mode = "awsvpc"
}

resource "aws_ecs_service" "svc-headless-chrome" {
  name = "svc-headless-chrome"
  cluster = "${module.ecs_private.cluster_name}"
  task_definition = "${aws_ecs_task_definition.svc-headless-chrome.id}"
  desired_count = "${lookup(var.task_desired_counts, "svc-headless-chrome")}"
  deployment_minimum_healthy_percent = 0

  network_configuration {
    subnets = ["${module.vpc.private_subnets}"]
  }

  service_registries {
    registry_arn = "${aws_service_discovery_service.headless-chrome.arn}"
    port = "9222"
  }

  depends_on = ["aws_ecs_task_definition.svc-headless-chrome"]
}

Expected Behavior

I expected the service to be setup with service registry

Actual Behavior

The service registry wasn't applied to the service. Using the aws cli I pulled the service record and there was no entry for the service registry and running terraform plan a subsequent time shows that it wants to add the registry value.

{
    "services": [
        {
            "serviceArn": "arn:aws:ecs:us-east-1:XXXXXXXXXXXX:service/svc-headless-chrome",
            "serviceName": "svc-headless-chrome",
            "clusterArn": "arn:aws:ecs:us-east-1:XXXXXXXXXXXX:cluster/ecs-staging-private",
            "loadBalancers": [],
            "status": "ACTIVE",
            "desiredCount": 2,
            "runningCount": 2,
            "pendingCount": 0,
            "launchType": "EC2",
            "taskDefinition": "arn:aws:ecs:us-east-1:XXXXXXXXXXXX:task-definition/svc-headless-chrome:52",
            "deploymentConfiguration": {
                "maximumPercent": 200,
                "minimumHealthyPercent": 0
            },
            "deployments": [
                {
                    "id": "ecs-svc/XXXXXXXXXXXXXXXXXXX",
                    "status": "PRIMARY",
                    "taskDefinition": "arn:aws:ecs:us-east-1:XXXXXXXXXXXX:task-definition/svc-headless-chrome:52",
                    "desiredCount": 2,
                    "pendingCount": 0,
                    "runningCount": 2,
                    "createdAt": 1524769220.223,
                    "updatedAt": 1524769220.223,
                    "launchType": "EC2",
                    "networkConfiguration": {
                        "awsvpcConfiguration": {
                            "subnets": [
                                "subnet-481f8465",
                                "subnet-9efdb6d7"
                            ],
                            "securityGroups": [],
                            "assignPublicIp": "DISABLED"
                        }
                    }
                }
            ],
            "roleArn": "arn:aws:iam::XXXXXXXXXXXX:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS",
            "events": [
                {
                    "id": "XXXXXXX-XXXX-XXXX-XXXX-9c704e6f3d67",
                    "createdAt": 1524769248.612,
                    "message": "(service svc-headless-chrome) has reached a steady state."
                },
                {
                    "id": "XXXXXXX-XXXX-XXXX-XXXX-9abbac830c59",
                    "createdAt": 1524769223.979,
                    "message": "(service svc-headless-chrome) has started 2 tasks: (task XXXXXXX-XXXX-XXXX-8850-fbbfe23a303f) (task XXXXXX-XXXXX-XXXX-91a7-5323ae9aede5)."
                }
            ],
            "createdAt": 1524769220.223,
            "placementConstraints": [],
            "placementStrategy": [],
            "networkConfiguration": {
                "awsvpcConfiguration": {
                    "subnets": [
                        "subnet-XXXX465",
                        "subnet-XXXX6d7"
                    ],
                    "securityGroups": [],
                    "assignPublicIp": "DISABLED"
                }
            }
        }
    ],
    "failures": []
}

# Initial apply
  + aws_ecs_service.svc-headless-chrome
      id:                                             <computed>
      cluster:                                        "ecs-staging-private"
      deployment_maximum_percent:                     "200"
      deployment_minimum_healthy_percent:             "0"
      desired_count:                                  "2"
      iam_role:                                       <computed>
      launch_type:                                    "EC2"
      name:                                           "svc-headless-chrome"
      network_configuration.#:                        "1"
      network_configuration.0.assign_public_ip:       "false"
      network_configuration.0.subnets.#:              "2"
      network_configuration.0.subnets.1436254125:     "subnet-XXXXXXX"
      network_configuration.0.subnets.643886198:      "subnet-XXXXXXX"
      service_registries.#:                           "1"
      service_registries.~948308048.port:             "9222"
      service_registries.~948308048.registry_arn:     "${aws_service_discovery_service.headless-chrome.arn}"
      task_definition:                                "${aws_ecs_task_definition.svc-headless-chrome.id}"

# Subsequent apply, you can see it want's to add the service_registries
-/+ aws_ecs_service.svc-headless-chrome (new resource required)
      id:                                         "arn:aws:ecs:us-east-1:XXXXXXX:service/svc-headless-chrome" => <computed> (forces new resource)
      cluster:                                    "ecs-staging-private" => "ecs-staging-private"
      deployment_maximum_percent:                 "200" => "200"
      deployment_minimum_healthy_percent:         "0" => "0"
      desired_count:                              "2" => "2"
      iam_role:                                   "aws-service-role" => <computed>
      launch_type:                                "EC2" => "EC2"
      name:                                       "svc-headless-chrome" => "svc-headless-chrome"
      network_configuration.#:                    "1" => "1"
      network_configuration.0.assign_public_ip:   "false" => "false"
      network_configuration.0.subnets.#:          "2" => "2"
      network_configuration.0.subnets.1436254125: "subnet-XXXXXXX" => "subnet-XXXXXXX"
      network_configuration.0.subnets.643886198:  "subnet-XXXXXXX" => "subnet-XXXXXXX"
      service_registries.#:                       "0" => "1" (forces new resource)
      service_registries.935493940.port:          "" => "9222"
      service_registries.935493940.registry_arn:  "" => "arn:aws:servicediscovery:us-east-1:XXXXXXX:service/srv-XXXXXXX"
      task_definition:                            "svc-headless-chrome:53" => "${aws_ecs_task_definition.svc-headless-chrome.id}"

[adamgotterer]

I pulled down my state file and removed every reference to services in the plan and pushed the state file back up. It appears to be working as expected now. Feels like something got our of sync and couldn't recover.

[nikovirtala]

Using following versions ecs_service creation ends in to an error when trying to assign service registry.

• terraform 0.11.7
• provider aws 1.24.0
• provider template 1.0.0

  service_registries.3438573323.registry_arn:        "" => "arn:aws:servicediscovery:eu-west-1:123456789012:namespace/ns-hoggwkp7p2lefor5"
  task_definition:                                   "" => "arn:aws:ecs:eu-west-1:123456789012:task-definition/resdash:10"
Releasing state lock. This may take a few moments...

Error: Error applying plan:

1 error(s) occurred:

* aws_ecs_service.service: 1 error(s) occurred:

* aws_ecs_service.service: InvalidParameterException: The format of the supplied service registry ARN is invalid: arn:aws:servicediscovery:eu-west-1:123456789012:namespace/ns-hoggwkp7p2lefor5.
    status code: 400, request id: 91fb981d-7941-11e8-b7e5-29454a8d84a0 "resdash"

for me the namespace ARN looks OK.

$ aws servicediscovery list-namespaces
{
    "Namespaces": [
        {
            "Type": "DNS_PRIVATE",
            "Id": "ns-hoggwkp7p2lefor5",
            "Arn": "arn:aws:servicediscovery:eu-west-1:123456789012:namespace/ns-hoggwkp7p2lefor5",
            "Name": "containerized.service.local"
        }
    ]
}

[sfzylad]

This looks more like AWS issue to me. I landed here because I was troubleshooting the The format of the supplied service registry ARN is invalid error that I got from Cloud Formation.

[ldormoy]

@nikovirtala I ran into the exact same issue and finally found the issue: the ecs registry_arn paramater takes the ARN of aws_service_discovery_service as value.

It seems you used aws_service_discovery_private_dns_namespace instead

[nikovirtala]

Thanks for the tip @ldormoy!

[theaayushanand]

can i specify multiple port in the service registry reference within the task definition ?
will this work in case different tasks want to talk to either of the port through private dns and port number ?
I am setting as type "SRV" in my dns_config .

[mathieufortin01]

I think im experimenting the same issue, where an update to the service_registries does not trigger an actual update (and neither an update to the state file) even though an update to be done IS detected and a success message is shown on apply:

````
aws_ecs_service.app: Modifying... (ID: arn:aws:ecs:xxx:service/xxx)
service_registries.1722657496.container_name: "" => ""
service_registries.1722657496.container_port: "0" => "0"
service_registries.1722657496.port: "0" => "0"
service_registries.1722657496.registry_arn: "arn:aws:servicediscovery:xxx:service/srv-xxx-old" => ""
service_registries.426326138.container_name: "" => "xxxxxxx"
service_registries.426326138.container_port: "" => ""
service_registries.426326138.port: "" => ""
service_registries.426326138.registry_arn: "" => "arn:aws:servicediscovery:xxx:service/srv-xxx-updated"
aws_ecs_service.app: Modifications complete after 0s (ID: arn:aws:ecs:xxx:service/xxx)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
````

The only workaround I found is to remove the service_registries block, which forces the ecs_service to be destroyed/re-created and then re-add the service_registries block.

[nidr0x]

I think im experimenting the same issue, where an update to the service_registries does not trigger an actual update (and neither an update to the state file) even though an update to be done IS detected and a success message is shown on apply:

aws_ecs_service.app: Modifying... (ID: arn:aws:ecs:xxx:service/xxx)
  service_registries.1722657496.container_name: "" => ""
  service_registries.1722657496.container_port: "0" => "0"
  service_registries.1722657496.port:           "0" => "0"
  service_registries.1722657496.registry_arn:   "arn:aws:servicediscovery:xxx:service/srv-xxx-old" => ""
  service_registries.426326138.container_name:  "" => "xxxxxxx"
  service_registries.426326138.container_port:  "" => ""
  service_registries.426326138.port:            "" => ""
  service_registries.426326138.registry_arn:    "" => "arn:aws:servicediscovery:xxx:service/srv-xxx-updated"
aws_ecs_service.app: Modifications complete after 0s (ID: arn:aws:ecs:xxx:service/xxx)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

The only workaround I found is to remove the service_registries block, which forces the ecs_service to be destroyed/re-created and then re-add the service_registries block.

I have the same problem :(

[JoshuaEdwards1991]

I added this to the service

lifecycle { create_before_destroy = true }

Then removed the original service_registries block, then updated the name of the service with a 2, so it doesn't throw the idempotent error, and deploy.

Then add the new service_registries block, and remove the 2 from the name, and deploy.

[eedwards-sk]

Service Registries for ECS are straight up broken.

It's "successfully modifying" on an apply, but never actually changing anything, and then always wanting to plan again. Ouch.

https://gist.github.com/eedwards-sk/f25b3a4226b25ce1361d2e78c3432816

[eedwards-sk]

ECS services aren't modifiable -- seems like it's a bug that it's getting marked as "change in place" instead of "force recreate"

[jry-anchor]

Confirmed this needs to cause force recreate: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

Service discovery can only be configured when first creating a service. Updating existing services to configure service discovery for the first time or change the current configuration is not supported.

[DeltaByte]

This is still a problem on Terraform 0.13-beta3 with aws provider 3.0