Terraform-provider-aws: intermittent issue with aws_appautoscaling_target and aws_appautoscaling_policy creation

[hashibot]

_This issue was originally opened by @brikis98 as hashicorp/terraform#10737. It was migrated here as part of the provider split. The original body of the issue is below._

---

Terraform Version

Terraform v0.7.7

Affected Resource(s)

• aws_appautoscaling_target
• aws_appautoscaling_policy

Terraform Configuration Files

resource "aws_appautoscaling_target" "target" {
  name = "MyService"
  resource_id = "service/MyCluster/MyService"
  role_arn = "${aws_iam_role.role.arn}"
  min_capacity = 1
  max_capacity = 10
}

resource "aws_appautoscaling_policy" "scale_out" {
  name = "MyServiceScaleOut"
  resource_id = "service/MyCluster/MyService"

  adjustment_type = "ChangeInCapacity"
  cooldown = 60
  metric_aggregation_type = "Average"

  step_adjustment {
    metric_interval_lower_bound = 0
    scaling_adjustment = 1
  }

  depends_on = ["aws_appautoscaling_target.target"]
}

Expected Behavior

The auto scaling policy and target are both created successfully.

Actual Behavior

Intermittently, you get an error like this:

Error putting scaling policy: ObjectNotFoundException: No scalable target registered for service namespace: ecs (... sorry, I forgot to copy/paste the rest of the error ...)

If you re-run terraform apply, the error usually goes away.

Steps to Reproduce

1. terraform apply

Important Factoids

If you re-run terraform apply, sometimes multiple times, it works. I suspect this is a classic issue with the fact that AWS is asynchronous and eventually consistent and Terraform is not properly waiting for the aws_appautoscaling_target resource to be created.

[aleerizw-zz]

I am still experiencing the issue in Terraform v0.10.2. Creating an aws_appautoscaling_policy fails with Error putting scaling policy: ObjectNotFoundException: No scalable target registered for service namespace: ecs, resource ID: service/signin-service/signin-service-test, scalable dimension: ecs:service:DesiredCount

As can be seen by the following log:

module.signin-service-test.aws_appautoscaling_target.target: Creating...
  max_capacity:       "" => "3"
  min_capacity:       "" => "1"
  resource_id:        "" => "service/signin-service/signin-service-test"
  role_arn:           "" => "arn:aws:iam::489198589229:role/ecsAutoscaleRole"
  scalable_dimension: "" => "ecs:service:DesiredCount"
  service_namespace:  "" => "ecs"
module.signin-service-test-auto-scale-down.aws_appautoscaling_policy.policy: Creating...
  adjustment_type:                                        "" => "ChangeInCapacity"
  arn:                                                    "" => "<computed>"
  cooldown:                                               "" => "60"
  metric_aggregation_type:                                "" => "Average"
  name:                                                   "" => "signin-service-test-CPU_util_low-scale-down"
  policy_type:                                            "" => "StepScaling"
  resource_id:                                            "" => "service/signin-service/signin-service-test"
  scalable_dimension:                                     "" => "ecs:service:DesiredCount"
  service_namespace:                                      "" => "ecs"
  step_adjustment.#:                                      "" => "1"
  step_adjustment.2173517692.metric_interval_lower_bound: "" => ""
  step_adjustment.2173517692.metric_interval_upper_bound: "" => "0"
  step_adjustment.2173517692.scaling_adjustment:          "" => "-1"
module.signin-service-test-auto-scale-up.aws_appautoscaling_policy.policy: Creating...
  adjustment_type:                                        "" => "ChangeInCapacity"
  arn:                                                    "" => "<computed>"
  cooldown:                                               "" => "60"
  metric_aggregation_type:                                "" => "Average"
  name:                                                   "" => "signin-service-test-CPU_util_high-scale-up"
  policy_type:                                            "" => "StepScaling"
  resource_id:                                            "" => "service/signin-service/signin-service-test"
  scalable_dimension:                                     "" => "ecs:service:DesiredCount"
  service_namespace:                                      "" => "ecs"
  step_adjustment.#:                                      "" => "1"
  step_adjustment.2087484785.metric_interval_lower_bound: "" => "0"
  step_adjustment.2087484785.metric_interval_upper_bound: "" => ""
  step_adjustment.2087484785.scaling_adjustment:          "" => "1"
module.signin-service-test.aws_appautoscaling_target.target: Creation complete (ID: service/signin-service/signin-service-test)
Error applying plan:

2 error(s) occurred:

* module.signin-service-test-auto-scale-down.aws_appautoscaling_policy.policy: 1 error(s) occurred:

* aws_appautoscaling_policy.policy: Error putting scaling policy: ObjectNotFoundException: No scalable target registered for service namespace: ecs, resource ID: service/signin-service/signin-service-test, scalable dimension: ecs:service:DesiredCount
    status code: 400, request id: fe5e66a1-8712-11e7-af2f-f9c0a5cbf2c7
* module.signin-service-test-auto-scale-up.aws_appautoscaling_policy.policy: 1 error(s) occurred:

* aws_appautoscaling_policy.policy: Error putting scaling policy: ObjectNotFoundException: No scalable target registered for service namespace: ecs, resource ID: service/signin-service/signin-service-test, scalable dimension: ecs:service:DesiredCount
    status code: 400, request id: fe5f9ee6-8712-11e7-92e4-8b76b81521b8

The aws_appautoscaling_target is created before the policy but the policy creation still fails. Re running terraform apply solves the problem.

[spanktar]

I can confirm in 0.10.7

Added a depends_on block to no avail.:

depends_on = ["aws_alb_target_group.alb_tg_external"]

[spanktar]

I have an absolutely horrifying hack that fixes this for now. Read it and weep:
/cc @apparentlymart (thought you'd appreciate this)

resource "aws_appautoscaling_target" "ecs_asp_target" {
  max_capacity       = "${var.globals["container_max"]}"
  min_capacity       = "${var.globals["container_min"]}"
  resource_id        = "service/${var.globals["ecs_name"]}/${aws_ecs_service.ecs_service.name}"
  role_arn           = "${var.globals["iamrole_ecs_autoscaling_arn"]}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"

  lifecycle {
    ignore_changes = ["max_capacity", "min_capacity"]
  }
}

resource "aws_appautoscaling_policy" "ecs_asp_down" {
  count = "${local.enabled}"

  name               = "tf_${var.globals["environment"]}_${var.globals["app_name"]}_scale_down"
  resource_id        = "service/${var.globals["ecs_name"]}/${aws_ecs_service.ecs_service.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"

  step_scaling_policy_configuration {
    adjustment_type         = "ChangeInCapacity"
    cooldown                = 120
    metric_aggregation_type = "Maximum"

    step_adjustment {
      metric_interval_upper_bound = 0
      scaling_adjustment          = -1
    }
  }

  depends_on = ["aws_alb_target_group.alb_tg_external","null_resource.timer"]

}

resource "null_resource" "timer" {
  provisioner "local-exec" {
    command = "sleep 9"
  }
  triggers {
    trigger = "${aws_appautoscaling_target.ecs_asp_target.resource_id}"
  }
}

[luc-tielen]

Any updates on this? Also running into this?

[indranil272]

I'm running into this issue with Terraform version v.0.11.0

Edit: i had typo in my policy

[tpoindessous]

Hi

I just got hit by this issue with

Terraform v0.11.5

• provider.aws v1.46.0

And just like you, I re-ran terraform apply and it fixed the problem.

Thomas

[bflad]

Hi folks 👋 Sorry for the unexpected behavior here. It turns out there was a very subtle bug that prevented proper recreation of Application AutoScaling Policies when only the resource_id argument changed. We also now properly ignore ObjectNotFoundException errors during Terraform resource destroy. A full writeup of these changes can be found in #7982. These fixes will be released in version 2.3.0 of the Terraform AWS Provider in the next day or two.

Even without upgrading your Terraform AWS Provider to the newer version, you _may_ be able to workaround the original issue with a simple configuration update. Using direct references to the relevant aws_appautoscaling_target resource for the resource_id, scalable_dimension, and service_namespace arguments in the aws_appautoscaling_policy resource configuration should be enough in many cases to ensure Terraform has proper ordering information to handle recreations of the Application AutoScaling Policy resource when the underlying Application AutoScaling Target or resource associated with the target changes:

resource "aws_appautoscaling_target" "example" {
  # ... other configuration ...
}

resource "aws_appautoscaling_policy" "example" {
 # ... other configuration ...

  resource_id        = "${aws_appautoscaling_target.example.resource_id}"
  scalable_dimension = "${aws_appautoscaling_target.example.scalable_dimension}"
  service_namespace  = "${aws_appautoscaling_target.example.service_namespace}"
}

If you are still having trouble after upgrading to version 2.3.0 of the Terraform AWS Provider (when its released) and with a configuration looking similar to the above, please create a new GitHub issue with the relevant details from the issue template and we can further triage. Thanks!

[bflad]

This has been released in version 2.3.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[tomelliff]

Oddly I've just hit this issue in one of our temporary environments when the target group changed.

We're using my fork from https://github.com/terraform-providers/terraform-provider-aws/pull/6314 which is based off 2.4.0 (with my change applied over the top) and the above change in https://github.com/terraform-providers/terraform-provider-aws/pull/7982/files looks good to me so I'm not too sure why this has failed.

Partial output from the failed apply:

module.frontend_service.aws_appautoscaling_policy.ecs_web_policy: Modifying... (ID: service/test/frontend-feat-website-ALBRequestCountPerTarget)
  target_tracking_scaling_policy_configuration.0.predefined_metric_specification.0.resource_label: "app/ecs-external-test/REDACTED/targetgroup/feat-website-frontend-ext/REDACTED" => "app/ecs-internal-test/REDACTED/targetgroup/feat-website-frontend-int/REDACTED"
module.frontend_service.aws_lb_listener_rule.host_based_routing: Creating...
  action.#:                      "" => "1"
  action.0.order:                "" => "<computed>"
  action.0.target_group_arn:     "" => "arn:aws:elasticloadbalancing:eu-west-1:REDACTED:targetgroup/feat-website-frontend-int/REDACTED"
  action.0.type:                 "" => "forward"
  arn:                           "" => "<computed>"
  condition.#:                   "" => "1"
  condition.3210165143.field:    "" => "host-header"
  condition.3210165143.values.#: "" => "1"
  condition.3210165143.values.0: "" => "REDACTED"
  listener_arn:                  "" => "arn:aws:elasticloadbalancing:eu-west-1:REDACTED:listener/app/ecs-internal-test/REDACTED/REDACTED"
  priority:                      "" => "<computed>"
module.frontend_service.aws_lb_listener_rule.host_based_routing: Creation complete after 0s (ID: arn:aws:elasticloadbalancing:eu-west-1:...ca27/REDACTED/REDACTED)
module.frontend_service.aws_ecs_service.web_service: Creation complete after 0s (ID: arn:aws:ecs:eu-west-1:REDACTED:service/test/frontend-feat-website)
module.frontend_service.null_resource.wait_for_service_deploy: Creating...
  triggers.%:               "" => "1"
  triggers.task_definition: "" => "arn:aws:ecs:eu-west-1:REDACTED:task-definition/frontend-feat-website:17"
module.frontend_service.null_resource.wait_for_service_deploy: Provisioning with 'local-exec'...
module.frontend_service.null_resource.wait_for_service_deploy (local-exec): Executing: ["/bin/sh" "-c" "aws ecs wait services-stable --services frontend-feat-website --cluster test --region eu-west-1"]
module.frontend_service.null_resource.wait_for_service_deploy: Still creating... (10s elapsed)
module.frontend_service.null_resource.wait_for_service_deploy: Still creating... (20s elapsed)
module.frontend_service.null_resource.wait_for_service_deploy: Still creating... (30s elapsed)
module.frontend_service.null_resource.wait_for_service_deploy: Creation complete after 31s (ID: 5160621590471942553)

Error: Error applying plan:

1 error(s) occurred:

* module.frontend_service.aws_appautoscaling_policy.ecs_web_policy: 1 error(s) occurred:

* aws_appautoscaling_policy.ecs_web_policy: Failed to update scaling policy: ObjectNotFoundException: No scalable target registered for service namespace: ecs, resource ID: service/test/frontend-feat-website, scalable dimension: ecs:service:DesiredCount
    status code: 400, request id: c377d376-5b97-11e9-9fd1-07a87949a6fb

[tomelliff]

Oh, derp, misread the change.

I guess we need to catch the ObjectNotFoundException during update and if so retry it?

[bflad]

Thanks for the additional fix here @tomelliff, which looks more appropriate for this particular issue. I've updated the milestone here to version 2.6.0 since it will likely be more helpful. Version 2.6.0 is planned to ship later today.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!