Terraform: [AWS] Creating of ALB, Target Group and Service in one script causes errors

[Timboo89]

Hi there,

Terraform Version

v0.8.8

Affected Resource(s)

• aws_alb
• aws_alb_target_group
• aws_ecs_service

Error

first run

• aws_ecs_service.servicelayer: InvalidParameterException: The target group with targetGroupArn arn:aws:elasticloadbalancing:eu-central-1:xxx:targetgroup/prod-test-servicelayer-tg/xxx does not have an associated load balancer.
  status code: 400, request id: xxx "servicelayer"

second run and further on

• aws_ecs_service.servicelayer: InvalidParameterException: Creation of service was not idempotent.
  status code: 400, request id: xxx "servicelayer"

Cause

• I am starting an ALB, some target groups and a bunch of (container-)services in one terraform - script.
• At the first run the target groups cannot connect to the ALB since that is created but not active yet.
• At the following runs, the ALB is fully loaded, so that the target groups can connect to the ALB, BUT the services are already started, so that the creation of services is not idempotent. Yey!

Ad-hoc Solution

• Set the disired count of the services manually to 0
• delete services
• rerun terraform apply

Desired Solution

It would be great, if you guys could wait until the ALB is active before terraform says it's ready or to check before trying to connect a target group to it and if it's not active yet, wait for it.

[sunckell]

I was able to get around this error by adding a depends_on to the aws_alb_target_group resource.

depends_on = [
"aws_alb.",
]

Not sure if it's the right approach, but it work successfully (consistently). Before I had that, the success rate for the same code was spotty...

[Timboo89]

Thanks, I will try that tomorrow and give a response to that approach.

[guitmz]

IMO this is the right approach, even with Terraform's dependency process being quite good already

[Timboo89]

Well, this didn't do the trick for me, since the ALB and the Target Groups are in different modules.
I did write a little script now, instead of doing terraform apply directly, the script runs first only the load-balancer-module. So that the Load Balancer is fully active before the other modules get applied.

#!/bin/bash
terraform apply -target=module.ecs_load_balancer
terraform apply

[Timboo89]

So in conclusion: A depends_on between modules would be a great idea!

[eedwardsdisco]

@Timboo89 Please re-open this

This is still a bug in terraform 0.9.1

Error applying plan:

2 error(s) occurred:

* module.mgmt_web_green_aws_ecs_service.aws_ecs_service.service: 1 error(s) occurred:

* aws_ecs_service.service: InvalidParameterException: The target group with targetGroupArn arn:aws:elasticloadbalancing:us-west-2:REDACTED:targetgroup/edwards-ml-mgmt-web/REDACTED does not have an associated load balancer.
    status code: 400, request id: a021864c-0d91-11e7-ae70-63189998ff8d "edwards-machlearn-mgmt-web-green"
* module.mgmt_web_blue_aws_ecs_service.aws_ecs_service.service: 1 error(s) occurred:

* aws_ecs_service.service: InvalidParameterException: The target group with targetGroupArn arn:aws:elasticloadbalancing:us-west-2:REDACTED:targetgroup/edwards-ml-mgmt-web/REDACTED does not have an associated load balancer.
    status code: 400, request id: a022becd-0d91-11e7-ae70-63189998ff8d "edwards-machlearn-mgmt-web-blue"

Same issue... I have an ALB in one module which has an output that maps to its ARN.

I then reference that ARN as an input to the service modules.

Terraform does not appear to be waiting for the ALB module's resource to finish getting created before attempting to create the service modules, OR perhaps the AWS API returns the ARN immediately even though the ALB is not actually 'ready' yet (similar to IAM resources), and thus maybe terraform needs a pause\retry behavior to ensure the ALB is 'ready' before proceeding or allowing its outputs to be used.

[aholbreich]

@eedwardsdisco i run in the same.. Alb is created i one module and target gorups and ecs services in another. Something goes wrong.

@sunckell depends_on is not working i my case since target group is created in one module and target gorup in another. Modules can't really depend on each other :(

[sunckell]

@aholbreich - yeah I have been seeing some issues with using a lot of modules.

I was doing this in a blue-green-ecs-alb module. I re-wrote it so that all the blue resources are separated from the green. And that helped me make the module "tighter"...

What I did noticed was that the error: "InvalidParameterException: The target group with targetGroupArn does not have an associated load balancer." was a little mis-leading in my case.

I actually tracked this back to my aws_alb_listener_rule resource. In my case I was trying to register two rules with the same resource. I had to split out the aws_alb_listener_rule resource into a primary and secondary rule and register them separately.....

[aholbreich]

@sunckell thank you for trying to help. I saw simmilat issues.

 "InvalidParameterException: The target group with targetGroupArn does not have an associated load balancer." 

But i didn't got the pointg with

I had to split out the aws_alb_listener_rule resource into a primary and secondary rule and register them separately....

my rule is inside the service-task-target-gorup-module.. so one rule per service...

P.S. mabye i need to be more careful how to destroy resources... some order one by one...

[pug5]

Unless I'm missing something, I believe this issue needs to be reopened. I am currently using Terraform v0.9.6.

The current implementation uses two modules today:

1. The first module creates aws_alb, aws_alb_listener, aws_alb_target_group (as a default). Then I register multiple other services to this ALB.

2. The second module creates a new aws_alb_listener_rule, aws_ecs_service, aws_ecs_task_definition, aws_alb_target_group. I was planning on reusing this module for the other services using the same ALB.

If the current recommended approach is with a depends_on statement, I am unable to use this due to a module (not resource) needing to depend on another module. I receive the error even after the stated workaround of removing the ECS service and rerunning an apply.

* aws_ecs_service.main: InvalidParameterException: The target group with targetGroupArn arn:aws:elasticloadbalancing:us-east-1:XX:targetgroup/XX/XX does not have an associated load balancer.

[agutierrezl]

Still happening in Terraform v0.9.8. A depends_on will be nice.

Thanks

[apparentlymart]

Hi everyone! Thanks for the great discussion here.

Just want to confirm that I'm understanding correctly the issue here. I think we're talking about a situation similar to the following:

resource "aws_alb" "example" {
  # ...
}

resource "aws_alb_target_group" "example" {
  # ...
}

resource "aws_alb_listener" "example" {
  load_balancer_arn = "${aws_alb.example.arn}"

  # ...

  default_action {
    target_group_arn = "${aws_alb_target_group.example.arn}"
    # ...
  }
}

resource "aws_ecs_service" "example" {
  # ...

  load_balancer {
    target_group_arn = "${aws_alb_target_group.example.arn}"
    # ...
  }
}

The problem as I understand it is that the aws_ec2_service gets created after the aws_alb_target_group is created, due to the implied dependency, but it actually needs to wait until after the aws_alb_listener, since the ECS API uses this connection to understand which load balancer should be associated with the ECS service. Is that right?

From the rest of the discussion it seems like some of you were able to solve this by placing an explicit depends_on in the aws_ecs_service resource:

  depends_on = ["aws_alb_listener.example"]

...but that didn't work for others because they are creating the ALB listener and the ECS service in separate modules, and so they can't depend directly on each other.

Is that a good characterization of the problem here?

[aholbreich]

@apparentlymart yes look like good characterization

[josh-padnick]

With helpful input from @brikis98, I was able to use the following workaround to "fake" a dependency on the ALB from within our ecs-service module:

MyModule.tf:

variable "alb_arn" {
   description = "The Amazon Resource Name (ARN) of the ALB that this ECS Service will use as its load balancer."
}

resource "aws_ecs_service" "ecs_service" {
   ...
   # By depending on the null_resource, this resource effectively depends on the ALB existing.
   depends_on = ["null_resource.alb_exists"]

   load_balancer {
      target_group_arn = "${data.template_file.alb_target_group_arn.rendered}"
      ...
   }
}

resource "null_resource" "alb_exists" {
  triggers {
    alb_name = "${var.alb_arn}"
  }
}

[Dwaynekj]

@josh-padnick You are a life saver

[pecigonzalo]

Hey @apparentlymart I believe this should be reopened as it was never fixed actually, just workaround.

[codesurf42]

@pecigonzalo @apparentlymart I can confirm this is still a bug in the latest terraform 11.1, just got to this issue recently. Example ECS code with "dependency fix" workaround https://github.com/codesurf42/terraform-ecs-cluster/commit/54a2b79d6ba602dd2bb8aa11c55bfbd26b391daa

It would be nice to at least mention it in TF doc if this is too hard to be fixed and managed by terraform itself.

[pedros007]

FWIW, I ran into a similar problem when using not TerraForm but CloudFormation. The error is confusing because the ECS Service failed to create because target group/load balancer dependencies:

The target group with targetGroupArn arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/foo-H8CVY7A9IZ3B/946df8097c2b477a does not have an associated load balancer

However, the problem is with both the TargetGroup _and_ any Listeners or ListernerRules! When using CloudFormation I fixed this by setting the following DependsOn dependencies:

• The AWS::ElasticLoadBalancingV2::TargetGroup DependsOnthe AWS::ElasticLoadBalancingV2::LoadBalancer
• The AWS::ECS::Service DependsOn the AWS::ElasticLoadBalancingV2::Listener (and/or AWS::ElasticLoadBalancingV2::ListenerRule, if one exists)

Posting here because it may help when implementing a fix for Terraform.

[weklund]

I've just received the same issue, trying one of the workarounds now, would be nice to be reopened for implementing a proper fix for Terraform

[ayr-ton]

@weklund Could you provide more details about your issue?
It seems that we have different scenarios that end on the same error.

[solsglasses]

I am also having this issue. A depends_on does not fix it. It seems that, although the API is reporting it as complete, the resource is not yet fully created over in AWS-land. So depends_on is almost useless.

If I could, I would add a "wait for 30 seconds" somewhere in there, but I don't think terraform supports that (and I'm not happy doing it anyway).

[pedros007]

@seanscottking did you try what I said? You probably need _two_ depends_on's: The TargetGroup depends_on the ALB. The ECS Service depends_on a Listener and/or ListenerRule.

[simondiep]

I'm noticing that with AWS Provider versions 1.9 and up that @josh-padnick 's proposed workaround no longer works. There must have been a change in how the values resolve in AWS Provider 1.9

data.aws_lb.main: Resource 'data.aws_lb_listener.main' does not have attribute 'load_balancer_arn' for variable 'data.aws_lb_listener.main.load_balancer_arn'

[brikis98]

@josh-padnick and I found we had to update the workaround as follows:

variable "alb_arn" {
   description = "The Amazon Resource Name (ARN) of the ALB that this ECS Service will use as its load balancer."
}

resource "aws_ecs_service" "ecs_service" {
   ...
   # By depending on the null_resource, this resource effectively depends on the ALB existing.
   depends_on = ["null_resource.alb_exists"]

   load_balancer {
      target_group_arn = "${data.template_file.alb_target_group_arn.rendered}"
      ...
   }
}

resource "aws_alb_target_group" "alb_target_group" {
  ...
   # By depending on the null_resource, this resource effectively depends on the ALB existing.
   depends_on = ["null_resource.alb_exists"]
}

resource "null_resource" "alb_exists" {
  triggers {
    alb_name = "${var.alb_arn}"
  }
}

The new addition is that not only does the aws_ecs_service resource depend on the ALB, the aws_alb_target_group does too.

[brikis98]

Why are you using a data sources to fetch the ALB data? Perhaps those data sources are looking up the ALB before it exists?

[simondiep]

I was able to solve this by declaring a data source of the resource I just created. I no longer see the original issue:

* aws_ecs_service.main: 1 error(s) occurred:

* aws_ecs_service.main: InvalidParameterException: The target group with targetGroupArn arn does not have an associated load balancer.
    status code: 400

What I did:

resource "aws_lb_target_group" "default" {
  ...
}

# Workaround to make sure the target group is created before referencing it
# Do not reference the arn here or it will cause the issue
data "aws_lb_target_group" "default" {
  arn = "${aws_lb_target_group.default.arn}"
}

resource "aws_ecs_service" "main" {
  load_balancer {
    target_group_arn = "${aws_lb_target_group.default.arn}"
  }
  ...
  depends_on = ["data.aws_lb_target_group.default"]
}

[fatgit]

I faced this issue. Resolved help add only
depends_on = ["aws_lb_listener_rule.main"] to _aws_ecs_service_

Terraform v0.11.5.

• provider.aws v1.13.0

Now It doesn't need

resource "null_resource" "alb_exists" {
  triggers {
    alb_name = "${var.alb_arn}"
  }
}

[philippeckel]

Like @sunckell suggested, adding

depends_on = [ "aws_alb.<your-load-balancer>", ]

to

resource "aws_alb_target_group"

did the job on Terraform v0.11.7 and AWS Provider v1.14.1.

Thanks a lot!

[inversion]

This should be reopened given that these esoteric workarounds are still necessary.

[athinktank]

I am still having this issue. I have applied all of the h@X suggested above to no joy. This is intermittent and frustrating. For example, plan and apply worked just fine 20 minutes ago. And now on a subsequent destroy and apply, it fails. If you look in the web gui, you the target group clearly has the correct balancer associated.

Error: Error applying plan:

1 error(s) occurred:

• module.ecs.aws_ecs_service.blah: 1 error(s) occurred:

• aws_ecs_service.blah: InvalidParameterException: The target group with targetGroupArn arn:aws:elasticloadbalancing:us-west-1:xxxxxxxxx:targetgroup/dev-dashboard-blah/blahblahblah

does not have an associated load balancer.
status code: 400, request id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx "dev-blah"

[pedros007]

@athinktank Have you tried aws_alb_target_group.blah has a depends_on the aws_alb.blah _and_ your aws_ecs_service.blah has a depends_on the aws_alb_listener.blah?

[athinktank]

I just destroyed my modules and put them all in one file... so far it works.
Modules - no depends_on
Modules - modularity

It seems that there is an underlying issue. There are plenty of threads that speak to this, but there isn't a fix, just compromised code.

Update: And it's still happening and code refactor to allow for depends_on everywhere.

aws_ecs_service.blah: InvalidParameterException: The target group with targetGroupArn arn:aws:elasticloadbalancing:us-west-1:xxxxxxxxxxxx:targetgroup/blah
does not have an associated load balancer.

[subzero112233]

same here. i added the "depends_on" but still receive that error.

[simondiep]

@fatgit 's simple solution worked for me on 0.11.7 Terraform with AWS Provider 1.19

[subzero112233]

@simondiep can you please share your code? I have an ecs service, 2 alb listeners, a target group, log group and a dns record.

[simondiep]

@subzero112233 I removed all depends_on tags from resources and then added depends_on = ["aws_lb_listener_rule.main"] to the ECS Service resource. No other workarounds.

[ryanisnan]

Why is this issue still closed? The fix that has been discussed above (adding explicit dependency to tangential resources, e.g. aws_lb_listener_rule and aws_alb_target_group) are not in line with what the failure messaging indicates.

[mikemac42]

I think this issue should have been copied to the terraform-provider-aws repository.

There is an open issue there that looks the same, although I'm not 100% sure it has the same root cause. https://github.com/terraform-providers/terraform-provider-aws/issues/3495

[sorabhm]

All,

I was also facing the same issue and it was fixed for me after adding ALB resource as a dependency inside the ALB Target Group resource. Given I am creating these resources inside single module as of now, hence didnt face the problem of module level dependency as few has mentioned.

The other part what i looked at is the logs, where i could clearly see that Target groups were getting initialized before ALB activation is completed.

Below are the version of terraform and AWS provider i am currently using
Terraform v0.11.7

• provider.aws v1.28.0

[mildwonkey]

Hi all,

Issues with the terraform aws provider should be opened in the aws provider repository.

Because this closed issue is generating notifications for subscribers, I am going to lock it and encourage anyone experiencing issues with the aws provider to open tickets there.

Please continue to open issues here for any other terraform issues you encounter, and thanks!