Terraform: AWS ALB resources managed by TF return errors around invalid ARNs when refreshing state

[hkjn]

Hi TF friends,

When attempting to terraform plan against an environment which last run had an empty diff, the AWS APIs seem to return errors around "invalid ARNs":

status code: 400, request id: 9dd2e471-f1b4-11e6-8528-e7b033ace7d3
        * aws_alb.my_lb: Error retrieving ALB: ValidationError: 'arn:aws:elasticloadbalancing:us-east-1:123456789:loadbalancer/app/my-lb/8901234567' is not a valid load balancer ARN
2017/02/13 06:21:12 [DEBUG] plugin: /usr/local/bin/terraform: plugin process exited
                status code: 400, request id: abcd1234-f1b4-11e6-9247-e16385e163a5
        * aws_alb_target_group.my_group: Error retrieving Target Group: ValidationError: 'arn:aws:elasticloadbalancing:us-east-1:1234567890:targetgroup/my-tg/89012345678abcdef' is not a valid target group ARN
                status code: 400, request id: defgh89012-f1b4-11e6-98eb-e73e8f31432f

However the aws CLI is showing these resources without errors:

$ aws elbv2 describe-load-balancers --region us-east-1 --load-balancer-arns arn:aws:elasticloadbalancing:us-east-1:123456789:loadbalancer/app/my-lb/8901234567
{
    "LoadBalancers": [
            {
                        "VpcId": "vpc-abcd1234",
                        "LoadBalancerArn": "arn:aws:elasticloadbalancing:us-east-1:123456789:loadbalancer/app/my-lb/8901234567",
                        "State": {
                            "Code": "active"
                        },
             [...]
             },
}

And as mentioned Terraform created these resources in the first place for us, with no errors from terraform apply.

We've manually recovered from this state before by doing the following:

1. Disable remote .tfstate config
2. Manually edit .tfstate to remove resources causing issue
3. Manually remove actual AWS resources
4. Plan and apply changes to rebuild resources
5. Delete remote .tfstate config on S3
6. Re-enable remote .tfstate config

After these changes (when first hitting this issue last week), subsequent terraform plan runs worked without error, until now, when a second small change caused these errors to reappear when trying to refresh state.

Once in this state, a terraform plan that's expected to produce an empty diff also produces these errors.

It's hard to say whether the issue is with Terraform logic, or with the AWS APIs used by TF to refresh state. Let me know if I can provide any more info.

Terraform Version

v0.8.6

Affected Resource(s)

• aws_alb
• aws_alb_target_group

Terraform Configuration Files

resource "aws_alb_target_group" "my_group" {
  name     = "my-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = "${var.vpc_id}"
}

resource "aws_alb" "my_lb" {
  name            = "my-lb"
  internal        = true
  security_groups = ["${var.security_groups}"]
  subnets         = ["${var.subnets}"]
}

resource "aws_alb_listener" "http_listener" {
  port              = "6789"
  protocol          = "http"
  load_balancer_arn = "${aws_alb.my_lb.arn}"

  default_action {
    target_group_arn = "${aws_alb_target_group.my_group.arn}"
    type             = "forward"
  }
}

resource "aws_alb_listener_rule" "my_rule" {
  listener_arn = "${aws_alb_listener.http_listener.arn}"
  priority     = 10

  action {
    type             = "forward"
    target_group_arn = "${aws_alb_target_group.my_group.arn}"
  }

  condition {
    field = "path-pattern"

    values = ["/my_path/*"]
  }
}

resource "aws_alb_target_group_attachment" "my_attachment" {
  target_group_arn = "${aws_alb_target_group.my_group.arn}"
  target_id        = "${var.ec2_instance}"
  port             = 6789
}

Expected Behavior

TF-managed ALB resources have state refreshed without errors.

Actual Behavior

TF-managed ALB resources produce 400 errors about invalid ARNs.

Steps to Reproduce

Once in this state, all attempts to refresh state fail, but we have several very similar plans (for different environments) and only some of them end up in this state, some of the time, so a repro case is difficult at this time.

[hkjn]

Any idea how we can provide more input or help debug what's going wrong here?

This is still hitting us fairly frequently, currently we're using TF version 0.8.8.

[hkjn]

Actually, while the output here from AWS APIs and Terraform is somewhat confusing, it looks like we misunderstood what was going on; our wrappers which ran TF inside a Docker container were not setting AWS_DEFAULT_REGION correctly.

So it looks like this somewhat confusing output is how AWS tries to communicate either "no such resources exist in this region" or "the us-east-1 part of the ARN makes no sense for this region".

I'm still not sure how the manual intervention on the .tfstate described in the issue "fixed" it for us, but I have to assume we ran TF against the correct AWS region after those cleanup steps.

I'll close this since the TF behavior was much more sane than we thought and this was mainly operator error, sorry for the noise!

[micemuni]

Hello,
I am facing the same issue, not a valid target group ARN & not a valid load balancer ARN,
I have verified the ARNs are correct from the region.
Please let me know how to fix. ?

Error: Error refreshing state: 2 error(s) occurred:

• module.ais.aws_lb_target_group.ais_nlb: 1 error(s) occurred:

• module.ais.aws_lb_target_group.ais_nlb: aws_lb_target_group.ais_nlb: Error retrieving Target Group: ValidationError: 'arn:aws:elasticloadbalancing:us-east-1:xxxxxxxxxxxxx:targetgroup/xxxxxx-us1-alb-target/ebde06216a600581' is not a valid target group ARN
  status code: 400, request id: ecfa3f74-5902-11e8-bb87-494cd5f68135

• module.ais.aws_lb.ais_nlb: 1 error(s) occurred:

• module.ais.aws_lb.ais_nlb: aws_lb.ais_nlb: Error retrieving ALB: ValidationError: 'arn:aws:elasticloadbalancing:us-east-1:xxxxxxxxxxxxxxxxx:loadbalancer/net/xxxxxxx-us1-nlb/f65622b3c4d1bc7c' is not a valid load balancer ARN
  status code: 400, request id: ed343b93-5902-11e8-adbe-25bc1659d76f

[dtelaroli]

For me the problem was the AWS credential, because I had used two profiles.
I had changed the provider to correct profile and worked fine.

[default]
aws_access_key_id = <secret>
aws_secret_access_key = <secret>
[my_profile]
aws_access_key_id = <secret>
aws_secret_access_key = <secret>

provider "aws" {
  region  = "${var.region}"
  profile = "my_profile"
}

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.