Terraform: `for_each` and data element is returning an Invalid index on attempt to plan a variation of the for_each map

[jufemaiz]

Terraform Version

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.14.1

Terraform Configuration Files

# -----------------------------------------
# S3 Bucket Notifications
# -----------------------------------------

resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name
  acl    = "private"
}

# -----------------------------------------
# S3 Bucket Notifications
# -----------------------------------------

resource "aws_s3_bucket_notification" "this" {
  bucket = aws_s3_bucket.this.id

  dynamic "topic" {
    for_each = var.s3_sns_notification_details
    content {
      topic_arn     = aws_sns_topic.this[topic.key].arn
      events        = ["s3:ObjectCreated:*"]
      filter_prefix = topic.value.prefix
      filter_suffix = topic.value.suffix
    }
  }
}

# -----------------------------------------
# SNS
# -----------------------------------------

resource "aws_sns_topic" "this" {
  for_each = var.s3_sns_notification_details

  name = each.key
}

resource "aws_sns_topic_policy" "sns_publish" {
  for_each = var.s3_sns_notification_details

  arn = aws_sns_topic.this[each.key].arn

  policy = data.aws_iam_policy_document.sns_publish[each.key].json
}

data "aws_iam_policy_document" "sns_publish" {
  for_each = var.s3_sns_notification_details

  statement {
    sid = "S3PublishToTopic"

    actions = ["SNS:Publish"]

    resources = [aws_sns_topic.this[each.key].arn]

    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
  }
}

Debug Output

~10k lines o.O

Crash Output

N/A

Expected Behavior

Plan the changes and allow me to apply.

Actual Behavior

aws_sns_topic.this["first"]: Refreshing state... [id=arn:aws:sns:ap-southeast-2:372213875655:first]
aws_s3_bucket.this: Refreshing state... [id=tf-data-invalid-index-poc]
data.aws_iam_policy_document.sns_publish["first"]: Refreshing state... [id=3665224188]
aws_sns_topic_policy.sns_publish["first"]: Refreshing state... [id=arn:aws:sns:ap-southeast-2:372213875655:first]
aws_s3_bucket_notification.this: Refreshing state... [id=tf-data-invalid-index-poc]

Error: Invalid index

  on main.tf line 58, in resource "aws_sns_topic_policy" "sns_publish":
  58:   policy = data.aws_iam_policy_document.sns_publish[each.key].json
    |----------------
    | data.aws_iam_policy_document.sns_publish is object with 1 attribute "first"
    | each.key is "second"

The given key does not identify an element in this collection value.

Steps to Reproduce

1. Clone the repository @ https://github.com/jufemaiz/tf-for_each-data-invalid-index-poc

2. terraform apply -var-file=values.tfvars first time

3. Uncomment the second entry for variable value
   s3_sns_notification_details and save

4. terraform apply -var-file=values.tfvars second time and ERROR

   aws_sns_topic.this["first"]: Refreshing state... [id=arn:aws:sns:ap-southeast-2:372213875655:first]
   aws_s3_bucket.this: Refreshing state... [id=tf-data-invalid-index-poc]
   data.aws_iam_policy_document.sns_publish["first"]: Refreshing state... [id=3665224188]
   aws_sns_topic_policy.sns_publish["first"]: Refreshing state... [id=arn:aws:sns:ap-southeast-2:372213875655:first]
   aws_s3_bucket_notification.this: Refreshing state... [id=tf-data-invalid-index-poc]
   
   Error: Invalid index
   
     on main.tf line 58, in resource "aws_sns_topic_policy" "sns_publish":
     58:   policy = data.aws_iam_policy_document.sns_publish[each.key].json
       |----------------
       | data.aws_iam_policy_document.sns_publish is object with 1 attribute "first"
       | each.key is "second"
   
   The given key does not identify an element in this collection value.
   

5. terraform destroy

References

Possibly related:

• https://github.com/hashicorp/terraform/issues/24594 ?

[wparad]

I would expect the problem is that terraform doesn't know the order to apply your resources and is actually creating the second for_each iteration of the data.aws_iam_policy_document.sns_publish after it is being used.

While I would swear to the problem, I will recommend a preferred pattern which is strong coupling. Since the resource aws_sns_topic_policy.sns_publish depends on the data.aws_iam_policy_document.sns_publish using the same for_each iterator is not a good idea. Instead change it to be

resource "aws_s3_bucket_notification" "this" {
  bucket = aws_s3_bucket.this.id

  dynamic "topic" {
    for_each = aws_sns_topic.this
    content {
      topic_arn     = each.value.arn
      events        = ["s3:ObjectCreated:*"]
      filter_prefix = topic.value.prefix
      filter_suffix = topic.value.suffix
    }
  }
}

# -----------------------------------------
# SNS
# -----------------------------------------

resource "aws_sns_topic" "this" {
  for_each = var.s3_sns_notification_details
  name = each.key
}

resource "aws_sns_topic_policy" "sns_publish" {
  for_each = data.aws_iam_policy_document.sns_publish

  arn = each.value.statement.resources.arn
  policy = each.value.json
}

data "aws_iam_policy_document" "sns_publish" {
  for_each = aws_sns_topic.this

  statement {
    sid = "S3PublishToTopic"
    actions = ["SNS:Publish"]
    resources = each.value.arn
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
  }
}

[jbardin]

Hi @jufemaiz

Thanks for filing the issue with the complete reproduction. This type of error has been fixed in 0.14 by the merging of the refresh and plan walks. I confirmed your example works with the latest development release.

@wparad, that is definitely a good workaround if it avoids the error in this case. Both styles should work however (though I do prefer the latter), but unfortunately there are cases where either can be made to fail with Terraform before 0.14, so it's not a guaranteed fix for all configurations.

Thanks!

[rtizzy]

Mentioned this in Slack @jufemaiz but another potential workaround (assuming you know it's safe for your configuration) while waiting for 0.14 would be running the plan or apply without a refresh.

~/r/v/tf-for_each-data-invalid-index-poc ❯❯❯ terraform plan -var-file=values.tfvars -refresh=false             ✘ 1 
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-east-1


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_iam_policy_document.sns_publish["first"] will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "sns_publish"  {
      ~ id      = "2490061105" -> (known after apply)
      ~ json    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action    = "SNS:Publish"
                      - Effect    = "Allow"
                      - Principal = {
                          - AWS = "*"
                        }
                      - Resource  = "arn:aws:sns:us-east-1:446075195327:first"
                      - Sid       = "S3PublishToTopic"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
      - version = "2012-10-17" -> null

      ~ statement {
            actions       = [
                "SNS:Publish",
            ]
          - effect        = "Allow" -> null
          - not_actions   = [] -> null
          - not_resources = [] -> null
            resources     = [
                "arn:aws:sns:us-east-1:446075195327:first",
            ]
            sid           = "S3PublishToTopic"

            principals {
                identifiers = [
                    "*",
                ]
                type        = "AWS"
            }
        }
    }

  # data.aws_iam_policy_document.sns_publish["second"] will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "sns_publish"  {
      + id   = (known after apply)
      + json = (known after apply)

      + statement {
          + actions   = [
              + "SNS:Publish",
            ]
          + resources = [
              + (known after apply),
            ]
          + sid       = "S3PublishToTopic"

          + principals {
              + identifiers = [
                  + "*",
                ]
              + type        = "AWS"
            }
        }
    }

  # aws_s3_bucket_notification.this will be updated in-place
  ~ resource "aws_s3_bucket_notification" "this" {
        bucket = "tf-data-invalid-index-poc-rtizzy"
        id     = "tf-data-invalid-index-poc-rtizzy"

        topic {
            events        = [
                "s3:ObjectCreated:*",
            ]
            filter_prefix = "first"
            filter_suffix = ".csv"
            id            = "tf-s3-topic-20201110141607149100000001"
            topic_arn     = "arn:aws:sns:us-east-1:446075195327:first"
        }
      + topic {
          + events        = [
              + "s3:ObjectCreated:*",
            ]
          + filter_prefix = "second"
          + filter_suffix = ".json"
          + topic_arn     = (known after apply)
        }
    }

  # aws_sns_topic.this["second"] will be created
  + resource "aws_sns_topic" "this" {
      + arn    = (known after apply)
      + id     = (known after apply)
      + name   = "second"
      + policy = (known after apply)
    }

  # aws_sns_topic_policy.sns_publish["first"] will be updated in-place
  ~ resource "aws_sns_topic_policy" "sns_publish" {
        arn    = "arn:aws:sns:us-east-1:446075195327:first"
        id     = "arn:aws:sns:us-east-1:446075195327:first"
      ~ policy = jsonencode(
            {
              - Statement = [
                  - {
                      - Action    = "SNS:Publish"
                      - Effect    = "Allow"
                      - Principal = {
                          - AWS = "*"
                        }
                      - Resource  = "arn:aws:sns:us-east-1:446075195327:first"
                      - Sid       = "S3PublishToTopic"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
    }

  # aws_sns_topic_policy.sns_publish["second"] will be created
  + resource "aws_sns_topic_policy" "sns_publish" {
      + arn    = (known after apply)
      + id     = (known after apply)
      + policy = (known after apply)
    }

Plan: 2 to add, 2 to change, 0 to destroy.

Changes to Outputs:
  ~ s3_sns_notification_details = {
        "first"  = {
            prefix = "first"
            suffix = ".csv"
        }
      + "second" = {
          + prefix = "second"
          + suffix = ".json"
        }
    }

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

This appears to complete successfully.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.