Terraform: Invalid for_each argument with undetermined resource before apply.

[marcincuber]

Terraform Version

v0.12.16
provider.aws v2.40.0

Terraform Configuration Files

So I have a modules which takes in a list of strings argument:

module "elasticache-redis" {
  source_security_group_ids = [aws_security_group.bastion.id, aws_security_group.test_sg.id]
}

The security groups list is used by following resource:

resource "aws_security_group_rule" "ingress_security_groups" {
  for_each = toset(var.source_security_group_ids)

  type                     = "ingress"
  from_port                = var.port
  to_port                  = var.port
  protocol                 = "tcp"
  source_security_group_id = each.value
  security_group_id        = aws_security_group.redis.id
}

and my security group is simply:

resource "aws_security_group" "test_sg" {
  name_prefix = "test-sg-"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Debug/Error Output

I am getting the following error:

Error: Invalid for_each argument

  on ../modules/elasticache-redis/main.tf line 72, in resource "aws_security_group_rule" "ingress_security_groups":
  72:   for_each = toset(var.source_security_group_ids)

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

Releasing state lock. This may take a few moments...
make: *** [tf-plan-dev] Error 1

Expected Behavior

I would expect for_each function to be able to handle values that will be computed later on in the process and not fail.

Actual Behavior

It is failing with an error mentioned above.

Additional Context

I believe this is a known issue and it has been happening since version v.0.12.6. We are now 10 version ahead and it doesn't seem to be fixed.

Any chances there is a workaround or a fix in progress? To be fair if there is a workaround using dynamic block or loops I would be more than happy to use it.

Thanks

[adamhorden]

I have been facing this exact same problem. I have been passing complex data structures in and out of modules and was hoping to use for_each.

If I do not use a resource attribute and a static value this works fine. When I use resource attributes such as in your case it leads me to the same error.

This is a major limitation of for_each. I would expect for_each to be able to loop around resource attributes.

@marcincuber I was going to report this issue but was struggling to come up with a test case, as I am working on a large framework, your test case is perfect to show the issue.

Thanks,

Adam Horden

[jbardin]

Hi @marcincuber,

Sorry you're having an issue with this, but it appears that everything is working as intended. In order to create a plan for the aws_security_group_rule, terraform needs to know the number of instances _and their keys_, which means that the values of for_each must be known before apply. Because the values are computed attributes of other resources, they are not known during the plan phase, which means they cannot be used in a for_each expression.

Possible solutions to this include applying multiple configurations, or using -target to initialize part of the configuration. Another solution using partial application may be possible in the future, and we are tracking that feature in #4149.

We use GitHub issues for tracking bugs and enhancements rather than for questions. While we can sometimes help with certain simple problems here, it's better to use the community forum where there are more people ready to help. The GitHub issues here are monitored only by our few core maintainers.

[marcincuber]

Hi @jbardin, thanks for your comprehensive response and opinion. But I do believe for_each case is an issue and feature tracked in #4149 for past 4 year is clearly not going to solved any time soon.

Additionally, if you passing two elements into the list of strings i.e. variable type list(string), for_each should be able to deduce that is has to iterate twice based on the list. Nevertheless, I do understand it is the design you guys came up with and it is not easily fixable.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.