Terraform: produce map/object from nested for loop in terraform >0.12

[sleterrier]

Current Terraform Version

Terraform v0.12.6dev

Use-cases

While I found some examples on how to produce a list of maps, I am currently failing at producing a map of maps with a _nested for loop_.

How would you go about producing:

Outputs:

association-map = {
  "policy1" = "user1"
  "policy2" = "user1"
  "policy2" = "user2"
}

From:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

Attempted Solutions

I have tried many variations of:

locals {
  association-map = merge({
    for policy, users in var.iam-policy-users-map : {
      for user in users : {
        policy => user
      }
    }
  })
}

with zero success so far. I have only managed to get the following errors depending on the variation:

Error: Invalid 'for' expression. Extra characters after the end of the 'for' expression.
Error: Missing attribute value. Expected an attribute value, introduced by an equals sign ("=").
Error: Invalid 'for' expression. Key expression is required when building an object.
Error: Missing key/value separator. Expected an equals sign ("=") to mark the beginning of the attribute value.

References

• #20288

Proposal

Assuming this is doable and I am just too dumb to figure it out (very likely), some documentation would be really helpful.

[teamterraform]

Hi @sleterrier ,
I refuse to believe that you are dumb - you're using terraform, after all! 😁You're also not mistaken.

Terraform 0.12 does not currently support nested for loops. You might find someone who can come up with a clever way to generate the structure you need in the community forum where there are more people ready to help, but I don't believe there's a direct function that will help here. Most of the functions work on maps, which can't have duplicate keys. It might also help to know more details of your use case, so we can see if there's another way to accomplish your goal.

I think there are two feature requests implicit in this issue: nested for expressions, and a function or functions that support creating objects (as opposed to maps, which as already said cannot have duplicate keys) from a for expression.

[sleterrier]

@teamterraform : Thank you for your answer! After sleeping on it and reading your comment, I realized I was stuck in a rabbit hole with no way out. We did prove I am dumb after all :)

Use-case

The use case - which should definitely have been described in the community forum instead of as an issue here - was to allow for _local.association-map_ to be looped over in a _for_each_ resource.
Even if an object with duplicate keys could be produced, it would therefore not have helped me.

Attempted Solutions

I wanted my team to be able to define GCP roles and members in two distinct lists to begin with:

variable admin_roles = {
  default = [
    "roles/resourcemanager.folderAdmin",
    "roles/resourcemanager.folderIamAdmin",
  ]
}

variable admin_members = {
  default = [
    "user:[email protected]",
    "group:[email protected]",
  ]
}

I then create a bindings map out of the two lists and feed it to a _google_folder_iam_binding_ resource with _for_each_:

locals {
  admin_bindings = {
    for role in var.admin_roles:
      role => var.admin_members
  }
}

resource "google_folder_iam_binding" "binding" {
  for_each = local.admin_bindings

  folder = "folder_1234"
  members = each.value
  role = each.key
}

Which works as expected and allows me to remove/add members and roles anywhere in the starting lists without triggering a delete/create of the resulting resources.
But I then realized I did not always want to be authoritative on the roles bindings, so I embarked on trying to produce a map I could feed to a _google_folder_iam_member_ resource with _for_each_:

# !! We established this is not possible !! Don't try this at home.
locals {
  local.admin_bindings_additive = merge({
    for role, members in local.admin_bindings : {
      for member in members : {
        role => member
      }
    }
  })
}

resource "google_folder_iam_member" "member" {
  for_each = local.admin_bindings_additive

  folder = "folder_1234"
  member = each.value
  role = each.key
}

But failed miserably, for obvious reasons now. I have to go back to the drawing board and re-think the whole approach...

Thanks again, and please let me know if you'd rather have me move this thread to the forum so we do not pollute the issues section.

[teamterraform]

No worries! I think your specific case is worth posing to the community forum, but (against all odds) this is the first request for nested for expressions, so we should keep it open as an enhancement.

[sleterrier]

Reference

There is however one last thing I am not clear about, based on #20288. I thought nested for loops were already supported given that:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

locals {
  association-list = flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : {
        user   = user
        policy = policy
      }
    ]
  ])
}

output association-list {
  value = local.association-list
}

would actually produce a list of maps:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

association-list = [
  {
    "policy" = "policy1"
    "user" = "user1"
  },
  {
    "policy" = "policy2"
    "user" = "user1"
  },
  {
    "policy" = "policy2"
    "user" = "user2"
  },
]

[mildwonkey]

HI @sleterrier! I'm the one who make the erroneous statement about nested loops, sorry! Let me spend a little longer thinking about this - I'm still fairly positive that we can't do _precisely_ what you are requesting at this time, but I might be able to come up with a workaround, or at least a better-thought-out feature request :)
Sorry again, cheers and thanks for the update.

[mildwonkey]

Would a list work, in place of a map? You might have to tweak the format output, but I got this:

Outputs:

association-list = [
  "policy1 = user1",
  "policy2 = user1",
  "policy2 = user2",
]

config:

locals {
  association-list = flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : [
        format("%s = %s", policy, user)
      ]
    ]
  ])
}

[sleterrier]

Hi @mildwonkey , thanks for taking a stab at this, much appreciated!

I am however in dire need of a map, so I can create multiple _google_folder_iam_member_ resources by iterating over its keys with _for_each_.
The end goal is to avoid the resulting resource(s) to be tied to a list index, but created with a unique name instead. So any elements of _var.iam-policy-users-map_ could be removed/added without triggering a delete/create of any other elements (hopefully this sentence makes sense).

I am still wondering how the association-list code you pasted above does not constitute a nested for loop though. And how come we could not use a similar logic to create a map of maps? (instead of a list of maps in my example or a list of strings in your example). But we already proved I am dumb, so that could just be that.

[mildwonkey]

It is a nested loop! The error was mine, I misstated. Nested loops do indeed work.

One issue with what you're trying to do is that this is an invalid map. A map, by definition, does not allow for duplicate keys, and you have "policy2" twice:

association-map = {
  "policy1" = "user1"
  "policy2" = "user1"
  "policy2" = "user2"
}

Map keys _have_ to be unique.

[sleterrier]

Oh, got it now, the different usernames confused me :)

I did realize my fist attempt was foolish because of the duplicate keys it would have produced. Even if such an object could have been created, it would not have helped me iterate over it with _for_each_.
I am however still interested in seeing how one could produce a map (of, let's say maps) in nested for loops?

[sleterrier]

To clarify, here is the new question.
Would there be a way to produce:

Outputs:

association-map = {
  "policy1_user1" = {
    "policy1" = "user1"
  }
  "policy2_user1" = {
    "policy2" = "user1"
  }
  "policy2_user2" = { 
    "policy2" = "user2" 
  }
}

From:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

If that's doable and unless I am missing something again, I believe that would allow for what I have in mind with for_each.

[mildwonkey]

It is possible to create a nested map of maps. The problem is that you need the map key at the top level.

This is (obviously) a useless structure, but I wanted to illustrate that it is possible:

locals {
  association-list = {
    for policy, users in var.iam-policy-users-map:
      policy => {      // can't have the nested for expression before the key!
        for u in users:
           policy => u...
      }
  }
}

Outputs:

association-list = {
  "policy1" = {
    "policy1" = [
      "user1",
    ]
  }
  "policy2" = {
    "policy2" = [
      "user1",
      "user2",
    ]
  }
}

Your example is not _directly_ possible using for expressions because you don't have all the information you need to format the 'key' before stepping into the nested loop, ie $policy_$user

This brings us back around to my earlier response: there may very well be a workaround to get the result you are looking for, but nothing direct nor obvious. Still, a good question for the community forum! Now that we've clarified my earlier mistake (re nested loops), I am going to remove the enhancement label.

No one here is dumb, even me (and I was feeling pretty dumb earlier, which is why I switched to my personal account to own up to the mistake). Sometimes we might sound or feel dumb, but we aren't dumb. I promise :)

[sleterrier]

Thank you very much for your answers @mildwonkey , that was really helpful. I had made changes to the expected output on my previous comment in the meantime - a bad edit habit of mine - which I reverted to keep this thread intelligible. I will take it to the forums from here if needed.

I believe the original intent of my issue remains though: some documentation on nested loops would be really helpful.

[sleterrier]

@apparentlymart gave a great answer in the community forum in case of anyone else was interested. Thanks again for all your help, much appreciated!

[arcotek-ltd]

I asked a similar question is SO: https://stackoverflow.com/questions/58343258/iterate-over-nested-data-with-for-for-each-at-resource-level

Although the answer solved the problem at the time, I've now adjusted the data set meaning I need to traverse three tiers.

[markchalloner]

This can be reduced down to a single variable:

variable "instance_types" {
  type = list(string)
  default = ["c5-2xlarge", "r5-4xlarge"]
}

locals {
  availability_zones = ["a", "b", "c"]
  # Create a map of availability zones and instance_types maps:
  # { ..., "c5-2xlarge-"b"" = {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ...,
  #   ..., "r5-4xlarge-"b"" = {"availability_zone" = "b", "instance_type" = "r5-4xlarge"}, ... }
  instance_types_availability_zones = {
    for instance_type_availability_zone in
      # Flatten the list of lists of availability zones and instance_types maps into a single list:
      # [ ..., {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ...,
      #   ..., {"availability_zone" = "b", "instance_type" = "r5-4xlarge"}, ... ]
      flatten(
        # Create a list of lists of availability zones and instance_types maps per instance type:
        # [
        #   [ ..., {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ... ],
        #   [ ..., {"availability_zone" = "b", "instance_type" = "r5-4xlarge"}, ... ]
        # ]
        [
          for instance_type in var.instance_types: [
            # Create a list of maps of the availability zone and instance_type:
            # [ ..., {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ... ]
            for availability_zone in local.availability_zones: {
              availability_zone = availability_zone
              instance_type = instance_type,
            }
          ]
        ]
    ):
      "${instance_type_availability_zone.instance_type}-${instance_type_availability_zone.availability_zone}"
        => instance_type_availability_zone
  }
}

output "instance_types_availability_zones" {
  value = local.instance_types_availability_zones
}

[abdrehma]

My crack at it:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

output "association-map" {
 value = flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : {
        policy = user
      }
    ]
  ])
}

Outputs:

association-map = [
  {
    "policy" = "user1"
  },
  {
    "policy" = "user1"
  },
  {
    "policy" = "user2"
  },
]

[mhumeSF]

@abdrehma The output association-map produces a value not consumable by for_each.

It'd need to be something like transforming

default = {
  "policy1" = [ "user1" ]
  "policy2" = [ "user1", "user2" ]
}

to

{
  "policy1" = "user1",
  "policy2" = "user1",
  "policy2" = "user2",
}

But this won't work since map keys are unique?

[abdrehma]

@mhumeSF Not pretty, but here's a quick example of turning that list of tuples into a usable map:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

locals{
 association-list =  flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : {
        "${policy}-${user}" = {
          "user" = user
          "policy" = policy
          }
      }
    ]
 ])

 association-map = { for item in local.association-list: 
     keys(item)[0] => values(item)[0]
   }
}

output "association-map" {
   value = local.association-map
}

Output:

association-map = {
  "policy1-user1" = {
    "policy" = "policy1"
    "user" = "user1"
  }
  "policy2-user1" = {
    "policy" = "policy2"
    "user" = "user1"
  }
  "policy2-user2" = {
    "policy" = "policy2"
    "user" = "user2"
  }
}

Example for_each usage:

resource "null_resource" "echo" {
  for_each = local.association-map
  provisioner "local-exec" {
    command = "echo 'policy - ${each.value.policy}, user - ${each.value.user}'"
  }
}

[mhumeSF]

@abdrehma Clever creating the unique key

[skgbox]

As an workaround the following code works for me:

locals {
routes_iteration = flatten([
for route_table_id in local.route_tables_all_ids_svc: [
[
for cidr in [local.vpc_cidr_dev, local.vpc_cidr_tst, local.vpc_cidr_prd]:
"${route_table_id}#${cidr}"
]
]
])
}

resource "aws_route" "svc_to_apps" {
provider = aws.svc
for_each = toset(local.routes_iteration)
route_table_id = split("#", each.value)[0]
destination_cidr_block = split("#", each.value)[1]
transit_gateway_id = aws_ec2_transit_gateway.ad-router.id
}

[sandy724]

BTW there is a hackish solution for nested loop

locals {
  rules = {
    rule_list = [
      {
        from_port    = 2181
        to_port      = 2181
        protocol     = "http"
        cidr         = []
        source_SG_ID = ["abc", "def", "ghi"]
      },
      {
        from_port    = 2181
        to_port      = 2181
        protocol     = "http"
        cidr         = []
        source_SG_ID = ["def"]
      }
    ]
  }

  rule_list1 = [
    for ruleList in local.rules.rule_list:
    {
        from_port = ruleList.from_port
        to_port = ruleList.to_port
        protocol = ruleList.protocol
        cidr = ruleList.cidr
        source_SG_ID = ruleList.source_SG_ID[0]
    }
  ]
  rule_list2 = [
    for ruleList in local.rules.rule_list:
    {
        from_port = ruleList.from_port
        to_port = ruleList.to_port
        protocol = ruleList.protocol
        cidr = ruleList.cidr
        source_SG_ID = ruleList.source_SG_ID[1]
    } if length(ruleList.source_SG_ID) > 1
  ]
  rule_list3 = [
    for ruleList in local.rules.rule_list:
    {
        from_port = ruleList.from_port
        to_port = ruleList.to_port
        protocol = ruleList.protocol
        cidr = ruleList.cidr
        source_SG_ID = ruleList.source_SG_ID[2]
    } if length(ruleList.source_SG_ID) > 2
  ]
  newRules = {
    rule_list = concat(local.rule_list1,local.rule_list2,local.rule_list3)
  }
}

output "rules" {
   value = local.rules.rule_list[0].source_SG_ID
}

output "newRules" {
   value = local.newRules
}

Don't bash me for this hack, but it meets my requirement

[melck]

Any news on this feature? Do you any plan to add it ?

I'm really interested too. At the moment, I use hack way like others or i change my data structure when i can.

Overall, it is really difficult to read / understand unlike nested expressions. I run the risk of losing my colleagues and dissuading them from using Terraform on a complex project.