Terraform-provider-aws: The ids attribute on aws_subnet_ids datasource should be TypeList

[jniesen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.0-alpha4

Affected Datasource(s)

• aws_subnet_ids

Terraform Configuration Files

data "aws_subnet_ids" "destination" {                              
  vpc_id = data.aws_vpc.destination.id                             
  tags   = {                                                       
    SUB-Type = "Public"                                            
  }                                                                   
}                                                                  

data "aws_subnet" "destination" {                                  
  count = length(data.aws_subnet_ids.destination.ids)              
  id    = element(data.aws_subnet_ids.destination.ids, count.index)
}                                                                  

Expected Behavior

The Terraform would iterate over the subnet ids and fetch the corresponding subnet for each one.

Actual Behavior

The following error is produced:

Error: Error in function call

  on modules/info/main.tf line 27, in data "aws_subnet" "destination":
  27:   id    = element(data.aws_subnet_ids.destination.ids, count.index)
    |----------------
    | data.aws_subnet_ids.destination.ids is set of string with 2 elements

Call to function "element" failed: cannot read elements from set of string.

When trying an alternate method of using indices to access the elements of the ids set, the following error is produced:

Error: Invalid index

  on modules/info/main.tf line 27, in data "aws_subnet" "destination":
  27:   id    = data.aws_subnet_ids.destination.ids[count.index]
    |----------------
    | data.aws_subnet_ids.destination.ids is set of string with 2 elements

This value does not have any indices.

Steps to Reproduce

1. terraform apply

[jniesen]

I found the following workaround to produce the expected results:

data "aws_subnet_ids" "destination" {                               
  vpc_id = data.aws_vpc.destination.id                              
  tags   = {                                                        
    SUB-Type = "Public"                                             
  }                                                                                                                             
}                                                                  

locals {                                                            
  subnet_ids_string = join(",", data.aws_subnet_ids.destination.ids)
  subnet_ids_list = split(",", local.subnet_ids_string)             
}                                                                   

data "aws_subnet" "destination" {                                   
  count = length(data.aws_subnet_ids.destination.ids)               
  id    = local.subnet_ids_list[count.index]                        
}                                                                   

[jniesen]

This may not require any changes afterall. It looks like the upcoming beta for Terraform 0.12 includes a tolist interpolation function. If there's a good reason for the ids attribute to be a set rather than a list then it could stay as a list and iterating over each id will still be possible.

[apparentlymart]

Hi @jniesen! Thanks for reporting this and for your perseverance in finding a workaround.

This is an unfortunate situation where what you'd done here was not intentionally working in Terraform 0.11 but had worked by coincidence because the element function is implemented in Go and so the set value was implicitly converted to a Go slice before executing it, and thus it was assigned indices to behave like a list only in that context.

I think the explicit conversion to list you used is the best path forward for now, since this makes it explicit that there's a conversion going on which is assigning an arbitrary order to the set elements that becomes part of the full identifier for each resource. In the case of data resources this is less important since they are not preserved between runs anyway, but it's important in managed resources (resource blocks) to be aware that changes to the ordering of the set elements may cause existing elements to be assigned new indices and thus cause more changes than desired, and so I think the "explicit is better than implicit" design principle applies here.

Although it will not be included in the initial v0.12.0 release, we are hoping to finish hashicorp/terraform#17179 in a minor release, which will get us a way to write this that's even better than what was possible in 0.11:

data "aws_subnet" "destination" {                                   
  for_each = data.aws_subnet_ids.destination.ids

  id = each.value
}

As well as introducing the more convenient each.value accessor, this also tells Terraform to use the strings from the set as the identifiers for the individual elements internally, making aws_subnet.destination behave as a map over those keys rather than as a list. That is, the individual instances can be accessed like data.aws_subnet.destination["subnet-abc123"] rather than data.aws_subnet.destination[0].

Since that special addressing behavior relies on this being a set rather than a list, I think it's better for us to accept the short-term annoyance of requiring an explicit conversion to list here for the longer-term benefit of for_each having a more useful default behavior once it's implemented.

---

I spent a little time trying to figure out if adding tolist(...) around these expressions was something we could reasonably do in the automatic upgrade tool for 0.12, but unfortunately I think this situation is a little too complicated for the upgrade tool to deal with, since it would need to make quite a messy change to each call to element(data.aws_subnet_ids.destination.ids, X).

We may be able to make a compromise of adding a special check to the existing rule that already converts element(X, Y) into X[Y] where it tries a static type-check on X (which may or may not succeed, depending on whether the type is decided dynamically) and produces a warning referring to an entry in the upgrade guide where we can give some more detailed guidance on ways this particular usage could be migrated for 0.12 compatibility.

[mstrejczek]

The tolist workaround works, I think the https://www.terraform.io/docs/providers/aws/d/subnet_ids.html page should be updated to work out of the box:

data "aws_subnet_ids" "example" {
  vpc_id = "${var.vpc_id}"
}

data "aws_subnet" "example" {
  count = "${length(data.aws_subnet_ids.example.ids)}"
  id    = "${tolist(data.aws_subnet_ids.example.ids)[count.index]}"
}

[dpetzold]

Updated docs https://github.com/terraform-providers/terraform-provider-aws/pull/8933

[goodspark]

Since the IDs are a set, tolist will not return a deterministic ordering of subnet IDs.

Since set elements are not ordered, the resulting list will have an undefined order that will be consistent within a particular run of Terraform.

You could sort it, but that will likely break whatever original ordering used in older 0.11 templates. In my case, this will cause an outage unless I put in an additional work to get a graceful migration, which shouldn't be required for what boils down to what should be a no-op change.

As an example, here's a simplified network load balancer setup across different availability zones that I had in 0.11. It's now broken on 0.12 with no good way to 'fix' the template.

data "aws_subnet_ids" "my_vpc" {
  vpc_id = "blah"
  tags = {
    NetworkZone = "external"
  }
}

resource "aws_eip" "lb" {
  count            = "${length(data.aws_subnet_ids.my_vpc.ids)}"
  vpc              = true
  public_ipv4_pool = "amazon"
}

resource "aws_lb" "nlb" {
  load_balancer_type = "network"
  subnet_mapping {
    subnet_id = "${data.aws_subnet_ids.my_vpc.ids[0]}"
    allocation_id = "${aws_eip.lb[0].id}"
  }
  subnet_mapping {
    subnet_id = "${data.aws_subnet_ids.my_vpc.ids[1]}"
    allocation_id = "${aws_eip.lb[1].id}"
  }
  subnet_mapping {
    subnet_id = "${data.aws_subnet_ids.my_vpc.ids[2]}"
    allocation_id = "${aws_eip.lb[2].id}"
  }
}

Options:

• sort(blah)[0] Creates deterministic ordering, but not the original, and thus my NLB has to be recreated
• Dynamic block - Doesn't really solve the problem here, but it will reduce the repeat code
• Hardcoding the subnet and EIP mappings - Works, but means the template now has little use when I want to spin up new mirror (eg. QA/test) environments.

I'd really prefer if the original return type and sorting were restored.

[apparentlymart]

This attribute has always been a set, and so its ordering was never well-defined. This new error message is specifically intended to avoid this trap: to warn you that relying on a particular ordering of these values risks it changing in future versions of Terraform.

Any implicit ordering from before was an implementation detail of the set hashing algorithm used in the provider SDK, so there is unfortunately no reasonable way to restore it in Terraform 0.12's new implementation of sets, which is forced to use a different internal model in order to fix element hash collision bugs in the SDK's set implementation. (More context in hashicorp/terraform#6477.)

Moving forward, using sort will ensure that a consistent ordering is applied regardless of what changes in the underlying set implementation in future. I understand that in some cases that were _already_ assuming a consistent set iteration order this can cause migration problems, and indeed @goodspark's example is a particularly annoying one....

I guess in that case the issue is that the pairings between the subnets and the elastic IP addresses are changing, so the plan is to swap around the elastic IPs into different subnets. For this particular scenario, perhaps using terraform state mv to swizzle the aws_ip.lb instances so that they correspond with the now-sorted subnet ids is the easiest migration path. Unfortunately reordering swapping instances is awkward because terraform state mv can only make one change at a time, and so :sad: it would require using a temporary extra index to hold one of the instances during a swap. For example, to swap iinstances 0 and 1:

terraform state mv aws_eip.lb[0] aws_eip.lb[3]
terraform state mv aws_eip.lb[1] aws_eip.lb[0]
terraform state mv aws_eip.lb[3] aws_eip.lb[1]

Changing this ids attribute to be a list cannot avoid this because there's no way for the data source to know what arbitrary ordering was assigned _before_ in order to be consistent with it: it would need to apply some specific ordering itself, which would end up causing the very same situation that adding an explicit sort(...) causes, where the items are no longer in the same (arbitrary) order as they were before.

[lifeofguenter]

Had the same issue with data.aws_route_tables and solved it exactly as described in the comments here.

[abjrcode]

Is the type returned actually a set or map because I couldn't find anything in official docs mentioning the type set.

If it is a map then this should work right:

count = length(data.aws_subnet_ids.destination.ids)
id = data.aws_subnet_ids.destination.ids[tostring(count.index)]

[malferov]

Hi Ibrahim :)
They mentioned type set in Advanced Type Details section. In your example tolist(data.aws_subnet_ids.destination.ids)[count.index] will work, if order not important, as mentioned above.

[abjrcode]

It didn't work @malferov . It complains about the same error :(

[sergeyr87]

tolist(data.aws_subnet_ids.example.ids)[0] will give you the first element

[jirib]

resource "aws_eip" "lb" {
  count            = "${length(data.aws_subnet_ids.my_vpc.ids)}"
  vpc              = true
  public_ipv4_pool = "amazon"
}

resource "aws_lb" "nlb" {
  load_balancer_type = "network"
  subnet_mapping {
    subnet_id = "${data.aws_subnet_ids.my_vpc.ids[0]}"
    allocation_id = "${aws_eip.lb[0].id}"
  }
  subnet_mapping {
    subnet_id = "${data.aws_subnet_ids.my_vpc.ids[1]}"
    allocation_id = "${aws_eip.lb[1].id}"
  }
  subnet_mapping {
    subnet_id = "${data.aws_subnet_ids.my_vpc.ids[2]}"
    allocation_id = "${aws_eip.lb[2].id}"
  }
}

What about this?

resource "aws_eip" "public" {
  count = length(data.aws_subnet_ids.public.ids)
  vpc      = true
}

resource "aws_lb" "this" {
...
  dynamic "subnet_mapping" {
    for_each = zipmap(data.aws_subnet_ids.public.ids, aws_eip.public.*.id)
    content {
      subnet_id = subnet_mapping.key
      allocation_id = subnet_mapping.value
    }
  }
}

[ish-xyz]

@jniesen I've used the following:

data "aws_subnet_ids" "example" {
  vpc_id = var.vpc_id
}

data "aws_subnet" "example" {
  count = length(data.aws_subnet_ids.example.ids)
  id    = element(tolist(data.aws_subnet_ids.example.ids), count.index)
}

output "subnet_cidr_blocks" {
  value = data.aws_subnet.example.*.cidr_block
}

element() + tolist()

[brianabston001]

I am trying to upgrade to 0.12.18 and I am running on this issue as well. This is what I have and not sure how to get the data with this new version. Any suggestions on how to get this with the for_each setup or a count setup like what you did @ish-xyz ?

existing code:

data "aws_subnet_ids" "task" {
  vpc_id = var.app_vpc_id

  tags = {
    id = "task"
  }
}

resource "aws_lambda_function" "function_name" {
  environment {
    variables = {
      SUBNET_1         = element(tolist(data.aws_subnet_ids.task.ids), count.index + 0)
      SUBNET_2         = element(tolist(data.aws_subnet_ids.task.ids), count.index + 1)
      SUBNET_3         = element(tolist(data.aws_subnet_ids.task.ids), count.index + 2)
    }
  }

error when going to 0.12.18

SUBNET_1         = element(tolist(data.aws_subnet_ids.task.ids), count.index + 0)

The "count" object can be used only in "resource" and "data" blocks, and only
when the "count" argument is set.

SUBNET_2         = element(tolist(data.aws_subnet_ids.task.ids), count.index + 1)

The "count" object can be used only in "resource" and "data" blocks, and only
when the "count" argument is set.

SUBNET_2         = element(tolist(data.aws_subnet_ids.task.ids), count.index + 2)

The "count" object can be used only in "resource" and "data" blocks, and only
when the "count" argument is set.

[mmreddy1582]

anyone please help to advise ,me too facing the same issue .in terraform version 0.12
rror: Invalid function argument

on .terraform/modules/vpc/main.tf line 33, in resource "aws_route_table" "private_RT":
33: tags = merge(var.tags, map("Name", format("%s-rt-private-", var.env, element(var.availability_zones, count.index))))
|----------------
| count.index is 3
| var.availability_zones is tuple with 6 elements

Invalid value for "args" parameter: too many arguments; only 1 used by format
string.

Error: Invalid function argument

on .terraform/modules/vpc/main.tf line 33, in resource "aws_route_table" "private_RT":
33: tags = merge(var.tags, map("Name", format("%s-rt-private-", var.env, element(var.availability_zones, count.index))))
|----------------
| count.index is 4
| var.availability_zones is tuple with 6 elements

Invalid value for "args" parameter: too many arguments; only 1 used by format

[bflad]

Hi folks 👋

The documentation page for the aws_subnet_ids data source has been updated with example usage for Terraform 0.12, notably with for_each which is the typical usage for sets:

data "aws_subnet_ids" "example" {
  vpc_id = var.vpc_id
}

data "aws_subnet" "example" {
  for_each = data.aws_subnet_ids.example.ids
  id       = each.value
}

The EC2 API provides no guarantees about the ordering of results from the DescribeSubnets API calls, therefore the usage of a TypeSet attribute is correct in this case. In the future, it is likely that other data sources that return items non-deterministically will also be updated from TypeList to TypeSet in a major version update if they are not that returning that type already.

Since the documentation now reflects the expected usage from the original report, I'm going to close this issue.

If you're looking for general assistance, please note that we use GitHub issues in this repository for tracking bugs and enhancements with the Terraform AWS Provider codebase rather than for questions. While we may be able to help with certain simple problems here it's generally better to use the community forums where there are far more people ready to help, whereas the GitHub issues here are generally monitored only by a few maintainers and dedicated community members interested in code development of the Terraform AWS Provider itself.

[woodcockjosh]

@bflad thanks for the quick resolution of this issue

[denzhel]

I had the same issue with aws_availability_zones and I managed to find a solution, I hope it's a straight forward one and not a workaround.

My goal was to create 3 subnets each on a different AZ

locals {                                                                                                                          
  aws_azs = data.aws_availability_zones.available
}

resource "aws_subnet" "public-subnet" {
  count                   = length(local.aws_azs["names"])
  availability_zone       = element(local.aws_azs["names"], count.index)
  vpc_id                  = var.vpc_id
  cidr_block              = cidrsubnet(var.public_cidr, 8, count.index)
  map_public_ip_on_launch = var.map_public_ip_on_launch
  tags = {
    Name = "public-subnet-${element(local.aws_azs["names"], count.index)}"
    env  = var.bp_env
  }
}

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!