Terraform: Add a "remove element from list" function

[featheredtoast]

Hi there,

From the documentation there doesn't seem to be a lot of interpolation functions regarding lists, specifically remove element from a list.

I was attempting to work around the aws issue where private_ips were returned in any order. I was attempting to strip the list of the primary IP to guarantee a secondary IP (to print in an aws tag).

I ended up doing something like this:

resource "aws_instance" "my_instance" {
...
tags {
...
lb_ip = "${element(compact(split(",", replace(join(",",aws_network_interface.lb1.private_ips), aws_eip_association.lb1_assoc.private_ip_address, ""))), 0)}"
...
}
}

• convert list to a string
• replace primary private IP (from an eip, bound to the primary IP) with nothing
• convert string to list
• remove empty elements
• get the first element, which is now not going to be a primary IP.

Tasks like this would be much easier to do via something like remove(list, element-to-remove)

Terraform Version

Terraform v0.10.4

References

https://github.com/terraform-providers/terraform-provider-aws/issues/836

[apparentlymart]

Hi @featheredtoast! Thanks for sharing this use-case.

We're about to start some broad work to improve the configuration language, and one of the parts of it is a generic construct for filtering and transforming lists, which I think will serve you use-case here, among others.

With that in mind, I'm going to leave this here for the moment and I'll check in again as we get deeper into the design/implementation of these new features to show what it might look like.

[nhi-vanye]

This would be useful for my use-case. Specifically creating a list of PEERs that a given system should synchronize with (which is basically everything except the current host)

[apparentlymart]

With the configuration language work now further along, I can share what this might look like under the current design:

# NOT YET IMPLEMENTED and details may change before release
  lb_ip = [
    for ip in aws_network_interface.lb1.private_ips:
    ip if ip != aws_eip_association.lb1_assoc.private_ip_address
  ][0]

This is definitely not the most readable example of a for expression, so we may still want to add a specialized function for this use-case to improve readability. However, at least the for expression feature will give a better way to make it work in the mean time, until we're able to add that function.

[davidnortonjr]

We could use this -- or more specifically, a "subtract list from list". Our use case is that a third party provider gives us a list of IPs: A, B, C, D but we know that IPs #C and #D are unhealthy and should not be included in our route 53 entry, so we want to blacklist them for a certain period of time.

[awilkins]

Another example use for "subtract list from list" :

# Want to make a SG rule that only allows HTTPS egress to AWS service endpoints

data "aws_ip_ranges" "aws_services_ireland" {
  regions = ["eu-west-1"]
  services = ["amazon"]
}

# This returns 78 CIDR blocks, but includes all the EC2 blocks which I don't need 
# after subtracting them, 41 blocks, hooray, I can make an SG with that

data "aws_ip_ranges" "aws_ec2_ireland" {
  regions = ["eu-west-1"]
  services = ["amazon"]
}

# Shoot, there's no way to subtract a list from a list.

[tsiq-tek]

I haven't tried this, but under the assumption that:
a) concat will create a list in the order of the arguments it's given; and
b) distinct will indeed discard duplicate items in the order they are found; then

I think the following works? You basically take the item you want to remove and put it at the front of the list, then run distinct to remove the original item, and then slice to get the whole list starting with the _second_ item (the one after the one you wanted to remove).

locals {
  all_items = "${list("foo", "bar")}"
  item_to_remove = "${list("bar")}"
  list_with_item_in_front = "${distinct(concat(local.item_to_remove, local.all_items))}"
  list_without_item = "${slice(local.list_with_item_in_front, 1, length(local.list_with_item_in_front))}"
}

[ramarnat]

Thank you @tsiq-tek

This worked for me:

data "aws_ip_ranges" "aws_ranges" {
  regions  = "${local.include_regions}"
  services = ["AMAZON"]
}

# exclude the routes for S3 and EC2 to reduce the number of cidrs
data "aws_ip_ranges" "exclude_ec2_aws_cidr" {
  regions  = "${local.include_regions}"
  services = ["S3", "EC2"]
}

locals {
  include_regions = "${list("ap-southeast-2", "GLOBAL", "ap-northeast-1")}"
  all_cidrs = "${distinct(concat(data.aws_ip_ranges.exclude_ec2_aws_cidr.cidr_blocks, data.aws_ip_ranges.aws_ranges.cidr_blocks))}"
  included_cidrs = "${sort(slice(local.all_cidrs, length(data.aws_ip_ranges.exclude_ec2_aws_cidr.cidr_blocks), length(local.all_cidrs)))}"
}

[mildwonkey]

Hi folks, thanks for the discussion here!

Terraform 0.12's for expressions support filtering items from lists/maps, so this previously-theoretical example works now:

variable "list" {
  default = [1,2,3,4]
}

// to get a specific element from the list, after filtering
output "list" {
  value = [
    for ip in var.list:
    ip if ip != 3
  ][0]
}

// or just print the whole list (after filtering)
output "list2" {
  value = [
    for ip in var.list:
    ip if ip != 3
  ]
}

$ terraform apply

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

list = 1
list2 = [
  1,
  2,
  4,
]

I am going to close this issue. If you have a new use-case for removing elements from a list that cannot be addressed this way, please open a new issue. Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.