Terraform: Ability to iterate over a list/map and create resources

[garo]

Currently Terraform and HCL can do iteration using count and then the lookup() and element() functions. These work for simple cases, but they fail if there's a need to do a more complex iterations and nested definitions.

Consider this example what I would want to do:

variable "topics" {
  default = ["new_users", "deleted_users"]
}

variable "environments" {
  default = ["prod", "staging", "testing", "development]
}

for $topic in topics {
  # Define SNS topics which are shared between all environments
  resource "aws_sns_topic" "$topic.$env" { ... }

  for $env in environments {
    # Then for each topic define a queue for each env
    resource "aws_sqs_queue" "$topic.$env-processors" { ... }

    # And bind the created queue to its sns topic
    resource "aws_sns_topic_subscription" "$topic.$env-to-$topic.$env-processors" {
      topic_arn = "${aws_sns_topic.${topic.$env}.arn}"
      endpoint = "${aws_sqs_queue.{$topic.$env-processors}.arn}"
    }
  }
}

The especially problematic parts are the topic_arn and endpoint properties in this example, where I want to reference an ARN for another resource, which name is created based on iterating two different lists.

There has been similar suggestions like #4410 and https://github.com/hashicorp/terraform/issues/58

The best what I've come up so far is this:

variable "foo" {
    default = ["1", "2", "3"]
}

variable "bar" {
    default = ["a", "b"]
}

resource "aws_sns_topic" "test" {
    name = "${element(var.foo, count.index / length(var.bar))}-${element(var.bar, count.index)}"
    count = "${length(var.foo) * length(var.bar)}"
}

but this doesn't solve the aws_sns_topic_subscription problem, it's error prone to mistakes as it requires complex expression in the name and count fields.

[mengesb]

@garo A more rich iterator strategy would certainly solve this problem (perhaps loop) but more so you're also looking for nested or dependent interpolation (variables containing interpolation) which certainly... I would love. It does, however, break the declarative nature of things and not all 'creative' solutions are able to function purely declaratively.

I do something similar to what you have with aws_sns_* or aws_sqs_*, but with subnets, networks, etc. I've chosen to go with maps, and utilize length(), element(), keys(), and values().

https://github.com/mengesb/tf_hachef/blob/master/main.tf#L40

In the above link you can see that I dynamically create the subnets based on the information held in the var.subnets map. I have a number of things keyed off this, however to the user, all they really need to do is create a map with suitable key-value pairs that aws consumes. I know this isn't really a solution as it were, but as close to a work-around as possible.

I, for one, would love to see a looping iterator and also a guard of sorts (like a not_if, only_if, etc..), however while not declarative I do have other plans with extremely complex math result sets to control count ... which I'd rather not do as it cannot be understood by many.

[garo]

Thanks @mengesb for the comments. I tend to think that using loops and iterations in this kind of setup doesn't break the declarative nature how Terraform currently works. As HCL is currently translated into JSON, this kind of logic could be implemented into HCL itself, so that it would expand the JSON it produces. So for example a code which would iterate over a list of two entries and declare two resources, the produced JSON would look like there would be no loop but just two declared resources.

There are other languages which have control and especially iteration structures, for example the C++ template metaprogramming which executes at compile time, but which are still declarative in nature as they only produce constant definitions and declarations.

Actually I think that the current limitations forces Terraform users to workaround with a really complex way of using count, element()and other functions, which could be made much more human readable and easier to debug with proper control structures instead.

[fgimian]

I would absolutely love to see loops (and conditionals) in HCL. It would improve Terraform significantly.

I think even our Hashicorp friends are craving for this when you see the effort they had to go to so they can iterate through items in their best-practices repo.

resource "aws_subnet" "public" {
  vpc_id            = "${var.vpc_id}"
  cidr_block        = "${element(split(",", var.cidrs), count.index)}"
  availability_zone = "${element(split(",", var.azs), count.index)}"
  count             = "${length(split(",", var.cidrs))}"

  tags      { Name = "${var.name}.${element(split(",", var.azs), count.index)}" }
  lifecycle { create_before_destroy = true }

  map_public_ip_on_launch = true
}

ewww 😄

Cheers
Fotis

[fgimian]

Just want to clarify that you don't actually have to do what was done above in the best-practices repo. Instead, use lists of the same length for everything and obtain the respective item in each list 😄

e.g.

resource "aws_subnet" "tools" {
  count = "${length(var.azs)}"
  vpc_id = "${var.vpc_id}"
  availability_zone = "${element(var.azs, count.index)}"
  cidr_block = "${element(var.tools_subnet_cidrs, count.index)}"
  tags {
    Name = "Tools ${element(var.az_labels, count.index)}"
  }
}

[davidk01]

I've given up on trying to use the declarative iteration/lookup/interpolation constructs within terraform. Instead whenever I need to create some linked set of resources I do it with ERB. I'm a lot happier after I gave up on trying to do everything from within terraform. This is not a ding against terraform. All custom external DSLs have this problem. It would have been different if terraform was a library like Chef that allowed constructing the template structure with actual code but alas it is not. This actually opens up a possibility for someone to take up that torch. Build a real DSL in a real language that can generate the proper terraform template.

[Crapworks]

Just as a heads up, I had the same problem myself and created a small wrapper that you can use to use Jinja2 templates in terraform. This script also makes all your variables available to Jinja. Just in case anyone is looking for a ready to use script :)

https://github.com/Crapworks/terratools/tree/master/terratemplate

[naikajah]

@garo This is a small implementation I wrote which iterates over a map.
The sample map looks like following:

default = {
    "eu-west-1a" = "10.xx.xx.xx/24,10.xx.xx.xx/24"
    "eu-west-1b" = "10.xx.xx.xx/24,10.xx.xx.xx/24"
    "eu-west-1c" = "10.xx.xx.xx/24,10.xx.xx.xx/24"
  }

And then using the combination of VPC's region, data source aws_availability_zones and some simple mathematic operations I was able to iterate over the map. Please have a look into the code below:

https://github.com/naikajah/AWS-Infrasetup/tree/master/subnet.tf

The code looks little complex but it does the trick

[ZeroPointEnergy]

@naikajah this is amazing and very helpful. Thought about something like this before but did not even try since I was burned so many times already by Terrafrom when I tried to do some clever workaround. I'm amazed this is actually working. Thanks for sharing.

[pll]

None of this helps with taking a basic Key/Value map to be used for tagging and applying it to something like an ASG, which uses a completely different tagging API.

It would be nice if there were a way to take a basic map iterate over that to create a decent map that could used for ASGs.

[shroo-paulli]

There is a simpler solution: Terraform in JavaScript (or Python or Ruby). The concept is simple, instead of enhancing DSL to provide basic programming constructs (if, else, for, while, etc.), we can put DSL in JavaScript. In the web ecosystem, we have seen a few good use cases of that: React JSX  -  HTML in JavaScript, styled components  - CSS in JavaScript.

Please refer to my article to see an example usage: Add loops, conditions and logic in Terraform or CloudFormation using JavaScript

[pll]

It appears that all you're doing is templating the DSL itself. Is there something special about JS that makes JS a better choice than python and jinja2 or ruby and erb as language choice?

I think I would not want to go that direction in general. The point of a DSL is to remove the overhead of a programming language and to be able to concisely declare the desired end-state. By tightly coupling and intertwining the DSL and a programming template layer, you obscure what that desired end-state is supposed to be. And, perhaps, if your infrastructure is so complex that there is no way around doing things this way, that might be a sign that your design is too complex to begin with and should be broken down into smaller, simpler components that can be declaratively stated using the DSL.

[shroo-paulli]

@pll No Paul, I am not saying it is better than jinja2 or ruby, just another alternative, but I do think it is better than the terraform -> count tricks (when it is complicated).

I have also demonstrated an advanced use case that you can never achieve using just the terraform DSL: find the ARN of an AWS Lambda function (provisioned by the serverless framework) then use it in the PreSignUp trigger of AWS Cognito (provisioned by terraform).

In addition, JavaScript will generate an output file that is your final terraform file that you can check-in to the source control. It will be easier to read than following (from the original issue comment):

resource "aws_sns_topic" "test" {
    name = "${element(var.foo, count.index / length(var.bar))}-${element(var.bar, count.index)}"
    count = "${length(var.foo) * length(var.bar)}"
}

[pll]

@shroo-paulli Hi Paul, okay, I wasn't sure if you were advocating for JS because of something inherent to JS when it comes to templating that makes it a better choice. I don't know JS, so I wasn't sure is there was something I was missing.

I'm not sure I fully understand your use-case wrt to finding the ARN. Are you saying the Lambda was created/generated by another Lambda and is not created by terraform to begin with? If so, I have not had to deal with that scenario yet, so I can see the difficulty there.

I do agree, there are many things terraform is not suited for. And in those cases, falling back to the AWS API in a real language is often the only choice. I try very hard to keep those things to a minimum, but they are inevitable!

[engineforce]

@pll Yes I am using serverless framework to provision AWS Lambda (because it is specialized in serverless deployment and development). Never mind, it is just and example to show that we can leverage the template layer (JavaScript or Python or Ruby) on top of terraform DSL when dealing with complex thing, instead of fallback all the way to AWS API.

[apparentlymart]

Hi all! I just spotted this issue again after losing track of it due to it using an outdated labeling scheme. In the mean time, the same need was described in #17179 which contains some discussion about how we're planning to address this.

Although this issue is older, because #17179 contains more recent discussion I'm going to close this one just to consolidate the discussion over there.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.