Terraform: Passing a data source as a list element from a calling module messes up the list

[horsey]

Terraform Version

Terraform v0.11.1
+ provider.aws v1.8.0

Terraform Configuration Files

#calling module: 
module "elb" {
  source = "./modules/elb"

  name = "${var.name}"

  subnets         = ["${var.subnets}"]
  security_groups = ["${var.security_groups}"]
  internal        = "${var.internal}"

  cross_zone_load_balancing   = "${var.cross_zone_load_balancing}"
  idle_timeout                = "${var.idle_timeout}"
  connection_draining         = "${var.connection_draining}"
  connection_draining_timeout = "${var.connection_draining_timeout}"


   listener = [                                                                                                                                                                     
       {                                                                                                                                                                            
       instance_port     = "80"                                                                                                                                                     
       instance_protocol = "HTTP"                                                                                                                                                   
       lb_port           = "80"                                                                                                                                                     
       lb_protocol       = "HTTP"                                                                                                                                                   
     },                                                                                                                                                                             
     {                                                                                                                                                                              
       instance_port     = "8443"                                                                                                                                                   
       instance_protocol = "HTTPS"                                                                                                                                                  
       lb_port           = "8443"                                                                                                                                                   
       lb_protocol       = "HTTPS"                                                                                                                                                  
       ssl_certificate_id = "${data.aws_acm_certificate.elb_cert.arn}"                                                                                                              
     },                                                                                                                                                                             
   ]                                                                                                                                                                                
  access_logs  = ["${var.access_logs}"]
  health_check = ["${var.health_check}"]

  tags = "${merge(var.tags, map("Name", format("%s", var.name)))}"
}

#callee module 
resource "aws_elb" "this" {
  name            = "${var.name}"
  subnets         = ["${var.subnets}"]
  internal        = "${var.internal}"
  security_groups = ["${var.security_groups}"]

  cross_zone_load_balancing   = "${var.cross_zone_load_balancing}"
  idle_timeout                = "${var.idle_timeout}"
  connection_draining         = "${var.connection_draining}"
  connection_draining_timeout = "${var.connection_draining_timeout}"
  listener = "${var.listener}"

  access_logs  = ["${var.access_logs}"]
  health_check = ["${var.health_check}"]

  tags = "${merge(var.tags, map("Name", format("%s", var.name)))}"
}

data "aws_acm_certificate" "elb_cert" {
  domain   = "abc.xyz.net"
  statuses = ["ISSUED"]
}

Debug Output

Error: module.elb.module.elb.aws_elb.this: "listener.0.instance_port": required field is not set
Error: module.elb.module.elb.aws_elb.this: "listener.0.instance_protocol": required field is not set
Error: module.elb.module.elb.aws_elb.this: "listener.0.lb_port": required field is not set
Error: module.elb.module.elb.aws_elb.this: "listener.0.lb_protocol": required field is not set

Crash Output

None

Expected Behavior

The receiving module should receive the listener list intact. However, the above output seems to say that the list elements are being unset.

Actual Behavior

The receiving module does not appear to be receiving the list properly.

Steps to Reproduce

1. terraform init
2. terraform apply

Additional Context

Additionally, if the list in question is not passed with the data source element, but hardcoded in the receiving module, the list is intact.

[jbardin]

Hi @horsey,

Unfortunately the configuration you have wasn't intended to work that way.

The listener field of an aws_elb isn't just a list of maps, it's a set of configuration blocks which is internally quite a different structure. Some users have discovered that applying a list of maps in some cases will get read into the configuration correctly, but that can't work in call cases.

We're working on configuration language improvements at the moment which should hopefully make the configuration more clear, and prevent things like this from "accidentally" working while providing better errors. Once we have that in place, we can work on the possibility of passing configuration blocks around somehow, as it's becoming an often requested feature in more complex configurations.

[horsey]

Hello @jbardin,
Is there are recommended way this can be handled? The ELB configuration in this scenario is a module and I need to pass the SSL certificates (that are created manually) as a parameter for different setups. How can this be achieved?

[jbardin]

Right now the best way is to pass the individual values into separate configuration blocks.
Keeping var.listener the same, that section of your config might look something like this (untested):

resource "aws_elb" "this" {
  ...
  listener = {
    instance_port     = "${lookup(var.listener[0], "instance_port")}"
    instance_protocol = "${lookup(var.listener[0], "instance_protocol")}"
    lb_port           = "${lookup(var.listener[0], "lb_port")}"
    lb_protocol       = "${lookup(var.listener[0], "lb_protocol")}"
  }

  listener = {
    instance_port      = "${lookup(var.listener[1], "instance_port")}"
    instance_protocol  = "${lookup(var.listener[1], "instance_protocol")}"
    lb_port            = "${lookup(var.listener[1], "lb_port")}"
    lb_protocol        = "${lookup(var.listener[1], "lb_protocol")}"
    ssl_certificate_id = "${lookup(var.listener[1], "ssl_certificate_id")}"
  }
}

[horsey]

@jbardin
Your suggestion is appreciated, but not sure it can be used in a module as the module would not know how many configuration blocks it will receive in advance.
I don't find an easy way to iterate either.

[jbardin]

@horsey

No there is not currently any way to iterate within the configuration.
In this case many users use external tools to generate the config on demand.

In future versions we may have the constructs to iterate over the values directly within the configuration.

[horsey]

Thanks @jbardin!

[dee-kryvenko]

After two months with terraform constantly hitting by issues like this (when something so obvious and intuitive yet for some reason doesn't work though seems like had to fit from the very first releases), I finally realized what terraform reminds me. It is this game https://www.youtube.com/watch?v=V_DtccNhLTs&t=35s which intentionally being designed to piss you off.

[dee-kryvenko]

So my 2 cents as to the possible workaround... I found using ALB resources instead of ELB, with some portion of magic, would allow to do it.

Module usage:

module "instance_with_lb" {
  # ...

  lb_listeners = [
    {
      "instance_port"     = 8080
      "instance_protocol" = "HTTP"
      "lb_port"           = 80
      "lb_protocol"       = "HTTP"
    },
    {
      "instance_port"      = 8080
      "instance_protocol"  = "HTTP"
      "lb_port"            = 443
      "lb_protocol"        = "HTTPS"
      "ssl_certificate_id" = "${aws_iam_server_certificate.fake_cert.arn}"
    },
  ]

  lb_health_checks = [
    {
      protocol            = "HTTP"
      port                = 8080
      path                = "/"
      matcher             = "200"
      healthy_threshold   = 2
      unhealthy_threshold = 2
      timeout             = 3
      interval            = 30
    },
  ]
}

Module itself:

resource "aws_lb" "lb" {
  # ...

  load_balancer_type = "application"
}

resource "aws_lb_target_group" "lb_target_group" {
  count        = "${length(var.lb_listeners)}"

  # ...

  protocol     = "${lookup(var.lb_listeners[count.index], "instance_protocol")}"
  port         = "${lookup(var.lb_listeners[count.index], "instance_port")}"
  health_check = ["${var.lb_health_checks}"]

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_lb_target_group_attachment" "lb_target_group_attachment" {
  count            = "${length(var.lb_listeners)}"
  target_group_arn = "${element(aws_lb_target_group.lb_target_group.*.arn, count.index)}"
  target_id        = "${aws_instance.instance.id}"
}

resource "aws_lb_listener" "lb_listener" {
  count             = "${length(var.lb_listeners)}"
  load_balancer_arn = "${aws_lb.lb.arn}"
  protocol          = "${lookup(var.lb_listeners[count.index], "lb_protocol")}"
  port              = "${lookup(var.lb_listeners[count.index], "lb_port")}"

  ssl_policy      = "${lookup(var.lb_listeners[count.index], "lb_protocol") == "HTTPS" ? "ELBSecurityPolicy-2016-08" : ""}"
  certificate_arn = "${lookup(var.lb_listeners[count.index], "lb_protocol") == "HTTPS" ? lookup(var.lb_listeners[count.index], "ssl_certificate_id", "") : ""}"

  default_action {
    target_group_arn = "${element(aws_lb_target_group.lb_target_group.*.arn, count.index)}"
    type             = "forward"
  }
}

[dee-kryvenko]

Actually it did worked for me without HTTPS in the list of listeners... now because my example above is using ssl_certificate_id with calculated value, hit by this https://github.com/hashicorp/terraform/issues/10857

What I just told about that game?.. exactly, I almost felt the joy of finally getting my super simple task of adding an ELB to my module (a 2-day endeavor!) done, and boooom!

In that specific case my workaround was to provide certificate_arn as a separate variable outside of the list, and it did make sense as it's more belongs to the LB and not to the particular listener. But I can easily see other cases it needs to work with calculated values...

[gaui]

This is a major blocker for us. Related: https://github.com/terraform-providers/terraform-provider-null/issues/18

[apparentlymart]

Hi all!

This issue is covering the same root problem as #7034, and a solution for this has now been merged into master for inclusion in the forthcoming v0.12.0 release.

This need was addressed by introducing a new dynamic block construct, which I discussed in more detail in my comment on the other issue.

The equivalent to that for the situation in this issue would be something like this:

  dynamic "listener" {
    for_each = var.listener
    content {
      instance_port     = each.value.instance_port
      instance_protocol = each.value.instance_protocol
      lb_port           = each.value.lb_port
      lb_protocol       = each.value.lb_protocol
    }
  }

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.