Terraform: Splat syntax + count + nested map attributes

[genevieve]

Related issue & conversation

https://github.com/terraform-providers/terraform-provider-google/issues/912

We opened this issue in the google provider, but @danawillow recommended we open it here as this is more of a core issue.

Here is the content of the linked issue -

Terraform Version

Terraform v0.11.1

• provider.google v0.1.3
• provider.random v1.0.0
• provider.tls v1.0.0

Affected Resource(s)

• google_sql_database_instance

Terraform Configuration Files

resource "google_sql_database_instance" "master" {
  region           = "${var.region}"
  database_version = "MYSQL_5_6"
  name             = "${var.env_name}-${replace(lower(random_id.db-name.b64), "_", "-")}"

  settings {
    tier = "${var.sql_db_tier}"

    ip_configuration = {
      ipv4_enabled = true

      authorized_networks = [
        {
          name  = "all"
          value = "0.0.0.0/0"
        },
      ]
    }
  }

  count = "${var.count}"
}

output "sql_db_ip" {
  value = "${google_sql_database_instance.master.*.ip_address.0.ip_address}"
}

Debug Output

* module.external_database.output.sql_db_ip: Resource 'google_sql_database_instance.master' does not have attribute 'ip_address.0.ip_address' for variable 'google_sql_database_instance.master.*.ip_address.0.ip_address'

Expected & Actual Behavior

If I were to change the output value to:

output "sql_db_ip" {
  value = "${google_sql_database_instance.master.*.ip_address}"
}

I get:

sql_db_ip = [
    [
        map[ip_address:35.184.66.76 time_to_retire:]
    ]
]

I expect that in the former version of the output syntax (as is specified in the docs), it would print 35.184.66.76 and not an error that the attribute is not available. I additionally expect the output not to cause an error when the instance was not created.

It seems that trying to use the splat syntax and asking for nested attributes (conditionally) is not possible in terraform. If there is a single ip address assigned, why does the attribute have to be gathered by indexing with ip_address.0.ip_address? Could the instance have a single, non-nested attribute for returning the address?

Other Attempts

• Using the element syntax is not possible as master.*.ip_address will return a nested map and not a flat list. We can flatten this list with flatten(google_sql_database_instance.master.*.ip_address) but this is a list of maps and so we cannot select the first map with element(). We can't element(concat(flatten(...), list(SOMETHING))) because you need to concat the flatted list with another list of maps. list() only takes strings.

• We cannot do conditionals because they are not lazily evaluated and for the case where the instance is not created (count == 0), it still tries to get attributes.

  value = "${length(google_sql_database_instance.master.*.ip_address) > 0 ? lookup(google_sql_database_instance.master.*.ip_address[0], "ip_address", "") : ""}"

Steps to Reproduce

1. terraform apply

[charandas]

Running into this with data.external. The external usage enforces a result map.

Using splat syntax like this fails "element(data.external.client_vm_moid.*.result.value, count.index)" fails though.

Resource 'data.external.client_vm_moid' does not have attribute 'result.value' for variable 'data.external.client_vm_moid.*.result.value'

~Would pass if the depends_on vms were already created, so clearly failing because of non-lazy nature.~ (EDIT: I was wrong, it cannot splat at the nested level, period).

[charandas]

Wanted to pass along a hidden gem for those who run into this.

The combination of lookup with [] (another sugar over element) seems to get me the nested value in the map. Would be nice to just be able to get it from the resource vsphere_virtual_machine though.

"moid": "${lookup(data.external.client_vm_moid.*.result[count.index], \"value\")}", // works
"moid": "${data.external.client_vm_moid.*.result[count.index].value", // does not work, the splat feature 2.0 ;)

[apparentlymart]

Hi @genevieve! Sorry for this strange behavior, and for the delay in responding here.

This is indeed some strange behavior. My current theory for what's going on here is that because the top-level ip_address attribute on this resource is "computed" there isn't an element zero yet at the point when Terraform is planning this operation, and so ip_address.0 fails to find a result.

The correct behavior here would be for Terraform to see that ip_address is <computed> and therefore treat any attributes of that as computed too, but the current expression interpreter handles the "splat syntax" as a special case and this is probably bypassing the usual rule that any operation on a computed value always produces a computed value itself.

I believe this will be fixed once we integrate the new version of the configuration language interpreter we're working on, since in the new interpreter the splat syntax is no longer a special case but is instead a proper operator in the language that is subject to all of the usual handling of computed values, though the new interpreter may require the expression to be written a slightly different way for correct results:

# NOT YET IMPLEMENTED and may change slightly before release
output "sql_db_ip" {
  value = google_sql_database_instance.master[*].ip_address[0].ip_address
}

The [*] here is one of the new _splat operators_ in the improved language interpreter, and specifically the _full_ splat operator that allows the use of the indexing syntax so that [0] can work.

The current focus of the Terraform Core team at HashiCorp is on integrating this new language interpreter and its associated parser. This is a complex project due to the scope of the changes, so we're currently planning how best to implement it and roll it out.

In the mean time, unfortunately I'm not sure if there's a way to express what you want to express here in the current interpolation language. To work around the current limitations of the language, in principle the Google Cloud provider could export a simple attribute like first_ip_address that provides a convenience syntax for the presumably-common case of accessing the IP address in the first element of that list, which we could discuss with the team working on terraform-provider-google to see if they'd be interested in supporting such a thing.

---

I see in terraform-providers/terraform-provider-google#912 you mentioned that this had worked prior to Terraform 0.11 due to the fact that output errors were silently ignored. This is an interesting extra detail and I expect this was working because the error is happening during the _plan_ phase, when the list isn't yet populated, but then working in the subsequent _apply_ phase because the list is then ready.

In that case, for this specific scenario it may work to opt out of the output error checking behavior using the TF_WARN_OUTPUT_ERRORS=1 environment variable, which was added in v0.11.1 to allow the new stricter error checking to be disabled. This environment variable will be supported at least until the new configuration language interpreter is fully integrated, at which point the workaround should no longer be needed.

[genevieve]

Thank you for the thorough response and the work around!

[genevieve]

I will add your suggestion for first_ip_address to the issue I opened in the google provider and see if they are interested.

[stevehorsfield]

I'm hitting the same with a conditional creation of aws_acm_certificate and accessing the domain validation attributes.

[harmy]

same here @stevehorsfield

[apparentlymart]

The new splat expression syntax I described above is now described in one of our "what's coming in v0.12" articles.

[jdimatteo]

As far as I can tell, none of the work arounds work when count is zero (I need to reference this variable both when count is zero and when the count is 1).

In my case this was for aws_eks_cluster certificate_authority.0.data, e.g. as in the example "${aws_eks_cluster.example.certificate_authority.0.data}" at https://www.terraform.io/docs/providers/aws/r/eks_cluster.html#example-usage .

I tried ${join("", lookup(aws_eks_cluster.main.*.certificate_authority[0], "data"))} and that failed when count was zero with error list "aws_eks_cluster.main.*.certificate_authority" does not have any elements so cannot determine type.

I tried TF_WARN_OUTPUT_ERRORS=1 and that didn't help.

I eventually gave up and worked around the problem in an absurd way by having two versions of the terraform module, one when the count was zero and one when the count was 1, and wrapping the terraform command in a script that put the necessary version of the module in place when needed.

[hashibot]

Hello! :robot:

This issue relates to an older version of Terraform that is no longer in active development, and because the area of Terraform it relates to has changed significantly since the issue was opened we suspect that the issue is either fixed or that the circumstances around it have changed enough that we'd need an updated issue report in order to reproduce and address it.

If you're still seeing this or a similar issue in the latest version of Terraform, please do feel free to open a new bug report! Please be sure to include all of the information requested in the template, even if it might seem redundant with the information already shared in _this_ issue, because the internal details relating to this problem are likely to be different in the current version of Terraform.

Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.