Terraform: v0.12 - Plan errors due to datasource being evaluated too early

[amarruedo]

Terraform Version

Terraform v0.12.3
+ provider.gitlab v2.2.0
+ provider.local v1.2.2
+ provider.vault v2.0.0

Terraform Configuration Files

The configuration creates a Vault approle and then a Gitlab project CI/CD variable, which is then filled with the role_id vaule of the previouslly created approle.

terraform {
  required_version = ">= 0.12.0"
}

provider "vault" {
  version = "2.0.0"
  max_lease_ttl_seconds = "3000"
}

provider "gitlab" {
    token = "mytoken"
    base_url = "https://mygitlab.com"
}

resource "vault_approle_auth_backend_role" "project_approle" {
  count = length(var.projects)
  role_name = replace(var.projects[count.index].name,"/","_")
  policies  = [
    for policy in var.projects[count.index].policies:
    replace(policy,"/","_")
  ]
  secret_id_num_uses = 1
  token_ttl = 100
  token_max_ttl = 100
}

data "vault_approle_auth_backend_role_id" "project_roles" {
  count = length(var.projects)
  role_name = replace(var.projects[count.index].name,"/","_")
  depends_on = [
      vault_approle_auth_backend_role.project_approle
    ]
}

resource "gitlab_project_variable" "project_role_id_variables" {
    count = length(var.projects)
    project = var.projects[count.index].id
    key = "ROLE_ID"
    value = data.vault_approle_auth_backend_role_id.project_roles[count.index].role_id
    protected = false
}

where projects:

projects = [
    {
        id = "1"
        name= "projectA"
        policies = [
            "secret/read-secretA",
            "secret/read-secretB"
        ]
    },
    {
        id = "2"
        name= "projectB"
        policies = [
            "secret/read-secretA"
            ]
    }
]

Debug Output

error.log

Expected Behavior

This used to work with Terraform v0.11.*, terraform plan succeeded with no problem.

Actual Behavior

terraform plan fails when trying to access a datasource that will get populated on terraform apply

Steps to Reproduce

1. terraform init
2. terraform apply

Additional Context

Modifying the configuration to the following example (creating just one dummy approle and using it's role_id), works as expected:

....

resource "vault_approle_auth_backend_role" "project_approle" {
  role_name = "test-role"
  policies  = [
    "default"
  ]
  secret_id_num_uses = 1
  token_ttl = 100
  token_max_ttl = 100
}

data "vault_approle_auth_backend_role_id" "project_roles" {
  role_name = "test-role"
  depends_on = [
      vault_approle_auth_backend_role.project_approle
    ]
}

resource "gitlab_project_variable" "project_role_id_variables" {
    count = length(var.projects)
    project = var.projects[count.index].id
    key = "TEST_ROLE"
    value = data.vault_approle_auth_backend_role_id.project_roles.role_id
    protected = false
}

....

success.log

so I think the issue only affects to datasources of type list.

References

• #20200

[surajmuthreja]

Please update once fix is available for Azure Provider as well.

[vorezal]

I've also been having this same issue with the Azure provider. It does seem related to lists of data sources and possibly also has something to do with the contents of the state file.

Terraform Version

Terraform v0.12.3
+provider.azurerm v1.31.0

Terraform Configuration Files

Here is a pretty minimal set of configurations to demonstrate.

variable "location" {
  default     = "East US 2"
}

variable "pubip_count" {
  default = 1
}

resource "azurerm_resource_group" "rg" {
  name     = "test_resource_group"
  location = var.location
}

resource "azurerm_public_ip" "test" {
  name                = "pubip-${count.index}"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method   = "Dynamic"
  domain_name_label   = "something-${count.index}"
  count               = var.pubip_count
}

data "azurerm_public_ip" "test" {
  name                = azurerm_public_ip.test[count.index].name
  resource_group_name = azurerm_resource_group.rg.name
  count               = var.pubip_count
}

resource "azurerm_redis_firewall_rule" "test" {
  name                = "Rule_${count.index}"
  redis_cache_name    = "redis-cache-test"
  resource_group_name = "redis-rg-test"
  start_ip            = data.azurerm_public_ip.test[count.index].ip_address
  end_ip              = data.azurerm_public_ip.test[count.index].ip_address
  count               = var.pubip_count
}

Expected Behavior

Same as original poster.

Actual Behavior

I did a lot of testing so my case goes a little all over the place. Sorry this may get long. Running terraform apply initially works perfectly fine, but if I run a terraform destroy and then a terraform apply again it fails with the following error:

data.azurerm_public_ip.test[0]: Refreshing state...

Error: Error: Public IP "pubip-0" (Resource Group "test_resource_group") was not found

  on main.tf line 23, in data "azurerm_public_ip" "test":
  23: data "azurerm_public_ip" "test" {

I am able to get around this by adding depends_on = [azurerm_resource_group.rg] to the data "azurerm_public_ip" "test" resource, but then I get the following instead:

Error: Invalid index

  on main.tf line 34, in resource "azurerm_redis_firewall_rule" "test":
  34:   start_ip            = data.azurerm_public_ip.test[count.index].ip_address
    |----------------
    | count.index is 0
    | data.azurerm_public_ip.test is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on main.tf line 35, in resource "azurerm_redis_firewall_rule" "test":
  35:   end_ip              = data.azurerm_public_ip.test[count.index].ip_address
    |----------------
    | count.index is 0
    | data.azurerm_public_ip.test is empty tuple

The given key does not identify an element in this collection value.

Now if I switch count = var.pubip_count out for count = length(data.azurerm_public_ip.test) on resource "azurerm_redis_firewall_rule" "test" instead, everything is happy. I can also get the same errors if I just run terraform refresh after the init and before the apply, so perhaps the state file missing information is part of the problem too.

Steps to Reproduce

1. terraform init
2. terraform apply
3. terraform destroy
4. terraform apply

or

1. terraform init
2. terraform refresh
3. terraform apply

[mikeh-kore]

I was having a similar problem... Invalid Index... Empty Tuple...

I found that I could avoid the error by deleting all my remote state files in S3 prior to planning. This is still a bug that needs fixed but at least there's a work around.

[amarruedo]

@mikeh-kore in my case there is no workaround, the tests I've done do not have remote state, so maybe you where experiencing a different issue.

[surajmuthreja]

@rkst -- Can you please confirm if this has been fixed for Azure Provider?

[diekmann]

Cross-referencing: This bug may be a duplicate of https://github.com/terraform-providers/terraform-provider-aws/issues/9733, or the other way round. GCP and AWS seem affected, I don't know about Azure. Most likely, this is a bug in terraform-core.

The temporary workaround we found in https://github.com/terraform-providers/terraform-provider-aws/issues/9733#issuecomment-520882423 seems to work for others, too. TL;DR: force the evaluation of the empty tuple resource in the count field, run terraform refresh.

[OJFord]

so I think the issue only affects to datasources of type list.

Just to add, though it's probably not surprising, this is also an issue with for_each data sources.

Whereby accessing the result of:

data "external" "datasource" {
  for_each = some_map

  depends_on = [
    some_resource,
  ]

  # ...
}

in a local variable or output will try to evaluate it at plan-time, so data.external.datasource["mapkey"] results in an error that it is an object of zero attributes, and has no "mapkey".

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.