Terraform: Latest Terraform version cannot handle count indexes

[taylorturner]

Terraform Version

❯ terraform --version
Terraform v0.12.28
+ provider.aws v2.70.0
+ provider.helm v1.0.0
+ provider.kubernetes v1.11.3
+ provider.local v1.4.0
+ provider.null v2.1.2
+ provider.random v2.3.0
+ provider.template v2.1.2

Terraform Configuration Files

resource "aws_vpc" "this" {
  count = var.create_vpc ? 1 : 0

  cidr_block = var.cidr_block
  enable_dns_hostnames = true

  tags = merge(
    var.custom_tags,
    {
      Name                                        = "${var.name}-vpc"
      "kubernetes.io/cluster/${var.name}" = "shared"
    }
  )
}

output "vpc_id" {
  description = "The ID of the VPC"
  value = aws_vpc.this[0].id
}

Expected Behavior

Terraform accepted the index.

Actual Behavior

Error: Invalid index

  on .terraform/modules/vpc/outputs.tf line 13, in output "vpc_id":
  13:   value = aws_vpc.this[0].id
    |----------------
    | aws_vpc.this is empty tuple

The given key does not identify an element in this collection value.

Steps to Reproduce

terraform init

terraform apply

Additional Context

N/A

References

• https://github.com/terraform-providers/terraform-provider-azurerm/issues/5675

[apparentlymart]

Hi @taylorturner!

From reviewing your configuration, it seems like if var.create_vpc were false there would be no elements of aws_vpc.this and so the result you saw here would be correct. If you wish to return an attribute from a conditional resource instance in your output then you'll need to write a conditional expression that tells Terraform which value to return when no VPC was created and therefore there's no valid id to return. For example:

output "vpc_id" {
  description = "The ID of the VPC, if create_vpc is true."
  value = var.create_vpc ? aws_vpc.this[0].id : null
}

In the above, the vpc_id output will be null if var.create_vpc is false.

If you're seeing this error in situations where var.create_vpc is _true_ then that does not match Terraform's intended behavior.

Can you confirm whether you saw this error in a situation where var.create_vpc was false, and if so whether a conditional expression like in my example above addresses the problem? Thanks!

[taylorturner]

Sorry @apparentlymart, I should've included more information about that. In the modules directory, I set create_vpc = true as the default and it wasn't modified beyond that. Upon downgrading to version 0.12.10 as suggested in the other issue, my problem was resolved.

[i7c]

We have a similar problem with a data source

data "azurerm_key_vault_secret" "sp" {         
  count = var.aks_sp_name == null ? 0 : 1      

  name         = var.aks_sp_secret_name        
  key_vault_id = data.azurerm_key_vault.aks.id 
}                                              

I have intercepted the HTTP traffic of a terraform destroy run, as suggested here https://github.com/hashicorp/terraform-plugin-sdk/issues/88

And I have noticed that there are repeated calls to the keyvault like so
https://XXXXXXXXX.vault.azure.net/secrets/XXXXXXXXXX/?api-version=2016-10-01 that return 401 and terraform does not seem to provide an authorization header on those calls. I'm not sure if this is related to the bug here, it just seemed very odd to attempt a GET on the keyvault without token.

[Clete2]

Same issue here, but with a set instead.

Error: Invalid index

  on terraform/eventing.event_controller.tf line 12, in module "event_controller_role":
  12:     aws_iam_policy.sqs_receive["event-controller-queue"].arn,
    |----------------
    | aws_iam_policy.sqs_receive is object with no attributes

The given key does not identify an element in this collection value.

Due to company policy, I cannot attach my actual terraform code. I can tell you that we are using for_each with a set and this is the same issue.

I was able to workaround it by doing the following:

1. Modify terraform state file manually and change the version from 0.12.28 to 0.12.10
2. Download 0.12.10 and run terraform destroy on the workspace
3. :)

[BrandonALXEllisSS]

I'm having the same error as the OP (however the output is the conditional output described by @apparentlymart), however, I noticed something peculiar.

Normally, when I run a "terraform destroy", I'll get a state file that looks like this:
```{
"version": 4,
"terraform_version": "0.12.26",
"serial": 39,
"lineage": "839fdcfb-d29b-b320-8499-fab339acb85c",
"outputs": {},
"resources": []
}

This looks normal. Everything has been deleted, so there's no resources managed

However, some operation somewhere made the state file look like this:
```{
  "version": 4,
  "terraform_version": "0.12.26",
  "serial": 2153,
  "lineage": "5628ff8b-f32d-4bf7-3c7d-e73543592a78",
  "outputs": {},
  "resources": [
    {
      "module": "module.CognitoUserPool",
      "mode": "managed",
      "type": "aws_cognito_user_pool",
      "name": "pool",
      "each": "list",
      "provider": "provider.aws",
      "instances": []
    }
  ]
}

(note: even though this is version 12.26, the same thing happens on version 12.28)
All the resources with a "count" value set still remain (In my case, it's the aws_cognito_user_pool from the CognitoUserPool module). However, there are no instances. To terraform, this is still an "empty state" (if you run "terraform state list" you get nothing).

HOWEVER, when running a "terraform destroy" with the latter state file, it produces the exact same error mentioned by the OP. Doing a "terraform destroy" on the former produces no error.

Why does this happen? What caused the state file to look all weird? Idk.

[danieldreier]

There are a few different issues that have been raised in this thread, so I'd like to re-focus on what we actually need to work on this productively, which is a reproduction case. In the original case, when var.create_vpc is false, the count of "aws_vpc" "this" resources is set to zero, and so aws_vpc.this[0].id fails because there are no this aws_vpc resources.

I made a simplified version of this reproduction case at https://github.com/danieldreier/terraform-issue-reproductions/blob/master/25578/repro.tf to show what I'm running, which does not reproduce the problem as described on v0.13.0RC1. I believe you that you're having a real problem, and I'd like to help, but the next thing we need is a clear, runnable reproduction case in order to help on this. If you can make a PR against that reproduction case that shows it, I'd appreciate the help.

[A30004028]

Can anyone confirm if this issue is present in Terraform 0.13?

Thanks in advance

[Speculor]

We are hitting this as well on Terraform 0.12.24

data "azurerm_virtual_machine" "disk" {
  for_each             = { for drives in var.vm_drives : drives.drives_vm_name => drives }
  name                 = each.value.vm_name 
  resource_group_name  = data.azurerm_resource_group.disk.name
}

resource "azurerm_managed_disk" "disk" {
  for_each             = { for drives in var.vm_drives : drives.drives_vm_name => drives }
  name                 = each.value.drives_vm_name 
  location             = data.azurerm_resource_group.disk.location
  resource_group_name  = data.azurerm_resource_group.disk.name
  disk_size_gb         = each.value.disk_size_gb
  storage_account_type = each.value.storage_account_type
  create_option        = "Empty"
  tags                 = var.tags
}

resource "azurerm_virtual_machine_data_disk_attachment" "disk" {
  count              = length(azurerm_managed_disk.disk)
  managed_disk_id    = azurerm_managed_disk.disk[count.index].id
  virtual_machine_id = data.azurerm_virtual_machine.disk[count.index].id
  lun                = "10"
  caching            = "ReadWrite"
}


Error: Invalid index



  on .terraform\modules\vm_adc_create_disks_ad_logs_db_dev\main.tf line 37, in resource "azurerm_virtual_machine_data_disk_attachment" "disk":
  37:   managed_disk_id    = azurerm_managed_disk.disk[count.index].id
    |----------------
    | azurerm_managed_disk.disk is object with 4 attributes
    | count.index is 2



The given key does not identify an element in this collection value.




Error: Invalid index



  on .terraform\modules\vm_adc_create_disks_ad_logs_db_dev\main.tf line 38, in resource "azurerm_virtual_machine_data_disk_attachment" "disk":
  38:   virtual_machine_id = data.azurerm_virtual_machine.disk[count.index].id
    |----------------
    | count.index is 0
    | data.azurerm_virtual_machine.disk is object with 4 attributes



The given key does not identify an element in this collection value.

[TheTaka96]

@A30004028 jep, using Terraform 1.13.2 and opentelekomprovider 1.19.2

using the following code:

```#....
resource "opentelekomcloud_vpc_subnet_v1" "subnet_env" {
count = length(local.subnets)
vpc_id = opentelekomcloud_vpc_v1.vpc_env.id

name = "${local.subnets.*.name[count.index]}"
#...
}

producing the same error

```Error: Invalid index

  on modules/networking/main.tf line 41, in resource "opentelekomcloud_vpc_subnet_v1" "subnet_env":
  41:   name   = "${local.subnets.*.name[count.index]}"
    |----------------
    | count.index is 6
    | local.subnets is object with 8 attributes

The given key does not identify an element in this collection value.

[ivankovnatsky]

@TheTaka96 terraform v0.13.2.

[ivankovnatsky]

getting this with v0.13.3:

didn't change the code structure at all since 0.13 upgrade.

added new item to the list which use in for_each loop, somehow getting this error:

masked some resource names for privacy.

Error: Invalid index

  on ../../../modules/sns-sqs/xxx.tf line 28, in resource "aws_sqs_queue_policy" "xxx_deadletter":
  28:   policy    = data.aws_iam_policy_document.xxx_deadletter[each.key].json
    |----------------
    | data.aws_iam_policy_document.xxx_deadletter is object with 2 attributes
    | each.key is "order_xxx"

The given key does not identify an element in this collection value.


Error: Invalid index

  on ../../../modules/sns-sqs/xxx.tf line 58, in resource "aws_sqs_queue_policy" "xxx":
  58:   policy    = data.aws_iam_policy_document.xxx[each.key].json
    |----------------
    | data.aws_iam_policy_document.xxx is object with 2 attributes
    | each.key is "order_xxx"

The given key does not identify an element in this collection value.

i suspect, that whereas in resource definitions it works as expected in data definitions it might seem not, though i'm not a big in-deep terraform core expert at all.

[IgorOrmus]

Running in to the same thing with
Terraform v0.12.20

• provider.azurerm v2.26.0

Terraform code :

resource` "azurerm_data_factory" "this" {

  count = var.enableADF
  name                = "${var.ADFname}-ADF"
  location            = var.rg_location
  resource_group_name = var.rg_name
  tags                = var.tags

  identity {

  type = "SystemAssigned"
}

dynamic "vsts_configuration" {

  for_each = var.vsts_configuration
    content {
      account_name = vsts_configuration.value["account_name"]
      branch_name = vsts_configuration.value["branch_name"]
      project_name = vsts_configuration.value["project_name"]
      repository_name = vsts_configuration.value["repository_name"]
      root_folder = vsts_configuration.value["root_folder"]
      tenant_id = vsts_configuration.value["tenant_id"]
    }
}
}
 output "tenant_id" {
  value = "${azurerm_data_factory.this[0].identity[0].tenant_id}"
 }

 output "principal_id" {
   value = "${azurerm_data_factory.this[0].identity[0].principal_id}"
 }

Error:

Error: Invalid index

  on modules/az-data-factory/main.tf line 28, in output "tenant_id":
  28:   value = "${azurerm_data_factory.this[0].identity[0].tenant_id}"
    |----------------
    | azurerm_data_factory.this is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on modules/az-data-factory/main.tf line 32, in output "principal_id":
  32:   value = "${azurerm_data_factory.this[0].identity[0].principal_id}"
    |----------------
    | azurerm_data_factory.this is empty tuple

The given key does not identify an element in this collection value.

[eherot]

Also seeing something similar to what @IgorOrmus is seeing. This is the minimally viable code I was able to get to reproduce the issue:

terraform {
  required_version = ">= 0.12"
}

provider "local" {}

locals {
  queues = {
    "existing_queue" = {}
    "new_queue" = {}
  }
}

resource "aws_sqs_queue" "primary" {
  for_each = local.queues

  name = each.key
}

resource "aws_iam_policy" "consumer" {
  for_each = local.queues

  name = "${each.key}-consume"
  policy = data.aws_iam_policy_document.consumer[each.key].json
}

data "aws_iam_policy_document" "consumer" {
  for_each = local.queues

  statement {
    actions = []
    resources = []
  }
}

Output basically looks like this (but is very hard to reproduce because seemingly irrelevant changes to the state cause it to stop happening):

Error: Invalid index

  on main.tf line 20, in resource "aws_iam_policy" "consumer":
   5:   policy = data.aws_iam_policy_document.consumer[each.key].json
    |----------------
    | data.aws_iam_policy_document.consumer is object with 1 attribute
    | each.key is "new_queue"

The given key does not identify an element in this collection value.

At any rate the salient point seems to be that for_each is also affected by this problem.

[bbros-dev]

@eherot we tried running your repro before submitting a PR to @danieldreier as requested.

Running your code:

Error: Error creating IAM policy existing_queue-consume: MalformedPolicyDocument: Policy statement must contain actions.                                                        status code: 400, request id: 3d101c86-bac1-44e6-b692-6b4289692d45


Error: Error creating IAM policy new_queue-consume: MalformedPolicyDocument: Policy statement must contain actions.
        status code: 400, request id: 22dea3f2-171f-496a-8aab-3b083dadfb05

[derekschauland]

This is impacting SQL identity configurations as well - if I use azurerm_sql_server.name.identity.0.principal_id it tells me I should use [0] - when I change to that, it is an empty resource... so there is no way this builds. Not sure where this falls in the roadmap, but currently, the only work around for me has been to rebuild the entire infra

[dev-head]

@eherot has a good example to highlight this possible issue. i'm seeing the same behavior just introduce itself into one of our modules that has been working great for a while now. tracking down the behavior was challenging enough, now i need to find a viable workaround before terraform is passed over for the console. no pressure though :D

thanks for looking into it.

[eherot]

@bbros-dev Unfortunately I have also not been able to completely reliably reproduce this issue even using the above code. It seems to depend a lot on the existing state having been created by a previous version. For example, I was able to work around the issue in a few cases by creating the policy manually and importing it. In fact I even tried starting with a non-working plan, blowing away the entire state, and then re-importing every existing resource, and doing so seemed to fix the issue. I haven't tried yet to see if the problem persists when adding _additional_ policies after doing this but I also haven't seen new complaints from devs so I suspect it may have worked.

TL;DR: This may be an upgrade-specific issue having to do with state created by a previous version of TF.

[taylorturner]

I'm still seeing this issue on version 0.14.2 and this is on a completely fresh state.

module1 snippet:

resource "azurerm_resource_group" "this" {
  count    = var.create_aks ? 1 : 0
  name     = var.resource_group_name
  location = var.location
}

output "resource_group" {
  value = azurerm_resource_group.this[0].name
}

Error when trying to use this module:

Error: Invalid index

  on .terraform/modules/aks/outputs.tf line 38, in output "resource_group":
  38:   value = azurerm_resource_group.this[0].name
    |----------------
    | azurerm_resource_group.this is empty tuple

The given key does not identify an element in this collection value.

The workaround for this is to create a data source, then reference it instead.

resource "azurerm_resource_group" "this" {
  count    = var.create_aks ? 1 : 0
  name     = var.resource_group_name
  location = var.location
}

data "azurerm_resource_group" "this" {
  name       = var.resource_group_name
  depends_on = [azurerm_resource_group.this]
}

output "resource_group" {
  value = data.azurerm_resource_group.this.name
}

[danieldreier]

@eherot I apologize for the slow response here. I just tried your reproduction code with 0.14.2, and ran into an AWS error:

Error: Error creating IAM policy existing_queue-consume: MalformedPolicyDocument: Policy statement must contain actions.
        status code: 400, request id: eb979e99-5415-493a-a63f-8b6ea76ef993



Error: Error creating IAM policy new_queue-consume: MalformedPolicyDocument: Policy statement must contain actions.
        status code: 400, request id: e2a29a3f-89f7-4371-a1d5-358419723f86

I haven't spent much time with SQS queues, so I'm not immediately sure how to troubleshoot that. I'm happy to re-run this and try to reproduce it if you can contribute an updated version.

To other folks here: I took a quick read through this, and I believe this is still blocked on needing a clear, locally-runnable reproduction case. I understand that a lot of folks are running into similar errors, but I don't know what condition is triggering that state, and simply saying that it's happening to you too doesn't help us figure out how to trigger this issue locally. If this is causing you problems, and you want to help get this fixed, the best thing to do is to contribute a reproduction case that we can copy-paste and run locally on a developer workstation. The ideal would be a case using null_resource because that doesn't require external dependencies, but we can work with the major cloud providers if necessary.

[taylorturner]

The bug appears when the following scenarios are true:

• Terraform module with resources using count arg
• Terraform module referencing resources by index as an output
• Deployment calls that module

I'm going to attempt to provide you with enough to get started testing. Assume the file tree looks like this:
-- terraform
---- module
---- deployment

module/main.tf

resource "azurerm_resource_group" "this" {
  count    = var.create_aks ? 1 : 0
  name     = var.resource_group_name
  location = var.location
}

locals {
  public_ip_name = var.public_ip_name == "" ? var.cluster_name : var.public_ip_name
}

resource "azurerm_public_ip" "this" {
  count               = var.create_aks ? 1 : 0
  name                = local.public_ip_name
  resource_group_name = azurerm_resource_group.this[0].name
  location            = azurerm_resource_group.this[0].location
  allocation_method   = "Static"
  ip_version          = "IPv4"
  sku                 = "Standard"
}

output "cluster_public_ip" {
  value = azurerm_public_ip.this[0].ip_address
}

output "resource_group" {
  value = azurerm_resource_group.this[0].name
}

module/variables.tf

variable "client_id" {}
variable "client_secret" {}
variable "cluster_name" {}
variable "resource_group_name" {}
variable "cluster_dns_name" {}
variable "kubernetes_version" {}

variable "public_ip_name" {
  default = ""
}
variable "location" {
  default = "East US"
}

deployment/main.tf

locals {
  cluster_name         = "example-001"
  resource_group_name  = "example_001"
  cluster_dns_name     = "example"
  storage_account_name = "example001"
  public_ip_name       = "example-001-ingress-ip"
  location             = "East US"
}

variable "client_id" {}
variable "client_secret" {}

terraform {
  required_version = ">= 0.14"
}

provider "azurerm" {
  features {}
  subscription_id = "9b9a8f8b-8217-407b-9007-4ef403ab3216"
}

provider "random" {}

provider "local" {}

module "aks" {
  source = "../module"

  create_aks     = true # this should tear down all resources if set to false

  client_id     = var.client_id
  client_secret = var.client_secret

  cluster_name        = local.cluster_name
  resource_group_name = local.resource_group_name
  cluster_dns_name    = local.cluster_dns_name
  public_ip_name      = local.public_ip_name
  location            = local.location
  kubernetes_version  = "1.18.10"
}

Execute terraform plan in the deployments dir and you should see:

Error: Invalid index

  on .terraform/modules/aks/outputs.tf line 38, in output "resource_group":
  38:   value = azurerm_resource_group.this[0].name
    |----------------
    | azurerm_resource_group.this is empty tuple

The given key does not identify an element in this collection value.

I tried modifying the output and omitted the [0] index reference. The error then changed:

Error: Missing resource instance key

  on .terraform/modules/aks/outputs.tf line 38, in output "resource_group":
  38:   value = azurerm_resource_group.this.name

Because azurerm_resource_group.this has "count" set, its attributes must be
accessed on specific instances.

For example, to correlate with indices of a referring resource, use:
    azurerm_resource_group.this[count.index]

---

The only way to get Terraform happy was to create matching data objects for anything I was trying to reference in an output object as I stated earlier.

I really hope this helps!

-Taylor

[a-mcf]

I think that this is happening when terraform is trying to index into something it hasn't created yet, like a data resource, resource, or module using for_each. I had this occur with this exact error while trying to do something similar to the way outputs are handled here, with modules created by for_each:

https://learn.hashicorp.com/tutorials/terraform/for-each
(note, I didn't use the above configuration, but something very similar)

It would return the same error everyone else is reporting when evaluating the outputs during a plan.

To workaround the problem, I pulled the list of keys from the module and not the variable.

Broken:

output "my_output" {
 value = {
    for i in keys(local.items) : i => module.my_module[i].name
  }
}

Working:

output "my_output" {
 value = {
    for i in sort(keys(module.my_module)) : i => module.my_module[i].name
  }
}

[infa-gkaur]

Facing the same issue with version Terraform v0.12.10

• provider.aws v3.18.0. Creating a new role and a trying to attach inline policy to the role.

Error: Invalid index

on role.tf line 40, in resource "aws_iam_role_policy" "test_policy":
40: role = aws_iam_role.ec2role[each.key].id
|----------------
| aws_iam_role.ec2role is object with 13 attributes
| each.key is "qa"

The given key does not identify an element in this collection value.

Error: Invalid index

on role.tf line 40, in resource "aws_iam_role_policy" "test_policy":
40: role = aws_iam_role.ec2role[each.key].id
|----------------
| aws_iam_role.ec2role is object with 13 attributes
| each.key is "dev"

The given key does not identify an element in this collection value.

[skeggse]

I suspect there's a number of factors that inform whether you run into this problem. We found that it went away after upgrading 0.13.5 → 0.14.2 with no change to the provider version.

[cocotton]

Just like @skeggse, the issue went away after upgrading from 0.13.5 -> 0.14.3

[IrmantasM]

Issue seen in 0.14.3