Terraform: Allow references to instances of the same resource

[Clete2]

Terraform Version

Terraform v0.12.18
+ provider.archive v1.3.0
+ provider.aws v2.43.0
+ provider.null v2.1.2

Terraform Configuration Files

resource "aws_sqs_queue" "queue" {
  count = length(var.queue_definitions)
  name  = "${var.name}${count.index}"

  delay_seconds              = var.queue_definitions[count.index].delaySeconds
  redrive_policy             = count.index == 0 ? null : "{\"deadLetterTargetArn\":\"${aws_sqs_queue.queue[count.index - 1].arn}\",\"maxReceiveCount\":${var.queue_definitions[count.index].maxReceiveCount}}"
  visibility_timeout_seconds = var.visibility_timeout
  kms_master_key_id          = var.kms_key_id
}

...

Module declaration:
module "sqs-backoff-sample" {
  source     = "./modules/sqs-exponential-backoff"
  name       = "clete-sqs-test-"
  kms_key_id = var.kms_key_id
  account    = var.account
  region     = var.region
  queue_definitions = [
    { maxReceiveCount : 2, delaySeconds : 0 },
    { maxReceiveCount : 4, delaySeconds : 5 },
    { maxReceiveCount : 6, delaySeconds : 15 },
    { maxReceiveCount : 8, delaySeconds : 60 }
  ]
}

Debug Output

Nothing interesting other than:
Error: Cycle: module.sqs-backoff-sample.aws_sqs_queue.queue[3], module.sqs-backoff-sample.aws_sqs_queue.queue[2], module.sqs-backoff-sample.aws_sqs_queue.queue[1], module.sqs-backoff-sample.aws_sqs_queue.queue[0]

Crash Output

Expected Behavior

I am trying to design a loop such that each queue cascades into the queue below it. The goal is to create an exponential backoff scenario where retries are slowed down as the failed messages fall into the lower queues (see queue_definitions above).

The key issue is this line:
redrive_policy = count.index == 0 ? null : "{\"deadLetterTargetArn\":\"${aws_sqs_queue.queue[count.index - 1].arn}\",\"maxReceiveCount\":${var.queue_definitions[count.index].maxReceiveCount}}"

For the redrive policy, I need to specify the previous queue's ARN such that queue 4 dumps into queue 3, 3 into 2, 2 into 1, and 1 is the final queue (count.index == 0 thus no redrive).

Actual Behavior

Terraform incorrectly identifies a cycle between 3->2->1->0, but in reality index 0 does not depend on any other queue.

Steps to Reproduce

Dynamically create multiple queues by putting the "aws_sqs_queue" into a module and then initializing it with the inputs given. Try to apply.

Additional Context

If you can think of another way to implement dynamically defined n number of queues that overflow into each other in case of failures, I am all ears.

References

[jbardin]

Hi @Clete2,

Thanks for the example config and use case.

This currently isn't supported in terraform. For various reasons dependencies are tracked at the resource level rather than individual instances, and references to resources need to first evaluate the entire resource as whole. This means that references to other instance indexes within the resource config is the same as a self reference and hence a cycle.

You're correct that there isn't a way to do this for any N resource at the moment, and the individual resources would have to be written out separately.

[Clete2]

@jbardin Thanks for your reply; I appreciate it. I'd like to see more advanced use cases supported in Terraform like conditionals being supported on all attributes and loops being more robust / less limitations. I hope this can come in the future.

[karl-cardenas-coding]

@jbardin is there something in the roadmap that will address this? Not sure if enabling depends_on for modules will help or address this.

[jbardin]

While this is something we are going to look into in the future, it's going to take some careful consideration because it's not how terraform was originally intended to operate.

[mtesch-um]

I've run into the same thing trying to setup a chain of (nearly identical) incoming SES rules where there "after" entry should refer to the previous aws_ses_receipt_rule. This is another vote to support such dependency at the instance level.

[jbardin]

After the deeper analysis done through the 0.13 development cycle, it's clear that terraform as it exists now cannot support this type of self-reference. We can keep this open for future enhancements, but this is not something that can be added to the roadmap in the foreseeable future.

[Clete2]

Thanks for the update @jbardin. For now I have abandoned attempting this sort of approach. For those looking for a quick fix on this issue, you should either utilize copy/paste to chain resources together, or look at something like CDK or Pulumi.