Terraform-provider-aws: aws_route53_zone resource fails when used with count

[hashibot]

_This issue was originally opened by @nick-o as hashicorp/terraform#12282. It was migrated here as part of the provider split. The original body of the issue is below._

---

Hi there,

I've run into a strange issue when trying to create multiple Route53 zones via single resource and count. It seems that the code to do so only generates a unique caller reference once (see here) which then gets used for multiple API calls.

Terraform Version

terraform -v
Terraform v0.8.7

Affected Resource(s)

• aws_route53_zone

Terraform Configuration Files

variable "route53_zone_names" {
  type    = "list"
  default = [
    "dev",
    "tst",
    "ppd"
  ]
}

resource "aws_route53_zone" "public" {
  count    = "${length(var.route53_zone_names)}"
  name     = "${element(var.route53_zone_names,count.index)}.project.abc.com"
}

Debug Output

https://gist.github.com/nick-o/d0b5a4e7ce5ef4e0ba9d74353daa518e

Panic Output

Expected Behavior

3 Public Route53 Zones should be created

Actual Behavior

Only one zone got created. The other 2 fail to get created due to non-unique caller reference

> terraform apply
aws_route53_zone.public.0: Creating...
  comment:        "" => "Managed by Terraform"
  force_destroy:  "" => "false"
  name:           "" => "dev.project.abc.com"
  name_servers.#: "" => "<computed>"
  vpc_region:     "" => "<computed>"
  zone_id:        "" => "<computed>"
aws_route53_zone.public.2: Creating...
  comment:        "" => "Managed by Terraform"
  force_destroy:  "" => "false"
  name:           "" => "ppd.project.abc.com"
  name_servers.#: "" => "<computed>"
  vpc_region:     "" => "<computed>"
  zone_id:        "" => "<computed>"
aws_route53_zone.public.1: Creating...
  comment:        "" => "Managed by Terraform"
  force_destroy:  "" => "false"
  name:           "" => "tst.project.abc.com"
  name_servers.#: "" => "<computed>"
  vpc_region:     "" => "<computed>"
  zone_id:        "" => "<computed>"
aws_route53_zone.public.1: Still creating... (10s elapsed)
aws_route53_zone.public.1: Still creating... (20s elapsed)
aws_route53_zone.public.1: Still creating... (30s elapsed)
aws_route53_zone.public.1: Still creating... (40s elapsed)
aws_route53_zone.public.1: Creation complete
Error applying plan:

2 error(s) occurred:

* aws_route53_zone.public.2: HostedZoneAlreadyExists: A hosted zone has already been created with the specified caller reference.
        status code: 409, request id: 151ed321-fd19-11e6-b1d9-afb55cf95a01
* aws_route53_zone.public.0: HostedZoneAlreadyExists: A hosted zone has already been created with the specified caller reference.
        status code: 409, request id: 151eac19-fd19-11e6-9011-f1429215f6fa

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

N/A

References

See AWS CLI documentation for explanation of caller-reference. I think the problem stems from trying to create all 3 zones in one go and it will try to use the same caller reference.

[pawelsocha]

Is this bug still valid?
I can't reproduce the problem on 1.7.1.

[anshulpatel25]

Hi @pawelsocha
Facing the same issue in Terraform v0.11.3 with aws provider version 1.9.0. However when I degrade my version to Terraform v.10.8 with aws provider version 1.9.0, it works.

Meanwhile I can avoid this, if I pass parallelism=1 in the terraform apply for version v0.11.3 and aws provider version 1.9.0

[bnm22]

I am running into the same issue on Terraform 0.11.7 with aws provider version 1.14.1. Adding the parallelism=1 flag to apply did help me get pass this issue. Thanks @anshulpatel25

[smastrorocco]

+1

[bflad]

The fix for this has been merged in via #4341 and will release with v1.17.0 of the AWS provider, likely in a week.

[bflad]

This has been released in version 1.17.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[smastrorocco]

Is there a regression with this? I'm on v1.22.0 and having this issue. Setting parallelism to 1 works around the issue.

[DJAlPee]

Just got this with v1.23

I'm creating within a module a private and a public hosted zone. The module is instantiated four times. The error occurs when creating the four private hosted zones.

[pawelsocha]

@AlPee-DU can you share your terraform config?

[DJAlPee]

Module:

resource "aws_route53_zone" "pub_zone" {
  name  = "${var.zone_name}"
}

resource "aws_route53_zone" "prv_zone" {
  name   = "${var.zone_name}"
  vpc_id = "${data.aws_vpc.vpc.id}"
}

Main:

module "prod_stage" {
  source    = "module"
  zone_name = "prod.${local.domain}"

  providers = {
    "aws" = "aws.prod"
  }
}

module "test_stage" {
  source    = "module"
  zone_name = "test.${local.domain}"

  providers = {
    "aws" = "aws.test"
  }
}

[...]

(Also a use case for count in modules 😉)

[pawelsocha]

And the error is HostedZoneAlreadyExists?

[DJAlPee]

HostedZoneAlreadyExists: A hosted zone has already been created with the specified caller reference.

[pawelsocha]

@AlPee-DU you can try depends_on to avoid CallerReference collision.

resource "aws_route53_zone" "pub_zone" {
  name  = "${var.zone_name}"
}

resource "aws_route53_zone" "prv_zone" {
  name   = "${var.zone_name}"
  vpc_id = "${data.aws_vpc.vpc.id}"

  depends_on = ["aws_route53_zone.pub_zone"]
}

[pawelsocha]

and I change resource code to create CallerReference using uuid
https://github.com/NikkeiFTLearning/terraform-provider-aws/commit/95ebc2abe19df2c45c327f2bd736f7620ae0fc46

@bflad answer to you - what you think? it's good or no? :-)

[bflad]

We can switch this to use resource.UniqueId(), which calls resource.PrefixedUniqueId() under the hood and provides a counter wrapped with a mutex.

[DJAlPee]

My config already changed... ^^
Meanwhile I'm using a different zone name for the private hosted zone. Regarding to your PR, it seems that a different zone name already fixed this behavior for me 😄

[DJAlPee]

And not to forget: Thanks for the quick response 👍

[bflad]

Courtesy of #4903, version 1.25.0 of the AWS provider will use a unique identifier for the aws_route53_zone resource CallerReference instead of being dependent on the zone name and time, which should fully alleviate issues with concurrency. It should be released middle of next week. If there are continuing issues after that release, please file a new issue following the issue template and we'll further troubleshoot. Thanks!

[bflad]

This has been released in version 1.25.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!