Terraform-provider-aws: Terraform apply does not delete Route53 record set while it says 'Resources: 1 added, 0 changed, 1 destroyed.'

[MBali-GitHub]

I have updated a Route53 record Set which needed the resource replacement. I get a success message from terraform apply for the same. But looking at Route 53 in AWS, i can see that the new record set was created but the old one was not deleted.

# aws_route53_record.recordSet must be replaced

$ terraform apply -input=false "planfile"
aws_route53_record.recordSet: Destroying... [id=xxxxxxxxxxx]
aws_route53_record.recordSet: Destruction complete after 1s
aws_route53_record.recordSet: Creating...
aws_route53_record.recordSet: Still creating... [10s elapsed]
aws_route53_record.recordSet: Still creating... [20s elapsed]
aws_route53_record.recordSet: Still creating... [30s elapsed]
aws_route53_record.recordSet: Creation complete after 38s [id=xxxxxxxxxxx]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

[KimSamba]

I have exactly the same issue, I need to delete the records manually after, otherwise i can have issues where i try to recreate it, but it already exists...

[grahamhar]

Adding some detail from my observation of this. I changed the domain that the records resided in, not sure if that has any relevance.

terraform version
Terraform v0.12.3

• provider.aws v2.16.0

aws_route53_record.wiremock[0]: Destroying... [id=XXXXXXXXX_www.redacted.com_CNAME]
aws_route53_record.wiremock[0]: Destruction complete after 2s
aws_route53_record.wiremock[0]: Creating...
aws_route53_record.wiremock[0]: Still creating... [20s elapsed]
aws_route53_record.wiremock[0]: Creation complete after 41s [id=YYYYYYYY_www.redacted.org_CNAME]

[Mochizuki-S2]

Adding my observation too.

Terraform Version

• terraform v0.12.0, v0.12.3, v0.12.4 (not happen on v0.11.14 )
• provider.aws v2.19.0

Affected Resource(s)

• aws_route53_record

Steps to Reproduce

1. create terraform.tf and terraform apply

~~~
$ cat terraform.tf
terraform {
}

provider "aws" {
region = "ap-northeast-1"
version = "~> 2.0"
profile = "sandbox"
}

resource "aws_route53_zone" "example" {
name = "my.foobar.example.com"
}

resource "aws_route53_record" "example" {
zone_id = "${aws_route53_zone.example.id}"
name = "foo"
records = ["1.1.1.1"]
type = "A"
ttl = 300
}
~~~

~~~
$ /tmp/terraform-v0.12.0 plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:

• create

Terraform will perform the following actions:

# aws_route53_record.example will be created

• resource "aws_route53_record" "example" {
  
  
  
  • allow_overwrite = (known after apply)
  
  
  • fqdn = (known after apply)
  
  
  • id = (known after apply)
  
  
  • name = "foo"
  
  
  • records = [
    
    
    
    
    
    • "1.1.1.1",
      
      
      ]
    
    
    
  
  
  • ttl = 300
  
  
  • type = "A"
  
  
  • zone_id = (known after apply)
    
    }
    
    # aws_route53_zone.example will be created
  
  
• resource "aws_route53_zone" "example" {
  
  
  
  • comment = "Managed by Terraform"
  
  
  • force_destroy = false
  
  
  • id = (known after apply)
  
  
  • name = "my.foobar.example.com"
  
  
  • name_servers = (known after apply)
  
  
  • vpc_id = (known after apply)
  
  
  • vpc_region = (known after apply)
  
  
  • zone_id = (known after apply)
    
    }
  
  

Plan: 2 to add, 0 to change, 0 to destroy.

$ /tmp/terraform-v0.12.0 apply -auto-approve
~~~

~
$ aws route53 list-resource-record-sets --hosted-zone-id Z1DIV15ETMOU37 --profile sandbox | jq '.ResourceRecordSets[].Name'
"my.foobar.example.com."
"my.foobar.example.com."
"foo.my.foobar.example.com."
~

1. change record name from foo to bar and terraform apply again

~~~
$ cat terraform.tf
terraform {
}

provider "aws" {
region = "ap-northeast-1"
version = "~> 2.0"
profile = "sandbox"
}

resource "aws_route53_zone" "example" {
name = "my.foobar.example.com"
}

resource "aws_route53_record" "example" {
zone_id = "${aws_route53_zone.example.id}"
# name = "foo"
name = "bar"
records = ["1.1.1.1"]
type = "A"
ttl = 300
}
~~~

~~~
$ /tmp/terraform-v0.12.0 plan

# aws_route53_record.example must be replaced
-/+ resource "aws_route53_record" "example" {
+ allow_overwrite = (known after apply)
~ fqdn = "foo.my.foobar.example.com" -> (known after apply)
~ id = "Z1DIV15ETMOU37_foo_A" -> (known after apply)
~ name = "foo" -> "bar" # forces replacement
records = [
"1.1.1.1",
]
ttl = 300
type = "A"
zone_id = "Z1DIV15ETMOU37"
}

Plan: 1 to add, 0 to change, 1 to destroy.
~~~

~
$ /tmp/terraform-v0.12.0 apply -auto-approve
~

1. list-resource-record-sets ( first record remains )

~
$ aws route53 list-resource-record-sets --hosted-zone-id Z1DIV15ETMOU37 --profile sandbox | jq '.ResourceRecordSets[].Name'
"my.foobar.example.com."
"my.foobar.example.com."
"bar.my.foobar.example.com."
"foo.my.foobar.example.com."
~

Expected Behavior

delete first route53 record (foo.my.foobar.example.com.)

Debug Output

before Destruction, findRecord not correct one (bar.my.foobar.example.com)

• detail is around 2nd findRecord

~
$ TF_LOG=DEBUG /tmp/terraform-v0.12.0 apply -auto-approve
...
2019-07-18T19:34:20.277+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:20 [DEBUG] Expanded record name: foo.my.foobar.example.com
...
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:33 [DEBUG] Expanded record name: bar.my.foobar.example.com
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:33 [DEBUG] List resource records sets for zone: Z1DIV15ETMOU37, opts: {
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: HostedZoneId: "Z1DIV15ETMOU37",
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: MaxItems: "1",
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: StartRecordName: "bar.my.foobar.example.com.",
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: StartRecordType: "A"
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: }
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:34:33 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ListResourceRecordSets Details:
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: GET /2013-04-01/hostedzone/Z1DIV15ETMOU37/rrset?maxitems=1&name=bar.my.foobar.example.com.&type=A HTTP/1.1
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: Host: route53.amazonaws.com
2019-07-18T19:34:33.975+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: User-Agent: aws-sdk-go/1.20.17 (go1.12.6; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.2
...
aws_route53_record.example: Destruction complete after 2s
...
2019-07-18T19:35:13.413+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 19:35:13 [DEBUG] Expanded record name: bar.my.foobar.example.com
...
~

terraform v0.11.14 looks correct

• around 2nd findRecord too

~
$ TF_LOG=DEBUG /tmp/terraform-v0.11.14 apply -auto-approve
...
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 20:49:06 [DEBUG] Expanded record name: foo.my.foobar.example.com
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 20:49:06 [DEBUG] List resource records sets for zone: Z392DQN8J7NMJQ, opts: {
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: HostedZoneId: "Z392DQN8J7NMJQ",
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: MaxItems: "1",
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: StartRecordName: "foo.my.foobar.example.com.",
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: StartRecordType: "A"
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: }
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: 2019/07/18 20:49:06 [DEBUG] [aws-sdk-go] DEBUG: Request route53/ListResourceRecordSets Details:
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: GET /2013-04-01/hostedzone/Z392DQN8J7NMJQ/rrset?maxitems=1&name=foo.my.foobar.example.com.&type=A HTTP/1.1
2019-07-18T20:49:06.196+0900 [DEBUG] plugin.terraform-provider-aws_v2.19.0_x4: Host: route53.amazonaws.com
...
aws_route53_record.example: Destruction complete after 39s
~

[axw-pivorra]

Hi,
Same issue detected for CNAME and PTR route53 records.
Old records are never destroyed but terraform apply said that resources are destroyed.

[vsalama]

Hi,
same issue with:
terraform v0.12.6
provider.aws v2.23.0
but not with:
terraform v0.11.10
provider.aws v2.23.0

[vsalama]

and work also on:
terraform v0.11.14
provider.aws v2.23.0
end does not work on:
terraform v0.12.1
provider.aws v2.23.0
I think it's regression since terraform 12

[stephendavidmarsh]

I am also experiencing this issue, where old records are not deleted. In my case it also seems to not check if a record already exists under that name before updating the zone. As a result, it will write over records, including ones that were not managed by Terraform or leftover records from the deletion issue.

[NurSpass]

I am experiencing this issue as well. I was able to work around the issue by using the lifecycle hook in my records. lifecycle { create_before_destroy = true }

[stephendavidmarsh]

I did more testing around the writing over existing records behavior that I had observed. It doesn't actually update existing records, it _deletes_ them, then creates a new record on top. This is obviously quite dangerous, because if you accidentally tell Terraform to create a new record with the same name as an old one, you will temporarily not have a record for that domain name.

With the workaround @NurSpass suggested (lifecycle { create_before_destroy = true }) it doesn't have this behavior. If it encounters an already existing record, it will stop and fail. When there is no existing record, it correctly deletes the old one like it should.

My theory is that Terraform, without create_before_destroy, is deleting the _new_ record instead of the old one. This is both why it will destroy existing records and why it doesn't clean up the old one.

[Mochizuki-S2]

I created pull request #9965

We might be able to avoid this issue to use not d.Get("name") but d.GetChange("name")

I confirmed the following.

---

I try printf debugging on findRecord as below when aws_route53_record.name forces replacement

~~~go
i := 0
for {
pt, file, line, ok := runtime.Caller(i)
if !ok {
break
}
funcName := runtime.FuncForPC(pt).Name()
log.Printf("file=%s, line=%d, func=%v\n", file, line, funcName)
i += 1
}

before_name, after_name := d.GetChange("name")
log.Printf("id=%s zone_id=%s name=%s change_name=%s,%s fqdn=%s type=%s\n", d.Id(), d.Get("zone_id"), d.Get("name"), before_name, after_name, d.Get("fqdn"), d.Get("type"))
~~~

then print as below

file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=629, func=github.com/terraform-providers/terraform-provider-aws/aws.findRecord
file=.../github/terraform-provider-aws/aws/resource_aws_route53_record.go, line=526, func=github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsRoute53RecordRead
...
id=Z17UM5KN8LG9IH_foo_A zone_id=Z17UM5KN8LG9IH name=foo change_name=,foo fqdn= type=A
...
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=629, func=github.com/terraform-providers/terraform-provider-aws/aws.findRecord
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=737, func=github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsRoute53RecordDelete
...
id=Z17UM5KN8LG9IH_foo_A zone_id=Z17UM5KN8LG9IH name=bar change_name=foo,bar fqdn=foo.my.foobar.example.com type=A
...
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=629, func=github.com/terraform-providers/terraform-provider-aws/aws.findRecord
file=.../terraform-provider-aws/aws/resource_aws_route53_record.go, line=459, func=github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsRoute53RecordCreate
...
id=Z1LGSSII0WM0DD_bar_A zone_id=Z1LGSSII0WM0DD name=bar change_name=,bar fqdn= type=A

• this time, I try chaning route53 zone too as below

resource "aws_route53_record" "example" {
  # zone_id = "${aws_route53_zone.example.id}"
  zone_id = "${aws_route53_zone.example2.id}"
  # name    = "foo"
  name    = "bar"
  records = ["1.1.1.1"]
  type    = "A"
  ttl     = 300
}

---

I think it may not be right that d.Get("name") returns new value when destroying.

But I'm not sure, so create workaroud pull request.

[pimperator]

also affected:
Terraform v0.12.24

• provider.archive v1.3.0
• provider.aws v2.56.0
• provider.template v2.1.2

[minhluantran017]

The same issue with Route53 record deletion:

• Terraform 0.12.25
• provider.aws: version = "~> 2.63"
• provider.template: version = "~> 2.1"

[gawbul]

Having the same issue with Route53 record deletion:

• Terraform version: 0.12.11
• provider.aws version: "= 2.62.0"
• provider.template version: "~> 2.1.2"
• provider.archive version: "~> 1.3.0"
• provider.external version: "~> 1.2.0"

Would be great to see https://github.com/terraform-providers/terraform-provider-aws/pull/11335 merged in as a possible fix please? 👍 @bflad

[bflad]

The fix for this has been merged and will release with version 2.69.0 of the Terraform AWS Provider, likely tomorrow. Thanks to @jbergknoff-rival for the implementation. 👍

[hashibot[bot]]

This has been released in version 2.69.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!