Terraform-provider-aws: Creating Route53 Record Hangs Indefinitely

[hashibot[bot]]

_This issue was originally opened by @tinyzimmer as hashicorp/terraform#18490. It was migrated here as a result of the provider split. The original body of the issue is below._

---

I have an issue on the latest version of Terraform (and this has been working fine for about 2 weeks), where it will hang indefinitely while trying to create a route53 record. I can see in the console the record gets created and resolves correctly, however, terraform just hangs out until I interrupt. Below debug is just one loop, however, I let this hang for as long as 10 minutes.

Terraform Version

Terraform v0.11.7

provider.aws: version =  1.27
                 AND
provider.aws: version =  1.28

Terraform Configuration Files

Just your standard (I think) RDS setup, I redacted a few parts of the database config. I am adding a CNAME Route53 zone that remains constant for configuration in artifacts at the moment.

resource "aws_db_subnet_group" "db_subnets" {
  name       = "subnet-group"
  subnet_ids = ["${var.subnet_ids}"]
}

resource "aws_db_instance" "db" {
  allocated_storage    = 10
  storage_type         = "gp2"
  engine               = "mariadb"
  engine_version       = "10.2.15"
  instance_class       = "db.t2.micro"
  name                 = "mydb"
  username             = "supersecretusername"
  password             = "supersecretpassword"
  parameter_group_name = "default.mariadb"
  db_subnet_group_name = "${aws_db_subnet_group.db_subnets.name}"
}

resource "aws_route53_record" "database" {
  zone_id = "${var.route53_zone}"
  name = "db.${var.environment}.example.net"
  type = "CNAME"
  ttl = "300"
  records = ["${aws_db_instance.db.address}"]
}

Debug Output

module.database.aws_route53_record.database: Still creating... (50s elapsed)

2018/07/18 18:38:58 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
2018/07/18 18:38:58 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "module.database.aws_route53_record.database"
2018/07/18 18:38:58 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.database.aws_route53_record.database"

2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/18 18:39:01 [DEBUG] [aws-sdk-go] DEBUG: Request route53/GetChange Details:
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: GET /2013-04-01/change/[secret?] HTTP/1.1
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Host: route53.amazonaws.com
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: User-Agent: aws-sdk-go/1.14.26 (go1.9.2; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8-dev
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=redacted/us-east-1/route53/aws4_request, SignedHeaders=host;x-amz-date, Signature=redacted
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: X-Amz-Date: 20180719T013901Z
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Accept-Encoding: gzip
2018-07-18T18:39:01.494-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-18T18:39:01.495-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-18T18:39:01.495-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: -----------------------------------------------------

2018-07-18T18:39:01.829-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/18 18:39:01 [DEBUG] [aws-sdk-go] DEBUG: Response route53/GetChange Details:
2018-07-18T18:39:01.829-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: HTTP/1.1 200 OK
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Connection: close
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Length: 285
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Type: text/xml
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Date: Thu, 19 Jul 2018 01:39:00 GMT
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: X-Amzn-Requestid: 836fb72b-8af4-11e8-a81e-572a2010db76
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: -----------------------------------------------------
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/18 18:39:01 [DEBUG] [aws-sdk-go] <?xml version="1.0"?>
2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: <GetChangeResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ChangeInfo><Id>/change/redacted</Id><Status>PENDING</Status><SubmittedAt>2018-07-19T01:38:05.321Z</SubmittedAt><Comment>Managed by Terraform</Comment></ChangeInfo></GetChangeResponse>
2018-07-18T18:39:01.831-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/18 18:39:01 [TRACE] Waiting 10s before next try

2018/07/18 18:39:03 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
2018/07/18 18:39:03 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "module.database.aws_route53_record.database"
2018/07/18 18:39:03 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.database.aws_route53_record.database"

module.database.aws_route53_record.database: Still creating... (1m0s elapsed)

Expected Behavior

Everything is as it should be, let's continue on...

Actual Behavior

No...it's mad about something...

Steps to Reproduce

Hopefully, the code above (with needed variables) can reproduce. If it doesn't well damn...I've tried everything.

Additional Context

This is being run from a Jenkins job. We had some DNS issues earlier internally, but they got resolved, and it is clear from the debug output that it is getting a response from Amazon.

[tinyzimmer]

I slept on it, tried again in the morning, and it was able to move along as normal...

I'll leave it to yall if this should stay open or not. The fact that terraform succeeded in what it was trying to do but got stuck in endless PENDING loops _feels_ like a bug, But it could also have very well been AWS side.

[bflad]

Hi @tinyzimmer 👋 Sorry you ran into trouble here.

Its hard to discern from the AWS SDK debug log output above if there is anything wrong. As far as Terraform is concerned, this is a normal response, where Route53 has yet to make the requested change:

2018-07-18T18:39:01.830-0700 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: <GetChangeResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><ChangeInfo><Id>/change/redacted</Id><Status>PENDING</Status><SubmittedAt>2018-07-19T01:38:05.321Z</SubmittedAt><Comment>Managed by Terraform</Comment></ChangeInfo></GetChangeResponse>

Notably the Status:

<Status>PENDING</Status>

Which is signaling that the Route53 API has yet to successfully finish its work.

If you have additional logging to provide that somehow shows an unexpected status that Terraform doesn't understand or it is retrying on some error it shouldn't please do let us know and we'll reopen this. Thanks.

[jamesalbert]

Experiencing this as well. Terraform v0.11.13, aws provider 1.46.0. This started out of the blue so I'm going to wait like @tinyzimmer did and see if it heals itself. Getting an identical debug trace.

edit: aws outage. Not provider related.

[diegogamboa]

same problem, tried again in the morning and everything worked as expected

[philipp-classen]

I ran also into the exact issue (also reproducible with v2.14.0). Terraform was able to create the record, but it always got stuck in the sync state.

When I tried in a different AWS account, it immediately worked. One hour later, I tried again and it worked also in the previously broken account.

Looks like a temporary problem in AWS itself, but it can be very annoying when you are affected.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!