Terraform-provider-digitalocean: Multiple digitalocean_record: 500 Server was unable to give you a response.

[MarianoRD]

Terraform Version

Terraform v0.11.11
+ provider.digitalocean v1.1.0
+ provider.google v2.3.0

Affected Resource(s)

• digitalocean_record

Terraform Configuration Files

// blindcut-vpn.com
resource "digitalocean_domain" "default" {
  name         = "example.com"
}

// Monitoring
resource "digitalocean_record" "test" {
  domain = "${digitalocean_domain.default.name}"
  type   = "A"
  name   = "test"
  value  = "10.10.10.10"
}

Expected Behavior

Plan: x to add, x to change, x to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Actual Behavior

Error: Error refreshing state: 1 error(s) occurred:

* digitalocean_record.test: 1 error(s) occurred:

* digitalocean_record.test: digitalocean_record.test: GET https://api.digitalocean.com/v2/domains/blindcut-vpn.com/records/xxxxxxxx: 500 Server was unable to give you a response.

Sometimes it's just one record, sometimes there are 2 or 3 records that throw the error 500. Each run is usually a different record and a different amount.

Steps to Reproduce

1. Create about 25 records and 1 domain
2. terraform apply

[glynnforrest]

I'm seeing a similar issue running terraform plan, it's only started happening recently. Perhaps it's a network / API issue? Maybe the provider could handle 500 errors from the API better and retry?

[MarianoRD]

Yes @glynnforrest I think it's an issue that DigitalOcean might not be able to response to so many requests in so little time. It happens both with terraform plan and terraform apply.

Sometimes when terraform apply loads all the resources and terraform starts to perform the changes, it fails at the end but the resource was created.

Maybe just adding some kind of delay between each request might be enough, adding also a retry might be useful.

[eddiezane]

@MarianoRD @glynnforrest thanks for reporting and apologies for the inconvenience this may have caused. Looks like the team is aware of an issue here on our end (not related to Terrafrom at the moment). Will update when I have more info.

[acejam]

I have also been seeing this recently. Typically running a second or third time allows for things to pass.

[eddiezane]

Update:

DNS can sometimes just be flaky in general. We're baking more retry logic into a backend service that should fix this. Hoping it lands this week.

Thanks for your patience.

[eddiezane]

This landed today. Please reopen the issue if you're still seeing this and we'll keep at it.

[z3ntu]

Just got this error (plus a couple more of the same kind) - every time I try it's for different dns records

* digitalocean_record.z3ntu-aaurbs-CNAME: 1 error(s) occurred:

* digitalocean_record.z3ntu-aaurbs-CNAME: digitalocean_record.z3ntu-aaurbs-CNAME: GET https://api.digitalocean.com/v2/domains/z3ntu.xyz/records/40728127: 500 Server was unable to give you a response.

[alexburlton]

This is not fixed as of today. I've got a config using 22 digitalocean_record resources and it has basically become unusable. terraform apply bombs out 9 times out of 10.

My relevant versions are:

Terraform v0.11.13
+ provider.digitalocean v1.1.0

[eddiezane]

We pushed out some other changes and I think we were having a :cloud: :fire: when @alexburlton reported still seeing the issue.

Is anyone able to reproduce the failures now?

[z3ntu]

Working fine for me now.

[glynnforrest]

No problems over here! Thanks for following up @eddiezane.

[alexburlton]

Yep, can also confirm it's all good now. Thanks! 🎉

[alexburlton]

...and now it's happening again. Repeated 500 errors every terraform apply that we attempt. Another :cloud: :fire: ?

[eddiezane]

@alexburlton thanks for the heads up.

Looks like we had an issue this morning that should be stabalizing now http://status.digitalocean.com/incidents/b8r9g5t3rvxf. Going to tie what you saw to that.

Apologies for the inconvenience here.

[eddiezane]

Thanks for the patience, everyone. Going to (once again) close this but please pipe up if y'all see it again. We really do want to get it right and appreciate your help.

[StevenMilne-Sonocent]

Hey guys, I've just hit this issue today having to do a terraform plan about 3 or 4 times before it finally stopped throwing errors for Digital Ocean resources. The only difference is I'm not getting a 500 status code but a 503 Service is currently unavailable. The versions I am using are;

• Terraform v0.11.13
• provider.digitalocean v1.2.0

The errors that I was getting are;

* digitalocean_record.wildcard_glean_sonocent_dev: digitalocean_record.wildcard_glean_sonocent_dev: GET https://api.digitalocean.com/v2/domains/sonocent.dev/records/70336309: 503 Service is currently unavailable.
* digitalocean_domain.staging_sonocent_dev: 1 error(s) occurred:

* digitalocean_domain.staging_sonocent_dev: digitalocean_domain.staging_sonocent_dev: Error retrieving domain: GET https://api.digitalocean.com/v2/domains/staging.sonocent.dev: 503 Service is currently unavailable.
* digitalocean_domain.dev_sonocent_com: 1 error(s) occurred:

* digitalocean_domain.dev_sonocent_com: digitalocean_domain.dev_sonocent_com: Error retrieving domain: GET https://api.digitalocean.com/v2/domains/dev.sonocent.com: 503 Service is currently unavailable.
* digitalocean_domain.dev_sonocent_dev: 1 error(s) occurred:

* digitalocean_domain.dev_sonocent_dev: digitalocean_domain.dev_sonocent_dev: Error retrieving domain: GET https://api.digitalocean.com/v2/domains/dev.sonocent.dev: 503 Service is currently unavailable.

[eddiezane]

@StevenMilne-Sonocent thanks for the report and apologies for the inconvenience. Will raise with the team but it might be related to http://status.digitalocean.com/incidents/m75n07yv8khy

[pjanuario]

FYI - This problem was happening yesterday evening also.

[eddiezane]

We've made a few other changes over the past few weeks that should have hopefully helped. Anyone experience anything either way?

[MarianoRD]

Everything seems to be working fine here, and has been working fine the last couple of weeks. Thanks for everything @eddiezane