Terraform-provider-newrelic: Internal Server Error - newrelic_alert_policy_channel linking not working

[hackjoy]

Terraform Version

0.11.11

Affected Resource(s)

• terraform-provider-newrelic -> newrelic_alert_policy_channel

Terraform Configuration Files

Sample below - I have 6 app policies linked to same alert channel

resource "newrelic_alert_policy" "app_policy" {
  name = "app-policy"
}
resource "newrelic_alert_policy" "app2_policy" {
  name = "app2-policy"
}
resource "newrelic_alert_channel" "slack" {
  name = "Alerts"
  type = "slack"
  configuration = {
    url              = "${var.slack_alerts_channel_webhook_url}"
    channel = "alerts"
  }
}
resource "newrelic_alert_policy_channel" "alert_slack_app" {
  policy_id  = "${newrelic_alert_policy.app_policy.id}"
  channel_id = "${newrelic_alert_channel.slack.id}"
}
resource "newrelic_alert_policy_channel" "alert_slack_app2" {
  policy_id  = "${newrelic_alert_policy.app2_policy.id}"
  channel_id = "${newrelic_alert_channel.slack.id}"
}

Debug Output

https://gist.github.com/hackjoy/e5423875f2b4e2b5aeedb078b0df659a

Expected Behavior

• No internal server errors should occur in output
• Policies should be correctly linked to alert channel in New Relic Alerts UI

Actual Behavior

• All resources (policies and alert channels) are created successfully however
  
  
  
  • Internal server errors shown in output
  
  
  • Some alert policies are linked to the desired channel but not all. The result varies on each run of terraform apply. Of the 6 newrelic_alert_policy_channel's I have, sometimes only 2, 3 or 4 of the policies are correctly linked together.
  
  

Steps to Reproduce

1. Create 6 alert policies linked to a single alert channel and apply the changes with terraform apply

References

Potentially relevant issues discussed:

• https://discuss.newrelic.com/t/internal-server-error-when-building-synthetic-monitors-in-a-terraform-conifguration/64785/4
• https://github.com/terraform-providers/terraform-provider-aws/issues/817

[slizco]

I'm also seeing this same exact behavior. I am trying to link ~50 policies to a single channel during one apply. About half succeed after the first apply. Each subsequent apply only adds a few more policy_channels, with those that failed creation resulting in errors like the following:

 newrelic_alert_policy_channel.container_orch_policy_channels.14: Internal Server Error
* module.policies.newrelic_alert_policy_channel.lambda_orch_policy_channels[5]: 1 error(s) occurred:

* newrelic_alert_policy_channel.lambda_orch_policy_channels.5: Internal Server Error
* module.policies.newrelic_alert_policy_channel.lambda_orch_policy_channels[2]: 1 error(s) occurred:

* newrelic_alert_policy_channel.lambda_orch_policy_channels.2: Internal Server Error

[slizco]

I was able to force this to work by passing -parallelism=1. Terraform is recognizing the resource dependencies correctly, so this leads me to believe the issue may be with the New Relic REST API itself. The New Relic API docs state that linking a policy to a channel is actually a PUT (update) operation which leads me to suspect that New Relic itself cannot handle more than one PUT at a time for a single channel. I have a support ticket in with them right now about this and will share back any meaningful results. Obviously, restricting Terraform to mutating only 1 resource at a time is not ideal.

[paultyng]

If this is just an issue with this resource, we can add some throttling specific to it (mutex or something) so we don't have to reduce the parallelism for everything.

[sgsiebers]

I've seen similar behavior as well, particularly when applying large sets of changes, and I'm fairly certain I've seen it with other resource types. I suspect this is caused by API overload protections as documented here:

https://docs.newrelic.com/docs/apis/rest-api-v2/requirements/api-overload-protection-handling-429-errors

Perhaps the provider just needs to know how to recognize API overload error codes and pause/retry transactions when they fail due to overload?

[slizco]

I heard back from New Relic support on this, and they confirmed our suspicion that the issues stems from updates to the channel occurring in parallel. If we could write something in the provider code that forces serial creation ofpolicy_channels, that will fix our problem. It sounds like New Relic themselves are tracking the bug as well, but there's no guarantees or indication of when it will be addressed.

I can't speak to how this interplays with the overload protection handling as I've only seen issues with this particular resource and have also seen it when only attaching 3-4 policies to a channel at a time.

[hackjoy]

Nice work @slizco, i've confirmed that -parallelism=1 solves the issue in the interim.
Look forward to your PR getting merged 😄

[slizco]

@hackjoy I just heard from New Relic that they resolved this issue on their end. I did some testing, and the policy_channel updates are working for me now without --parallelism=1. Hopefully it will work for you too.

[paguos]

We just encountered the same problem. Is anyone else had the same issue?

Luckily the -parallelism=1 flag solved the problem.

[jthurman42]

@paguos This issue is over a year old, what version of Terraform and the New Relic provider are you using now?

[paguos]

I was surprised too.

@paguos This issue is over a year old, what version of Terraform and the New Relic provider are you using now?

Thanks for the reply! It seems it was a temporary problem with the API. Today it worked as expected.

[jthurman42]

Interesting, thanks for confirming it was a transient issue!