Terraform-provider-newrelic: Issue with New Relic Provider migration to 2.X

[BS075209]

Hi Team,

We are currently using Terraform New Relic provider version – 1.16 which works fine for all our environments.

We wanted to upgrade to the Terraform New Relic provider version 2.6.1 as per the guidelines for which we have tried the below actions:
• We added the below fields in the provider block as variables:
provider "newrelic" {
api_key = "NRAK-####"
admin_api_key = "NRAA-####"
account_id=**
region=US
}
• As recommended we also tried migrating to the latest version of 1.X that is 1.20.1 and then moved to 2.X.
• We have also tried passing the above values as environment variables.

All of the above ways have error-ed out with 403 response as below. A snapshot of the error log has been attached for the same. Also, the error is quite inconsistent as it passes sometimes and fails most of the times.

Kindly suggest if the below could be resolved or if we are missing out on any other details for the migration.

[zlesnr]

Hi there,

We do recommend using the environment variables for configuring the provide as perhaps you've seen here: https://registry.terraform.io/providers/newrelic/newrelic/latest/docs/guides/provider_configuration

Though, the 403 indicates that the credentials are recognized by the service, but the account from which those credentials are pulled does not have the appropriate rights to manage the resources being attempted.

Can you double check the access for the user who's creds you are using?https://account.newrelic.com/accounts//users

[BS075209]

Hi @zlesnr
We cross-checked the credentials we are using and it seems to be correct.
Also, isn't the permission failure should continue for all the builds. Permission should always be checked and should fail every time. But in this case it's very inconsistent where it passes sometimes and fails sometimes.
Is it possible that it might be a security issue with New Relic itself.

Please suggest on the same. Thanks.

[BS075209]

We ran the terraform plan with TRACE ON by which we got 403 errors for different services every time we ran the build and sometimes it builds successfully without any errors at all.

Please suggest on the same. Thanks.

[sanderblue]

Hi @BS075209, I noticed this ticket looks like a post in our community portal. Were you able to resolve the permissions issue? If not, let's continue the conversation here on GitHub so our dev team can track it a little more closely. 🙂

[BS075209]

Hi @sanderblue
Thanks for the quick response. Sorry but the permission issue still exists. Also, have provided the checks made along with other findings on the post in your community portal.

Note : One more thing to add here is that the change includes s3(with locking via DynamoDB) enabled, but when we disable that, the terraform plan runs smoothly. This approach however might not be considered as a migration as that would be a fresh start with 2.X provider.

[BS075209]

Hi @sanderblue
There's a similar issue raised on the post in your community portal. Could you please provide any update/comments on the same. Thanks.

[SiddhantAgarwal17]

@sanderblue @zlesnr This issue has been preventing us from accessing lot of the features that are available with 2.x version of this provider. Any suggestion as to what we can try out to get more details around this issue? As @BS075209 mentioned, we leverage the keys from an admin account (someone with Full admin access). The same account is used with v1.16 as well, although without any issues.

[zlesnr]

How many alerts resources are we talking about for this terraform repo? Another issue I was on a call with yesterday sounds somewhat similar to this if its intermittent.

[SiddhantAgarwal17]

How many alerts resources are we talking about for this terraform repo? Another issue I was on a call with yesterday sounds somewhat similar to this if its intermittent.

So, we create about 500 odd policies & around 1250 conditions. Accordingly the alert policy and condition binding resource is created. Currently, we have only one notification channel but it binds to 500 policies. This is how it looks like:

We create separate alerting configuration against each of our environment ( dev, staging, cert, sandbox and production). dev, staging and cert, report to one New Relic account. Sandbox and Production report to a different one. So, 5 times the configuration mentioned above. However, we haven't breached the New Relic Alerting limits yet

[SiddhantAgarwal17]

Another issue I was on a call with yesterday sounds somewhat similar to this if its intermittent.

It is intermittent. Sometimes the plan passes but the apply fails. Sometimes both the stages pass and sometimes none of them.

[zlesnr]

There are a few things in flight about this sort of issue at the moment. We're planning to test a code change this week that was unblocked by an API update a couple weeks ago, which should reduce the call count and thus the load on the alerts service. The backing service teams are chasing the intermittent 403 as well. Give us some time to chase this and we'll see if we can come up with something in this provider. Note that will almost certainly require you to upgrade the provider, so preparing for that if you need to would be prudent.

[SiddhantAgarwal17]

We planned to do an upgrade anyway. Only thing is that we'll have to worry about the resources or parameters that would no longer be supported with the latest version that you plan to release. This helps a lot. Let us know whenever the version with the changes is released.

[zlesnr]

While digging in the code, I also noticed that the change we intend to make in https://github.com/newrelic/terraform-provider-newrelic/pull/919 is already implemented for the nrql_alert_condition resource. So if you need to remain on an older version of the provider for whatever reason, converting your conditions to use that resource may also help here.

[zlesnr]

v2.8.0 has been released. Let us know if that resolves this issue.

[BS075209]

@zlesnr @SiddhantAgarwal17
Thanks for your suggestion and quick response. After the 2.8.0 upgrade only two builds have failed so far among 15 builds due to timeout and 403 respectively.

The upgrade has made the success rate very high compared to how it was earlier. But still there's a small chance of failure, also we would need to try out the same with the other environments as well which approximately would be five times of the current configuration without breaching the New Relic Alerting limits.

Kindly suggest if it would be fine to upgrade to the latest New Relic provider now from 1.X version or we should wait until the 2.X version becomes more stable.

[zlesnr]

Thank you for the update. Can you tell me, for the failures you are seeing after the upgrade, which resources you are receiving the 403 for? I did not modify the synthetics condition resources yet, since I think there is still some API work going on there. There should be a DEBUG level output that has the request failure message in it as well that I would like to see please.

I can reread the code for the 1.x, but I would expect the failure rate for the resources in question here to be higher on the v1 after the modification that was done in 2.8.0. The issue appears to be high call count against the alerts api causing some perfect storms to throw back the 403, which teams are working to resolve currently.

[BS075209]

Hi @zlesnr
Please find the below attached error for the same.

Note: The API as in the image above varies from time to time and is random. Also, after the current upgrade, the success rate of the builds has gone up(approx. 1 out of 30 builds is failing at the moment). Let us know, if you need further details for it. Thanks.

[BS075209]

Hi @zlesnr
We can see the latest version for the terraform Newrelic provider is 2.10.2. Could you please confirm if it's the stable version for 2.X or 2.8 is good to go for the migration from 1.X. Thanks in advance.

[BS075209]

hi @zlesnr @sanderblue
With the 2.8.0 version the success rate has gone down. Compared to 3 days back where 1 out of 30 builds were failing, now fails 1 out of 7 builds(approx.). It seems the 2.8.0 is not a stable version. It fails with "403 error", "Internal server error" and "timeout error" at the moment. Also, we can see the latest version as of now is 2.10.3, could you please confirm which version for 2.X should be used for the migration from 1.X.

[BS075209]

hi @zlesnr @sanderblue any updates on the above, we have tried using the 2.10.3 version which increased the success rate but still cannot be called a stable version I assume. Please let me know your thoughts on the same.

[zlesnr]

Some API side changes were made today that may help the situation. As long as you are 2.8.0 or greater, you should have some improvement. Let us know how that goes.

[BS075209]

Thanks @zlesnr We were testing out the builds using the 2.10.3 provider as of now. It works fine in our testing environment. We will try the same upgrade on our dev environment where the load would be much high and see how the success rate varies. Thanks for your help and confirmation.

[sanderblue]

Closing as this seems to have been resolved. If you run into any other issues, please open a new ticket 🙂