Terraform-provider-newrelic: Cannot create alert channels with personal API key with Alert Manger add-on

[mlitwin]

Terraform Version

0.12.28

Affected Resource(s)

• newrelic_alert_channel

Terraform Configuration Files

provider "newrelic" {
  # https://github.com/newrelic/terraform-provider-newrelic/releases/latest
  version = "2.2.1"
  region = "US"
  account_id = "1234567"
  api_key = "NRAK-***"
  admin_api_key = "NRAA-***"
}

resource "newrelic_alert_policy" "foo" {
  name = "example"
  incident_preference = "PER_POLICY" # PER_POLICY is default
}

resource "newrelic_alert_channel" "first" {
  name = "bar"
  type = "email"

  config {
    recipients              = "[email protected]"
    include_json_attachment = "1"
  }
}

resource "newrelic_alert_channel" "second" {
  name = "bar"
  type = "email"

  config {
    recipients              = "[email protected]"
    include_json_attachment = "1"
  }
}

resource "newrelic_alert_policy_channel" "testing" {
  policy_id = newrelic_alert_policy.foo.id
  channel_ids = [
    newrelic_alert_channel.first.id,
    newrelic_alert_channel.second.id,
  ]
}

Debug Output

https://gist.github.com/mlitwin/77f2b00b62b7b45db93f3abd8e5cd3e6

Expected Behavior

Successful creation

Actual Behavior

Error: 401 response returned: User does not have permission.

Steps to Reproduce

1.export NEW_RELIC_API_KEY="NRAK-***" #773

1. terraform apply

Important Factoids

The personal API key was created by a user with Alert Manger add-on, but not an Admin.

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• #773
• #790

[dvishniakov]

From NR support:

While an Alerts Manager can indeed manage notification channels in the UI, our Alerts REST API explicitly requires an Admin API key which only an Admin can generate. In general, our REST API is gated to Admin-level users, as it requires either that key or the account-wide REST API key to use.

Please consider modifying this plugin to use Admin key for managing Alert Notification Channel instead of personal user key.

[jthurman42]

Terraform today requires both API_KEY and ADMIN_API_KEY to be set for full coverage. Notification channels are still created via our older REST API (relevant Terraform and newrelic-client-go code sections).

Until the upstream API has been migrated to NerdGraph, this is working as expected. Please ensure that both of the following variables are exported or configured:

• NEW_RELIC_API_KEY (Personal API Key)
• NEW_RELIC_ADMIN_API_KEY (Admin API key for account in question)

[skyzyx]

NOTE: @mlitwin and @dvishniakov are my co-workers.

@jthurman42: We are providing BOTH. Here is a real world example:

NEW_RELIC_REGION=US \
NEW_RELIC_ACCOUNT_ID=-****** \
NEW_RELIC_API_KEY=NRAK-****** \
NEW_RELIC_ADMIN_API_KEY=NRAA-****** \
terraform plan

What appears to be happening is that although we are providing both an Admin and a Personal key, the Terraform Provider is using the Personal Key when making an HTTP request to the REST API (to create the Notification Channel) instead of the Admin Key.

[jthurman42]

I am unable to reproduce using the config above (with valid email addresses), commenting out account_id/api_key/admin_api_key and sending those via environment variables.

Config

provider "newrelic" {
  # https://github.com/newrelic/terraform-provider-newrelic/releases/latest
  version = "2.2.1"
  region = "US"
  #account_id = "<REDACTED>"
  #api_key = "<REDACTED>"
  #admin_api_key = "<REDACTED>"
}

resource "newrelic_alert_policy" "foo" {
  name = "example"
  incident_preference = "PER_POLICY" # PER_POLICY is default
}

resource "newrelic_alert_channel" "first" {
  name = "bar"
  type = "email"

  config {
    recipients              = "<REDACTED>"
    include_json_attachment = "1"
  }
}

resource "newrelic_alert_channel" "second" {
  name = "bar"
  type = "email"

  config {
    recipients              = "<REDACTED>"
    include_json_attachment = "1"
  }
}

resource "newrelic_alert_policy_channel" "testing" {
  policy_id = newrelic_alert_policy.foo.id
  channel_ids = [
    newrelic_alert_channel.first.id,
    newrelic_alert_channel.second.id,
  ]
}

Shell execution

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "newrelic" (terraform-providers/newrelic) 2.2.1...


$ export NEW_RELIC_API_KEY=<REDACTED>
$ export NEW_RELIC_ADMIN_API_KEY=<REDACTED>
$ export NEW_RELIC_ACCOUNT_ID=<REDACTED>
$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # newrelic_alert_channel.first will be created
  + resource "newrelic_alert_channel" "first" {
      + id   = (known after apply)
      + name = "bar"
      + type = "email"

      + config {
          + include_json_attachment = "1"
          + recipients              = "<REDACTED>"
        }
    }

  # newrelic_alert_channel.second will be created
  + resource "newrelic_alert_channel" "second" {
      + id   = (known after apply)
      + name = "bar"
      + type = "email"

      + config {
          + include_json_attachment = "1"
          + recipients              = "<REDACTED>"
        }
    }

  # newrelic_alert_policy.foo will be created
  + resource "newrelic_alert_policy" "foo" {
      + account_id          = <REDACTED>
      + id                  = (known after apply)
      + incident_preference = "PER_POLICY"
      + name                = "example"
    }

  # newrelic_alert_policy_channel.testing will be created
  + resource "newrelic_alert_policy_channel" "testing" {
      + channel_ids = (known after apply)
      + id          = (known after apply)
      + policy_id   = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

newrelic_alert_policy.foo: Creating...
newrelic_alert_channel.first: Creating...
newrelic_alert_channel.second: Creating...
newrelic_alert_policy.foo: Creation complete after 1s [id=982341]
newrelic_alert_channel.first: Creation complete after 1s [id=3820899]
newrelic_alert_channel.second: Creation complete after 1s [id=3820898]
newrelic_alert_policy_channel.testing: Creating...
newrelic_alert_policy_channel.testing: Creation complete after 1s [id=982341:3820898:3820899]

[dvishniakov]

This is a redacted output using TF_LOG=trace:

2020-07-25T02:46:57.890Z [DEBUG] plugin.terraform-provider-newrelic_v2.3.0: time="2020-07-25T02:46:57Z" level=trace msg="request details" body="&{{0 XYZ-CHANNEL-NAME webhook { ....headers="{\"Api-Key\":[\"NRAK-**\"],\"Auth-Type\":[\"User-Api-Key\"]...

[dvishniakov]

@jthurman42 , can you please double check that you don't have Admin role in the account you're using?
Also, I see that you've reproduced using version 2.2.1 of the plugin, while I've tried with 2.3. Not sure if that changes anything, but just to note the difference.

[skyzyx]

If you include appropriate environment variables for:

• Admin Key
• Personal Key of a non-admin user

The Personal Key of a non-admin user is being used for creating a new notification channel instead of the Admin Key. This is causing a failure since creating notification channels in the REST API is admin-only.

From what I can tell, the fix is to update the provider to use the Admin Key instead of the Personal Key when making the request to the REST API for managing notification channels.

[mlitwin]

I've re-verified the issue still occurs with valid emails and commenting out account_id/api_key/admin_api_key and sending those via environment variables.

[jthurman42]

Alright, after tracing this down to the HTTP authorizer in newrelic-client-go (the library under TF and other tools), the preference is to Authenticate via the Personal API key for all requests that allow for it in place of an Admin API key, which today DOES include Notification Channels.

However, the REST endpoint does not respect add-on roles, only Admin/Owner/User/Restricted, so a Non-Admin user's Personal API Key can not be used for creating notification channels, and the Admin API key is ignored in this case. Since the user would have access to the Admin API key to configure Terraform, it is assumed that they would have Admin access.

So while this doesn't work perhaps as expected, it is working as currently designed until Notification channels are migrated to NerdGraph. As that work is in-flight, we will not be addressing this in the Terraform provider, and add clarification to the documentation on the current requirement for Admin rights for all keys to configure all resources (keeping with existing 1.x behavior).

[skyzyx]

Thank you for your follow-up @jthurman42. It is appreciated, even if I'm unhappy with the end-result. Your explanation about the assumptions made in the REST API explain why we've encountered this issue.

Looking forward to more NerdGraph in the future.

[jthurman42]

Thanks for the report, and while we do not support it today, there are a lot of API related improvements for upstream going on right now. I'm making a PR now to update the docs to be more specific, and will try to call out the auth a little better here.

[mlitwin]

Thanks @jthurman42 - docs updates are appreciated. And like Ryan, I look forward to the more sane API key simplicity of the NerdGraph approach.

[dvishniakov]

Thank you @jthurman42; you can also mention support ticket 414582 in PRs if that adds any value.

[mlitwin]

Confirming for the record here: If you use a Personal API Key of a user with Admin level permissions, this test case works.