Terraform-provider-newrelic: Eliminate drift in `newrelic_alert_channel` resources

[ctrombley]

The alert channel API endpoint fails to return resource fields that it deems sensitive, which currently causes drift when using affected channel configurations. In the past, we have recommended using the lifecycle block with an ignore_changes attribute as described in the getting started guide to eliminate drift.

In situations where the underlying API fails to return a resource field that has been supplied in the config, a pattern that can be used to avoid drift is to simply not set the sensitive field when flattening the API response. This pattern has been used to implement the newrelic_synthetics_secure_credential resource and it should be able to eliminate the drift in the newrelic_alert_channel resource as well. We should experiment with this pattern and see if there is a way to eliminate the drift associated with sensitive data in a backward-compatible manner.

[sanderblue]

Another thing that could help with detecting drift is making sure we call READ at the end of all modifications to synchronize state. We don't currently do that in the newrelic_alert_channel resource.

Reference: Detecting Drift

... it is standard practice to call READ at the end of all modifications to synchronize immediately and avoid that diff.

We have several other resources that need to be updated to use this pattern as well.

[davidji99]

It seems importing newrelic_alert_channel also doesn't set any of the config attributes. My state file for this resource looks like this:

"attributes": {
"config": [
{
"api_key": "",
"auth_password": "",
"auth_type": "",
"auth_username": "",
"base_url": "",
"channel": "",
"headers": {},
"headers_string": "",
"include_json_attachment": "",
"key": "",
"payload": {},
"payload_string": "",
"payload_type": "",
"recipients": "",
"region": "",
"route_key": "",
"service_key": "",
"tags": "",
"teams": "",
"url": "",
"user_id": ""
}
],
"configuration": null,
"id": "<SOME_ID>",
"name": "<SOME_NAME>,
"type": "<SOME_TYPE>"
},

On a subsequent plan, the diff shows a destroy and recreate:

  # newrelic_alert_channel.pagerduty-page-oncall must be replaced
-/+ resource "newrelic_alert_channel" "pagerduty" {
      ~ id   = "<SOME_ID>" -> (known after apply)
        name = "<SOME_NAME>"
        type = "pagerduty"

      ~ config {
          - headers     = (sensitive value)
          - payload     = (sensitive value)
          + service_key = (sensitive value)
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

The documentation doesn't mention the need to define the headers or the payload attributes so it is odd that this diff is showing up.

I guess I'll use the data source for now.

[ZsoltPath]

In case of "slack" type channel it's more of problem about the available config parameters.
As @davidji99 wrote it above the state stores the header and payload config parameters while they are only valid to the Webhook type.
And even the NR API explorer not returning them.

[samjgalbraith]

It kind of points to future APIs for infrastructure management like New Relic API having to either:

• Accept sensitive values on the read APIs for comparison OR
• Return cryptographic hashes of sensitive values for client side comparison

Any system which needs to reconcile configuration is going to need to do something like this, ultimately, otherwise it's going to just have to accept drift of anything that it can't either explicitly request reconciliation for or read back out in raw or hashed form.

[casey-robertson]

I wonder if this is what I'm running into. Doing a quick POC of the provider (2nd hand via Pulumi) and it behaves oddly (at least to me). I think it's around the changing IDs. I have a simple stack that ends up in a bad state because it ends up not being able to find an InfraAlertCondition.

My stack creates:

Notification channel 1
Notification channel 2 (both are slack with a url and channel name)

creates an AlertPolicy that depends on the Notification channel 1 being created (so it can be referenced) in channelIDs

Creates an InfraAlertCondition (the InfraAlertConditionArgs reference the id of the AlertPolicy)

Somehow after multiple runs, the program will end up with the InfraAlertCondition in a state of Not Found.

More work to do.

[aries-zhang]

Is it possible to raise priority on this issue?

This is invalidating terraform's output about resources changes. The plan result can not be trusted anymore. Real-unwanted changes may be hidden in too many unnecessary changes and cause real problems.

It's a pain to check each one of the resources to make sure I'm not destroying the world.

Please, fix it soon.

[sanderblue]

Hi all! A solution is in the works :)

@aries-zhang (and anyone else too) - Due to the multitude of scenarios this resource handles, do you mind sharing some of your alert_channel configurations (sensitive values redacted). We want to make sure we handle all of the scenarios/channels as best as possible so we need to know what your config attributes look like. Thanks in advance!

[hhughes0]

I'm also seeing the same destroy/re-create behaviour with a pagerduty alert channel.

# Creates a pagerduty alert channel
resource "newrelic_alert_channel" "pagerduty" {
  name = var.service_name
  type = "pagerduty"

  config {
    service_key = var.service_key
  }
}

Terraform plan output:

  # module.demo.newrelic_alert_channel.pagerduty must be replaced
-/+ resource "newrelic_alert_channel" "pagerduty" {
      ~ id   = "1234567" -> (known after apply)
        name = "demo"
        type = "pagerduty"

      ~ config {
          - headers     = (sensitive value)
          - payload     = (sensitive value)
          + service_key = (sensitive value)
        }
    }
Plan: 1 to add, 0 to change, 1 to destroy.

Prior to this, the terraform apply executes successfully with no errors and creates the alert channel.
Running a terraform plan following this with no changes results in the above output.

[ZsoltPath]

This is our config:

resource "newrelic_alert_channel" "slack" {
  count = var.environment == "prod" ? 1 : 0
  name  = "#********-alerts"
  type  = "slack"

  config {
    url     = "https://hooks.slack.com/services/*******/*********/**********"
    channel = "#***********-alerts"
  }
}

[dzadonskii]

Our config:

resource "newrelic_alert_channel" "opsgenie" {
  name = "${var.opsgenie_configuration.team_name} Opsgenie Hook ${var.name_suffix}"
  type = "webhook"
  lifecycle {
    ignore_changes = [
      "config"
    ]
  }
  config {
    base_url = "https://api.eu.opsgenie.com/v2/alerts"
    headers = {
      "Authorization" : "GenieKey ${var.opsgenie_configuration.api_key}"
    }
    payload_type   = "application/json"
    payload_string = <<EOF
    {
      "source": "New Relic",
      "priority": "P1",
      "message": "$CONDITION_NAME",
      "description": "$DESCRIPTION",
      "alias": "$EVENT_DETAILS",
      "responders": [
        {
          "name": "${var.opsgenie_configuration.team_name}",
          "type": "team"
        }
      ],
      "visibleTo": [
        {
          "name": "${var.opsgenie_configuration.team_name}",
          "type": "team"
        }
      ],
      "actions": [],
      "tags": ${jsonencode(var.tags)},
      "details": {
        "policyName": "$POLICY_NAME",
        "duration":"$DURATION",
        "state": "$EVENT_STATE",
        "runbook": "$RUNBOOK_URL",
        "severity": "$SEVERITY",
        "condition_name": "$CONDITION_NAME",
        "incidentUrl": "$INCIDENT_URL",
        "details":"$EVENT_DETAILS",
        "alertConditionId":"$CONDITION_ID",
        "target": "$TARGETS."
      }
    }
    EOF
  }
}