Terraform-provider-newrelic: Error unmarshaling `newrelic_alert_channel` configuration headers after release v1.12.0

[ahornerr]

Hi there,

It appears that the attempted forwards-compatible change for deprecating configuration in the newrelic_alert_channel has broken unmarshaling of configuration headers in the PR #307, release https://github.com/terraform-providers/terraform-provider-newrelic/releases/tag/v1.12.0

Terraform Version

0.12.6

Affected Resource(s)

• newrelic_alert_channel

Terraform Configuration Files

Working on v1.11.0, broken on v1.12.0

resource "newrelic_alert_channel" "slack_warning" {
  name = "Some slack channel"
  type = "slack"

  configuration = {
    channel = "#some-slack-channel"
    url     = var.slack_url
  }

  // https://github.com/terraform-providers/terraform-provider-newrelic/issues/5
  lifecycle {
    ignore_changes = ["configuration"]
  }
}

Expected Behavior

It appears as though there was an attempt to add forwards compatible support to v1.12.0 for the configuration object in order to support migration from configuration = { to config { so I would expect that to work with a deprecation warning.

Actual Behavior

While using the above Terraform configuration on v1.12.0 of the NR TF provider, I receive the following error running terraform plan. The deprecated warning is expected and we are working through the migration but the JSON unmarshal error is not expected:

Warning: "configuration": [DEPRECATED] use `config` block instead

  on .terraform/modules/newrelic-alerts/newrelic-alerts/alert_channels.tf line 1, in resource "newrelic_alert_channel" "slack_warning":
   1: resource "newrelic_alert_channel" "slack_warning" {

Error: json: cannot unmarshal string into Go struct field ChannelConfiguration.channels.configuration.headers of type map[string]string

Steps to Reproduce

1. terraform plan with the above example Terraform configuration and v1.12.0 of the TF NR provider

Important Factoids

It also appears that config is not backwards compatible before v1.12.0 which makes this a hard cutover (and potentially a breaking change for other consumers)

References

• #307

[sanderblue]

Hi there! Thanks for reporting. So far I'm unable to reproduce, but do you happen to have other channel resource blocks in your HCL such as a webhook? The field ChannelConfiguration.channels.configuration.headers is part of the webhook config arguments.

[ahornerr]

@sanderblue Thanks for responding so quickly! The Slack alert channel has a URL which it uses for a webhook but we supply it as newrelic_alert_channel.configuration.url as shown in my example above.

We don't appear to use any instances of the webhook resource block in this module.

We actually have 4 alert channels setup in this module configuration which I had omitted 3 from my initial example. There are two Slack, one Email, and one Victor Ops. I've created a gist with those 4 configuration blocks as well as the Terraform for those (with sensitive values marked as *****): https://gist.github.com/ahornerr/6250e226ca4e176c119530453afb488b

After some further debugging, with that above configuration, it appears that we get the JSON unmarshal error for every newrelic_alert_channel resource

Error: json: cannot unmarshal string into Go struct field ChannelConfiguration.channels.configuration.headers of type map[string]string

Error: json: cannot unmarshal string into Go struct field ChannelConfiguration.channels.configuration.headers of type map[string]string

Error: json: cannot unmarshal string into Go struct field ChannelConfiguration.channels.configuration.headers of type map[string]string

Error: json: cannot unmarshal string into Go struct field ChannelConfiguration.channels.configuration.headers of type map[string]string

Please let me know if there's anything additional I can provide to help with debugging. Thanks!

[sanderblue]

No problem. Thanks for the additional information. I was able to reproduce running the terraform destroy command. We'll continue debugging and will report back. Appreciate the cooperation :)

[ahornerr]

Thanks so much!

[ahornerr]

@sanderblue After a bit more testing on our end it appears that the issue doesn't surface when there is no existing Terraform state (or perhaps no existing alert channels in New Relic). Hopefully this helps!

[sanderblue]

Thanks. Ran into the same thing as well. For some reason, I was only able to reproduce it that one time. I haven't received an error after trying many different sequences of events and configurations.

[sanderblue]

Ok, I was able to reproduce the issue again and can continue debugging. This time with terraform apply (last time I reproduced was on terraform destroy).

1. Created an alert in the UI.
2. Created a terraform config with matching information from the UI configured channel.
3. terraform apply
4. Result: Error: json: cannot unmarshal string into Go struct field ChannelConfiguration.channels.configuration.payload of type map[string]string

Not the exact same error as you're receiving, but configuration.payload behaves similarly, so I should be able to figure out a fix for this.

[ahornerr]

This sounds really similar to our use case. Curious what's happening but looking forward to a fix. Thanks for digging into it!

[sanderblue]

So the issue resides in the client newrelic-client-go. We'll need a custom unmarshall method for the Header and Payload fields in the ChannelConfiguration struct. Hoping we have a fix for this released early next week.

[pneigel-ca]

I also encountered this error even once I updated the configuration reference to config in my module. I no longer encounter the deprecation warning, but still see the unmarshal error.

Error: json: cannot unmarshal string into Go struct field ChannelConfiguration.channels.configuration.headers of type map[string]string

[sanderblue]

The fix for this has been merged in the upstream client newrelic-client-go PR #102. We will be looking to release an update to the Terraform provider within the next day or so.

[sanderblue]

@ahornerr please give this a shot again and let us know if you run into any problems. This was fixed with the v1.12.1 release 😎Going to close this ticket, but if you this is still an issue, we can reopen.

[sanderblue]

Closing as this v1.12.1 release introduced the fix for this.

[pneigel-ca]

I am still receiving the error, using terraform version 12.19.

• manually removed .terraform
• updated provider constraint to 1.12.1
• verified the latest provider is being used

The output of _terraform providers_ :

├── provider.newrelic 1.12.1
├── module.health_check
│   └── provider.newrelic (inherited)
├── module.health_condition
│   └── provider.newrelic (inherited)
├── module.policy
│   └── provider.newrelic (inherited)
└── module.slack
└── provider.newrelic (inherited)

Let me know if there are any other steps I can try. For full transparency this is the full error string.

Error: json: cannot unmarshal array into Go struct field ChannelConfiguration.channels.configuration.payload of type string

on .terraform/modules/slack/newrelic/nr-channel-slack/main.tf line 1, in data "newrelic_alert_channel" "slack": 1: data "newrelic_alert_channel" "slack" {

[sanderblue]

@pneigel-ca, Hmm, can you provide your terraform config so we can debug?

[pneigel-ca]

@sanderblue I posted the two relevant modules and instance main.tf in pastebin

This is of course running against an existing tfstate because I am managing resources that already exist.

[sanderblue]

@pneigel-ca Thanks, any chance you can share what your current tfstate looks like? (sensitive information removed)

[pneigel-ca]

@sanderblue thanks for the quick responses on this. Here is my tfstate.

[sanderblue]

@pneigel-ca One more request, can you go to the New Relic v2 API explorer for Alerts Channels and use the "List" request for all your channels and share the response like you did the others? Much appreciated.

We're trying to figure out why your error message states "json: cannot unmarshal _array_ into Go struct field". We're wondering if the New Relic API is returning a slightly different response than expected. Also seems odd that you're getting this message as well "configuration.payload of type _string_". It shouldn't be of type string, so we're wondering why that's happening.

[pneigel-ca]

@sanderblue great point. Thanks for the explanation. Note the id for the slack channel we are using is 2728850.

Pastebin of list alert channels output.

[sgsiebers]

I know I didn't file this ticket, but I appear to be experiencing the same problem as of provider 1.12.0/1. I get the following error trying to deserialize an email alert channel (which of course doesn't have a payload attribute).

json: cannot unmarshal number into Go struct field ChannelConfiguration.channels.configuration.payload of type string

We have a very large, complex configuration so I can't paste all the details, but the relevant section of the alert channel JSON response from our API calls look perfectly normal:

{
"id": 2306972,
"name": "Name Redacted",
"type": "email",
"configuration": {
"include_json_attachment": "true",
"recipients": "[email protected]"
},
"links": {
"policy_ids": [
424998,
424997,
424890
]
}

[sanderblue]

Fairly certain we know what the issue is now. Still need to confirm 100% on this, but the issue stems from the channels that contain a payload that has any additional nesting in the JSON. Since terraform needs to get a list of your channels, any channel you have can causes this issue, so the error message is a bit misleading 😞

The structs expect payload to be something such as

{ "some_string_key": "a_string_value"}

but some of the webhook channels have a payload that is more complex with nested objects or arrays such as

{ 
  "string_key": [ "oops_an_array"] 
}

and/or

{ 
  "string_key": {
    "oops": "a nested object"
  } 
}

So we'll need to make an additional update. Sorry for inconvenience, but we'll try to get a fix out asap for this. In the meantime, it might be best to stick to the provider version v1.11.0 since it was working for you at that point. We'll get back to you once the next update is released.

[sgsiebers]

That sounds right to me. I was just noticing that this works for my prod account (where the webhook payloads don't have any nested JSON), but fails in our non-prod where we do have a payload with nested JSON.

[pgrefviau]

Same problem as @sgsiebers : getting the following error while trying to refresh the state on an existing alert channel of type "pagerduty":

newrelic_alert_channel.pagerduty: json: cannot unmarshal bool into Go struct field ChannelConfiguration.channels.configuration.payload of type string

Here's the JSON payload I'm getting when inspecting the traffic of the relevant channel's page in the New Relic web UI:

{
"id":2894745,
    "policyIds":[588314,580207],
    "name":"redacted",
    "type":"PAGERDUTY",
    "configuration": {"service_key":"redacted"},
    "severityThreshold":3,
    "accountId":0000,
    "familyId":3287593,
    "createdAt":1575487261077,
    "createdBy":"RPM_V2_API",
    "updatedAt":1576366050419,
    "updatedBy":"2168903"
}

[sanderblue]

We should have v1.12.2 released relatively soon. If not this evening, then Monday morning for sure. Thank you for your patience.