Terraform-provider-newrelic: 2.8.0 is getting 500 errors from API
API calls to PUT dashboard leading to 500 Internal Server Error
Terraform Version
v0.12.26
Affected Resource(s)
newrelic_dashboard at least
alx13
All 15 comments
And now I'm getting different error:
2020-10-01T12:24:33.928+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: time=newrelic_dashboard.dashboard: Refreshing state... [id=1431069]
"2020-10-01T12:24:33+02:00" level=debug msg="performing request" method=POST url="https://api.newrelic.com/graphql"
2020-10-01T12:24:33.928+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: time="2020-10-01T12:24:33+02:00" level=debug msg="performing request" method=POST url="https://api.newrelic.com/graphql"
2020-10-01T12:24:33.928+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: 2020/10/01 12:24:33 [INFO] Reading New Relic entities
2020-10-01T12:24:33.928+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: time="2020-10-01T12:24:33+02:00" level=debug msg="performing request" method=POST url="https://api.newrelic.com/graphql"
2020-10-01T12:24:33.931+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: 2020/10/01 12:24:33 [INFO] Reading New Relic dashboard 1431069
2020-10-01T12:24:33.931+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: time="2020-10-01T12:24:33+02:00" level=debug msg="performing request" method=GET url="https://api.newrelic.com/v2/dashboards/1431069.json"
Warning: `newrelic_alert_condition` has been deprecated. Use `newrelic_nrql_alert_condition` instead.
on alerts.tf line 11, in resource "newrelic_alert_condition" "apdex_score":
11: resource "newrelic_alert_condition" "apdex_score" {
(and 3 more similar warnings elsewhere)
Error: rpc error: code = Unavailable desc = transport is closing
Error: rpc error: code = Unavailable desc = transport is closing
Error: rpc error: code = Unavailable desc = transport is closing
Error: rpc error: code = Unavailable desc = transport is closing
2020-10-01T12:24:35.128+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0:
2020-10-01T12:24:35.128+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: goroutine 81 [running]:
2020-10-01T12:24:35.128+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: github.com/newrelic/terraform-provider-newrelic/v2/newrelic.flattenWidgets(0xc0003001e0, 0xc0000da000, 0xc000933958, 0xc0006d75a0, 0x2b4fc20)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/work/terraform-provider-newrelic/terraform-provider-newrelic/newrelic/structures_newrelic_dashboard.go:365 +0x3b8
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: github.com/newrelic/terraform-provider-newrelic/v2/newrelic.flattenDashboard(0xc000300100, 0xc0000da000, 0xc000300100, 0x0)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/work/terraform-provider-newrelic/terraform-provider-newrelic/newrelic/structures_newrelic_dashboard.go:332 +0x392
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: github.com/newrelic/terraform-provider-newrelic/v2/newrelic.resourceNewRelicDashboardRead(0xc0000da000, 0x1d55700, 0xc0003d86f0, 0xc0000da000, 0x0)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/work/terraform-provider-newrelic/terraform-provider-newrelic/newrelic/resource_newrelic_dashboard.go:387 +0x1ca
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0003033b0, 0xc0004e6230, 0x1d55700, 0xc0003d86f0, 0xc00000e258, 0x0, 0x0)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/helper/schema/resource.go:460 +0x119
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc000288b90, 0x2227080, 0xc0002b9cb0, 0xc000680420, 0xc000288b90, 0xc0002b9cb0, 0xc000640ba0)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/internal/helper/plugin/grpc_provider.go:525 +0x3d8
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ReadResource_Handler(0x1ecbf40, 0xc000288b90, 0x2227080, 0xc0002b9cb0, 0xc0006803c0, 0x0, 0x2227080, 0xc0002b9cb0, 0xc000118a00, 0x263)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/internal/tfplugin5/tfplugin5.pb.go:3269 +0x217
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003c3500, 0x22349e0, 0xc0006c2c00, 0xc000233300, 0xc000614510, 0x2b14130, 0x0, 0x0, 0x0)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1171 +0x50a
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: google.golang.org/grpc.(*Server).handleStream(0xc0003c3500, 0x22349e0, 0xc0006c2c00, 0xc000233300, 0x0)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1494 +0xccd
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0000342f0, 0xc0003c3500, 0x22349e0, 0xc0006c2c00, 0xc000233300)
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:834 +0xa1
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-10-01T12:24:35.129+0200 [DEBUG] plugin.terraform-provider-newrelic_v2.8.0: /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:832 +0x204
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020-10-01T12:24:35.132+0200 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalReadData, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/10/01 12:24:35 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
Hi @alx13, thanks for reporting. In order to debug and fix this, we need to see your your configuration (sensitive data redacted). While I can clearly see where an error is occurring in your log output, it's impossible to know what exact scenario is causing it without seeing some of your configuration. Once we get that, we can debug and get a fix out asap.
sanderblue
on 1 Oct 2020
I'm getting the same 500 error as well on newrelic_dashboard resources on v2.11.0. Since the provider wants to re-order all the widgets for the nth time, I'd suspect this time around, the API/provider no longer accepts a particular value or wants a different value.
@sanderblue Are you in contact with Newrelic support? Since our dashboard configuration is quite long and redacting it might skew your debugging, I could open a support ticket so that you can debug further.
davidji99
on 7 Oct 2020
Hi @davidji99, thanks letting us know you're seeing the issue too. Just to double check, does this still happen on the latest version? Also, if it's possible to share one or two of your dashboard resource configurations, that would be helpful. We haven't been able to reproduce this locally yet, so having a dashboard config or two from your scenario could help us pinpoint what's causing this.
sanderblue
on 7 Oct 2020
Yeah, it still happening on v2.11.0.
A redacted and shortened version of the configuration is as follows:
resource "newrelic_dashboard" "foobar" {
title = "Some title"
icon = "users"
visibility = "all"
editable = "editable_by_all"
widget {
width = 1
height = 1
column = 1
row = 1
title = ""
visualization = "markdown"
source = "Some Source"
}
widget {
width = 1
height = 1
column = 1
row = 2
facet = "host"
limit = 10
duration = 7776000000
entity_ids = [123456789]
title = "Some title"
visualization = "metric_line_chart"
order_by = "max_response_time"
metric {
name = "WebFrontend/QueueTime"
values = [
"average_response_time",
]
}
}
dynamic "widget" {
for_each = {"example" = "map"}
content {
width = 1
height = 1
column = 1 + index(keys(data.newrelic_application.foobar), widget.key)
row = 9
title = "some title"
visualization = "faceted_line_chart"
nrql = "some nrql"
}
}
}
Thanks, I'll dig into this and report back.
sanderblue
on 7 Oct 2020
I'm still not able to reproduce this, even with the dynamic widget and I kind of had to guess regarding the dynamic widget for_each because 1 + index(keys(data.newrelic_application.foobar), widget.key) throws an error from Terraform core.
I made some adjustment to get past the Terraform error and the dashboard was created. I tried on multiple versions of Terraform as well - from 0.12.20 to 0.12.29 and also in 0.13+.
That being said, does updating to the latest Terraform version (0.13.4) work for anyone?
It would be helpful if we could get a minimal example including the necessary data source(s), and the actual NRQL queries being used. It would also be helpful to get a working representation of the dynamic widget block so we can fully understand whats happening. A 500 error is a server error and 99% of the time that's not a Terraform issue. It could be any number things.
We really want to help, but if we can't reproduce the issue by guessing what the configurations should look like, we'll need to see a minimal "working" example.
sanderblue
on 8 Oct 2020
Hi Sander!
Can you please try to create a dashboard with previous version of provider
and then use the new one?
On Thu, 8 Oct 2020 at 20:04, Sander Blue notifications@github.com wrote:
I'm still not able to reproduce this, even with the dynamic widget and I
kind of had to guess regarding the dynamic widget for_each because 1 +
index(keys(data.newrelic_application.foobar), widget.key) throws an error
from Terraform core.I made some adjustment to get past the Terraform error and the dashboard
was created. I tried on multiple versions of Terraform as well - from
0.12.20 to 0.12.29 and also in 0.13+.That being said, does updating to the latest Terraform version (0.13.4)
work for anyone?It would be helpful if we could get a minimal example including the
necessary data source(s), and the actual NRQL queries being used. It would
also be helpful to get a working representation of the dynamic widget block
so we can fully understand whats happening. A 500 error is a server error
and 99% of the time that's not a Terraform issue. It could be any number
things.We really want to help, but if we can't reproduce the issue by guessing
what the configurations should look like, we'll need to see a minimal
"working" example.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/newrelic/terraform-provider-newrelic/issues/923#issuecomment-705734160,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAMAAABVNACMH7QOBR6CZV3SJX5LRANCNFSM4SAHMWBA
.
alx13
on 8 Oct 2020
I faced the same 500 error. I may provide details if needed.
aandrusyk-amplify
on 8 Oct 2020
@alx13
As a workaround, I destroyed all my dashboards applying terraform with terraform-provider-newrelic plugin of version 2.7.5 to configs where dashboards were removed, then removed .terraform folder (I am on Linux), then reinited terraform config (the latest terraform-provider-newrelic plugin was downloaded), and reapplied terraform config creating dashboards back.
It is applicable if your terraform state is on s3 bucket or elsewhere not in the .terraform forlder.
aandrusyk-amplify
on 8 Oct 2020
Thanks for mentioning the workaround. Destroying and re-applying certainly works, but it's not ideal. Either way, the mention of the workaround is appreciated. We're still working on how to resolve the bug itself though. It's a tricky balance between Terraform internals and API behavior.
sanderblue
on 9 Oct 2020
@sanderblue - Our team just got hit by this as well :(. I can provide more details if you still need samples to reproduce the error reliably.
onlyplo
on 13 Oct 2020
Hi @onlyplo, we're working on a solution now :)
sanderblue
on 13 Oct 2020
This was fixed in v2.12.1. Please let us know if upgrading to v2.12.1 works 🙂
sanderblue
on 16 Oct 2020
I can confirm it's working. Changes to our NR dashboards are no longer failing during our pipeline builds. Many thanks @sanderblue !!!
onlyplo
on 20 Oct 2020
Related issues
ryandzink
·
6Comments
jpd4dazn
·
3Comments
rdhar
·
6Comments
MAN98
·
3Comments
pickgr
·
3Comments
Most helpful comment
Hi @onlyplo, we're working on a solution now :)