Terraform-provider-aws: Terraform bug report with AWS provider: []. Errors: [unexpected EOF]

[hashibot[bot]]

_This issue was originally opened by @moorichardmoo as hashicorp/terraform#17546. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Hi

Terraform versions:
Terraform v0.11.3

• provider.aws v1.10.0
• provider.template v1.0.0

Crash log located here: https://gist.githubusercontent.com/moorichardmoo/5fb92530c7ec33de93163ce8d8517130/raw/9bc64e80a41d5d4d9c7af8d49e53bea00fc4fcc5/gistfile1.txt

Steps to reproduce:
terraform plan -var-file="terraform.tfvars" -var-file="secret.tfvars"

Expected Behavior:
Terraform plan created

Actual Behavior:
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Steps to Reproduce:
terraform init --backend-config="backend.conf" -reconfigure -upgrade
terraform plan -var-file="terraform.tfvars" -var-file="secret.tfvars"

I've tried a full reboot, upgrading, removing recently added parts of my .tf files, and have tried digging through the crash log but I can't see anything that might help. If you have any ideas I'd be grateful to hear them.

Thanks

Richard

[bflad]

Relevant portion of crash log:

2018/03/09 16:48:21 [TRACE] dag/walk: walking "aws_security_group_rule.allow_s3_endpoint_443"
2018/03/09 16:48:21 [TRACE] vertex 'root.aws_security_group_rule.allow_s3_endpoint_443': walking
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: panic: Error reading level config: '' expected type 'string', got unconvertible type '[]interface {}'
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: 
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: goroutine 286 [running]:
2018/03/09 16:48:21 [TRACE] vertex 'root.aws_security_group_rule.allow_s3_endpoint_443': evaluating
2018/03/09 16:48:21 [TRACE] [walkRefresh] Entering eval tree: aws_security_group_rule.allow_s3_endpoint_443
2018/03/09 16:48:21 [TRACE] root: eval: *terraform.EvalSequence
2018/03/09 16:48:21 [TRACE] root: eval: *terraform.EvalInterpolate
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*ResourceData).get(0xc04234ae70, 0xc0428f77d0, 0x1, 0x1, 0xc0428f7712, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource_data.go:502 +0x4b3
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*ResourceData).getChange(0xc04234ae70, 0x2a87f13, 0x3, 0x1201, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource_data.go:473 +0x13c
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*ResourceData).diffChange(0xc04234ae70, 0x2a87f13, 0x3, 0x29b4f80, 0xc042b79140, 0x8004284b4c0, 0x30, 0x30)
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource_data.go:450 +0xac
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.schemaMap.diffString(0xc042585ce0, 0x2a87f13, 0x3, 0xc0425ace10, 0xc042995618, 0x4626b80, 0xc04234ae70, 0x0, 0x0, 0xcb)
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/schema.go:1158 +0xa7
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.schemaMap.diff(0xc042585ce0, 0x2a87f13, 0x3, 0xc0425ace10, 0xc0427a2ba0, 0x4626b80, 0xc04234ae70, 0x0, 0x2, 0x0)
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/schema.go:774 +0x4a6
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.schemaMap.Diff(0xc042585ce0, 0x0, 0xc042833740, 0x0, 0x267af60, 0xc0422dc500, 0x412763, 0x498b338, 0x10)
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/schema.go:424 +0x251
2018/03/09 16:48:21 [TRACE] root: eval: *terraform.EvalGetProvider
2018/03/09 16:48:21 [TRACE] root: eval: *terraform.EvalValidateResource
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Diff(0xc04258fce0, 0x0, 0xc042833740, 0x267af60, 0xc0422dc500, 0xc04284c701, 0x0, 0x80000000001)
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:252 +0x18e
2018-03-09T16:48:21.743Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).ReadDataDiff(0xc0423cc460, 0xc0426c2370, 0xc042833740, 0x1b0d50, 0x0, 0x26ac9c0)
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:413 +0xaa
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).ReadDataDiff(0xc0422bd880, 0xc0428f76e0, 0xc0428f7790, 0x0, 0x0)
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:593 +0x55
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: reflect.Value.call(0xc04203c9c0, 0xc04215b360, 0x13, 0x2a883e2, 0x4, 0xc04284bf20, 0x3, 0x3, 0xc04284bec8, 0xc04284bee8, ...)
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /usr/local/go/src/reflect/value.go:434 +0x90c
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: reflect.Value.Call(0xc04203c9c0, 0xc04215b360, 0x13, 0xc04284bf20, 0x3, 0x3, 0x0, 0x0, 0x0)
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /usr/local/go/src/reflect/value.go:302 +0xab
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: net/rpc.(*service).call(0xc042154200, 0xc042074640, 0xc042b5ea48, 0xc04213c600, 0xc0422bc7c0, 0x21d7d60, 0xc0428f76e0, 0x16, 0x21d7da0, 0xc0428f7790, ...)
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /usr/local/go/src/net/rpc/server.go:381 +0x149
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe: created by net/rpc.(*Server).ServeCodec
2018-03-09T16:48:21.744Z [DEBUG] plugin.terraform-provider-aws_v1.10.0_x4.exe:  /usr/local/go/src/net/rpc/server.go:475 +0x372

[moorichardmoo]

I have been playing around and have noticed that the problem doesn't occur if I target any of the resources with a terraform plan individually. Only if I run a terraform plan without the target flag does the crash occur.

[moorichardmoo]

I can also confirm it doesn't occur when running a terraform apply or a terraform destroy if targeting the resources. But it does occur when not targeting.

[moorichardmoo]

Update: If I target a single launch configuration the problem occurs.

[bflad]

Does this still occur on Terraform (core) 0.11.4 and AWS provider 1.11.0?

[moorichardmoo]

Thanks for the help, @bflad. I'll upgrade to 0.11.4 and try it.

[moorichardmoo]

Yes, I can confirm that the problem occurs with 0.11.4 too.

Thanks for alerting me to the new version, @bflad

[moorichardmoo]

I'm getting lots of connection is shut down and unexpected EOF.
I've tried it with -parallelism=1 but still that doesn't help.

[bflad]

Can you provide the portion of the crash log on Terraform 0.11.4 and AWS provider 1.11.0, similar to this comment please? https://github.com/terraform-providers/terraform-provider-aws/issues/3706#issuecomment-371917923 (basically the part near and after panic:

[moorichardmoo]

I've tracked the problem down to a single data resource, with this as its contents:

data "aws_lb" "internal" {
arn = "${data.terraform_remote_state.vpc.intarn}"
}

The problem seems to be that the intarn output an empty array.

intarn = []

So, two things:
1) Does anyone know how terraform can accept an empty array (and just turn it into an empty string) without erroring?
2) This should probably be able to be handled by terraform without crashing.

Thanks

[moorichardmoo]

hi @bflad sorry I've only just seen your comment. Sure, I'll get that.

[bflad]

With your additional information, I was able to reproduce this with a simple terraform_remote_state lookup: https://gist.github.com/bflad/fc5b29503bb0177633a475a92b88e0b4

Code reference: https://github.com/hashicorp/terraform/blob/47b178aee95d115fb366accd9c3d3f6cf393c07d/helper/schema/resource_data.go#L494-L503

Unfortunately this is some sort of bug upstream in Terraform core and will need to be addressed there. 🙁 I'll reopen the issue that got migrated here. Do you mind if I close this one unless you're aware of something wrong in particular with the AWS provider?

[moorichardmoo]

No that's great. Thanks very much @bflad

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!