Terraform-provider-aws: resource/aws_launch_template: Getting "unexpected EOF" and "connection is shut down" when applying

[yzargari]

Hello,
I'm trying to replace aws_launch_configuration with aws_launch_template on all my autoscaling groups. This means that I have to add ~150 launch templates.
The execution is failing on me every time I attempt it, unless I target just few resources (smaller plan).

The error's I'd usually get is unexpected EOF on the resource it failed to create and then connection is shut down on all the remaining resources.

I'm running Terraform 0.11.10 with the AWS provider version 1.50.0. The only theory I have is related to the length of user_data, which in aws_launch_template takes it based64encoded. Is it possible that the length of this user_data breaks the API request?

Thanks,
Yosi

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[bflad]

Hi @yzargari 👋 Sorry for the trouble. unexpected EOF errors indicate a code issue in the Terraform AWS provider code. We'll need some more information before we can do any troubleshooting.

Terraform should be creating a crash.log file in this scenario. Can you share that please (e.g. via a Gist)? If not, try searching for the stack trace starting with panic: in any other Terraform logs if possible. See also: https://www.terraform.io/docs/internals/debugging.html

[yzargari]

Thanks for your prompt response @bflad
It didn't create a crash.log but I did find the panic message: https://gist.github.com/yzargari/6fbbcea434405aa5ddfc51dc4f670e04

[yzargari]

Is there any workaround I can try? This is really blocking me right now and I'm willing to try any ugly hack...

[mskucherawy]

Experiencing the same problem. Also doesn't leave a crash.log anywhere obvious.

Did a run with trace-level logging and this was in the output (which is enormous so I won't attach the whole thing here) right after a "refreshing state" line for an aws_s3_bucket resource, which I guess completed as this one seems to come through the transit gateway route code:

panic: runtime error: index out of range
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4:
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: goroutine 3693 [running]:
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsEc2TransitGatewayRouteRead(0xc005699960, 0x2ef7b80, 0xc0039b4a80, 0xc005699960, 0x0)
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_ec2_transit_gateway_route.go:128 +0x8e6
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(Resource).Refresh(0xc00595e690, 0xc004b9f9f0, 0x2ef7b80, 0xc0039b4a80, 0xc00311d138, 0x4bff01, 0x296c220)
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:352 +0x160
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(Provider).Refresh(0xc002bd33b0, 0xc004b9f9a0, 0xc004b9f9f0, 0xc002028000, 0xc00013abf0, 0x7f8424042d80)
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x92
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(ResourceProviderServer).Refresh(0xc00318e660, 0xc005da4f30, 0xc005da5000, 0x0, 0x0)
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: reflect.Value.call(0xc0003d5020, 0xc004127d88, 0x13, 0x34d8b81, 0x4, 0xc00013af18, 0x3, 0x3, 0xc0003d0480, 0xc005a4cf40, ...)
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/goenv/versions/1.11.1/src/reflect/value.go:447 +0x449
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: reflect.Value.Call(0xc0003d5020, 0xc004127d88, 0x13, 0xc00097c718, 0x3, 0x3, 0x5d46db, 0xc00097c728, 0x429732)
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/goenv/versions/1.11.1/src/reflect/value.go:308 +0xa4
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: net/rpc.(service).call(0xc0020b8ec0, 0xc0036d2140, 0xc003b8e240, 0xc003b8e250, 0xc00023c680, 0xc0009370e0, 0x296c1e0, 0xc005da4f30, 0x16, 0x296c220, ...)
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/goenv/versions/1.11.1/src/net/rpc/server.go:384 +0x14e
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: created by net/rpc.(*Server).ServeCodec
2019-01-08T16:27:34.244-0800 [DEBUG] plugin.terraform-provider-aws_v1.54.0_x4: /opt/goenv/versions/1.11.1/src/net/rpc/server.go:481 +0x47e

Everything after that is "unexpected EOF" mixed with the general appearance of the program unwinding and terminating for about a thousand trace log lines.

[bflad]

@mskucherawy can you please open a separate bug report for that panic? Its in a different resource and will require separate triage. Thanks.

[mskucherawy]

Wilco. (You're right, sorry, noticed this after I posted it.)

[bflad]

@yzargari 👋 Hi again. Sorry for the delayed response. Did you eventually figure out this issue? From the code lines in the panic report, it appears to be related to an "empty" configuration block for iam_instance_profile, e.g.

resource "aws_launch_template" "example" {
  # ... other configuration ...
  iam_instance_profile {}
}

That configuration block should be omitted if you do not wish to configure IAM instance profiles with the EC2 Launch Template or should contain either an arn or name configuration inside of it, e.g.

resource "aws_launch_template" "example" {
  # ... other configuration ...
  iam_instance_profile {
    name = "${aws_iam_instance_profile.example.name}"
  }
}

I have submitted a pull request for the resource to skip these empty configurations instead of crashing (Terraform will report a perpetual difference for the attribute until the issue is resolved with the configuration): #7134

[bflad]

The crash fix for the above mentioned issue with iam_instance_profile has been merged and will release with version 1.56.0 of the Terraform AWS provider, likely middle of this week. 👍

[bflad]

This has been released in version 1.56.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!