Terraform: Terraform crashes running in Docker

[steadysupply]

Terraform Version

/ # terraform --version
Terraform v0.12.29
+ provider.aws v3.0.0
+ provider.null v2.1.

Terraform Configuration Files

Probably irrelevant in this case (see Additional Context below), there's nothing fancy going on.

Debug/Crash Output

https://gist.github.com/steadysupply/3ab2d98bb5a69a425507d31374a5dad8
The above contains crash.log and the stdout/stderr files produced by the invocation

/ # TF_LOG=trace aws-profile -p abc-xyz terraform apply /mnt/repo/terraform > stdout 2
> stderr

. These files are encrypted with GPG using [email protected] pubkey.

Expected Behavior

Normal apply run

Actual Behavior

It crashed!

Steps to Reproduce

Run apply

Additional Context

We're running Terraform in the root of a Docker container where the repository containing our templates gets mounted read only at /mnt/repo, a named volume (read/write) is mounted at /.terraform to cache providers and plugins (OS is Ubuntu 20.04 with Docker 19.03.8). We're also using the Python script aws-profile to set AWS credentials in the environment (see invocation above).

[jbardin]

Hi @steadysupply,

Thanks, this appears to be a crash in the AWS provider. The relevant stack trace from the log is:

 panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x37f72e7]

 goroutine 61 [running]:
 github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsAcmCertificateCheckValidationRecords(0xc0016e0710, 0x1, 0x1, 0xc0010ef9e0, 0xc0002126c8, 0x1, 0x4b10100)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_acm_certificate_validation.go:138 +0x197
 github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsAcmCertificateValidationCreate(0xc0004bf570, 0x4c948a0, 0xc0003af900, 0x2, 0x94466a0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_acm_certificate_validation.go:61 +0x3bd
 github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).Apply(0xc0002aabd0, 0xc0013956d0, 0xc0013d8900, 0x4c948a0, 0xc0003af900, 0x4b10101, 0xc0013992b8, 0xc0013de510)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:310 +0x365
 github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Apply(0xc00021c180, 0xc0019d1a10, 0xc0013956d0, 0xc0013d8900, 0xc0013daf28, 0xc00000f2e0, 0x4b124a0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/provider.go:294 +0x99
 github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ApplyResourceChange(0xc000213c48, 0x6238aa0, 0xc0013d3b00, 0xc0004bf0a0, 0xc000213c48, 0xc0013d3b00, 0xc0013b3b78)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:885 +0x8b4
 github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0x56592e0, 0xc000213c48, 0x6238aa0, 0xc0013d3b00, 0xc001992480, 0x0, 0x6238aa0, 0xc0013d3b00, 0xc001640b40, 0x1d7)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3305 +0x217
 google.golang.org/grpc.(*Server).processUnaryRPC(0xc000eb5380, 0x62579e0, 0xc00065c300, 0xc0013ac300, 0xc000b20810, 0x940aca0, 0x0, 0x0, 0x0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1024 +0x501
 google.golang.org/grpc.(*Server).handleStream(0xc000eb5380, 0x62579e0, 0xc00065c300, 0xc0013ac300, 0x0)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1313 +0xd3d
 google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000041800, 0xc000eb5380, 0x62579e0, 0xc00065c300, 0xc0013ac300)
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:722 +0xa1
 created by google.golang.org/grpc.(*Server).serveStreams.func1
    /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:720 +0xa1

[hashibot[bot]]

This issue has been automatically migrated to terraform-providers/terraform-provider-aws#14542 because it looks like an issue with that provider. If you believe this is _not_ an issue with the provider, please reply to terraform-providers/terraform-provider-aws#14542.

[gustavobertolino]

Same thing here, but running in a local machine

`Terraform v0.12.21

• provider.archive v1.3.0
• provider.aws v3.1.0
• provider.kubernetes v1.12.0
• provider.local v1.4.0
• provider.null v2.1.2
• provider.random v2.3.0
• provider.template v2.1.2
  `

All the errors logged indicated that is a problem in the aws_provider

[gustavobertolino]

Rolling back aws_provider to version 3.0.0 works for me

https://registry.terraform.io/providers/hashicorp/aws/3.0.0

[steadysupply]

Hello, after some further thought and browsing the sources for terraform-provider-aws we have concluded that our Terraform templates (which were thought to be irrelevant to the issue) are the root cause -- this isn't anything to do with running in Docker!

Will update https://github.com/terraform-providers/terraform-provider-aws/issues/14542 with further details shortly.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.