Terraform: Terraform 12 crashes with Error: pipe: too many open files
Terraform Version
terraform -v
Terraform v0.12.17
+ provider.aws v2.41.0
+ provider.null v2.1.2
+ provider.template v2.1.2
terragrunt -v
terragrunt version v0.21.7
Terraform Configuration Files
My repository contains 200+ AWS EC2 instances defined using a shared AWS server module which uses chef, local-exec and remote-exec provisioners.
Module structure is:
resource "aws_instance" "aws_server" {
..
..
lifecycle {
ignore_changes = [
..
]
}
connection {
type = "ssh"
..
}
provisioner "local-exec" {
..
}
provisioner "remote-exec" {
..
connection {
type = "ssh"
host = self.private_ip
}
}
provisioner "chef" {
..
connection {
type = "ssh"
host = self.private_ip
}
}
provisioner "local-exec" {
..
}
provisioner "local-exec" {
..
}
provisioner "remote-exec" {
..
connection {
type = "ssh"
host = self.private_ip
}
}
}
Debug Output
..
..
2019/12/09 12:33:50 [TRACE] dag/walk: upstream of "module.PATH.NAME.provider.null (close)" errored, so skipping
2019/12/09 12:33:50 [TRACE] dag/walk: upstream of "provisioner.local-exec (close)" errored, so skipping
2019/12/09 12:33:50 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2019-12-09T12:33:50.330-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52421
2019-12-09T12:33:50.330-0600 [DEBUG] plugin: plugin exited
2019/12/09 12:33:50 [TRACE] [walkValidate] Exiting eval tree: provider.aws (close)
2019/12/09 12:33:50 [TRACE] vertex "provider.aws (close)": visit complete
2019/12/09 12:33:50 [TRACE] dag/walk: upstream of "root" errored, so skipping
Warning: The -var and -var-file flags are not used in validate. Setting them has no effect.
These flags will be removed in a future version of Terraform.
Error: pipe: too many open files
Error: fork/exec /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4: too many open files
Error: pipe: too many open files
Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: error while dialing: dial unix /var/folders/qg/dr4z0hq9249f7y4590ckyc7m0000gp/T/plugin031665962: socket: too many open files"
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: fork/exec /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4: too many open files
Error: pipe: too many open files
Error: pipe: too many open files
Error: Error in function call
on infrastructure/infrastructure.tf line 6, in resource "aws_key_pair" "terraform":
6: public_key = file("${path.module}/../lib/terraform.pub")
|----------------
| path.module is "infrastructure"
Call to function "file" failed: failed to read lib/terraform.pub.
Error: Error in function call
on infrastructure/name/name.tf line 2, in data "template_file" "name":
2: template = file("${path.module}/name.json.tpl")
|----------------
| path.module is "infrastructure/name"
Call to function "file" failed: failed to read
infrastructure/name/name.json.tpl.
Error: Error in function call
on infrastructure/name/name.tf line 11, in data "template_file" "name":
11: template = file("./${path.module}/name.json.tpl")
|----------------
| path.module is "infrastructure/name"
Call to function "file" failed: failed to read
infrastructure/name/name.json.tpl.
Error: Error in function call
on publishing/name/iam.tf line 18, in resource "aws_iam_group_policy" "name":
18: policy = file(
19:
20:
|----------------
| path.module is "publishing/name"
Call to function "file" failed: failed to read
publishing/name/policies/name.json.
2019-12-09T12:33:50.371-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52426
2019-12-09T12:33:50.371-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.371-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52477
2019-12-09T12:33:50.371-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.372-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52450
2019-12-09T12:33:50.373-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52491
2019-12-09T12:33:50.373-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52422
2019-12-09T12:33:50.373-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.373-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.373-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.382-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52447
2019-12-09T12:33:50.382-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.382-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52409
2019-12-09T12:33:50.382-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.388-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52454
2019-12-09T12:33:50.388-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.390-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52483
2019-12-09T12:33:50.390-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52434
2019-12-09T12:33:50.390-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.390-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.390-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52442
2019-12-09T12:33:50.390-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52480
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52448
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52443
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52459
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52482
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.390-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52416
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52481
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52476
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52458
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52472
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52432
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52487
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52420
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52471
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52438
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52484
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52465
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52436
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52415
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52419
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-template_v2.1.2_x4 pid=52425
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52428
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52427
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52433
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.401-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52452
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52414
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52441
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52494
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52460
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52451
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52423
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52440
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52413
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52479
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52473
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52412
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52490
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52469
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52466
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52485
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52492
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52489
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52486
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52461
2019-12-09T12:33:50.402-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52439
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52474
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52430
2019-12-09T12:33:50.419-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52493
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.419-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4 pid=52470
2019-12-09T12:33:50.419-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.419-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.418-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52429
2019-12-09T12:33:50.419-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.419-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52424
2019-12-09T12:33:50.419-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52478
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52488
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=52463
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=52410
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52435
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52449
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52418
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52417
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=52445
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52467
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:50.420-0600 [DEBUG] plugin: plugin exited
2019-12-09T12:33:52.339-0600 [WARN] plugin: plugin failed to exit gracefully
2019-12-09T12:33:52.342-0600 [DEBUG] plugin: plugin process exited: path=/Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4 pid=52495 error="signal: killed"
[terragrunt] [/Users/NAME/ss/infra_terraform] 2019/12/09 12:33:56 Module /Users/NAME/ss/infra_terraform has finished with an error: Hit multiple errors:
exit status 1
[terragrunt] 2019/12/09 12:33:56 Encountered the following errors:
Hit multiple errors:
exit status 1
Actual Behavior
terraform spawns around 81 terraform provider threads before crashing with too many open files
$ ps uaxw |grep terraform | wc -l
81
$ ps uaxw |grep terraform
NAME 49716 217.1 1.6 4703732 273728 s000 S+ 12:26PM 0:16.24 /usr/local/bin/terraform validate -var-file=/Users/NAME/ss/infra_terraform/remote_state.tfvars
NAME 50244 47.5 0.2 4541336 34316 s000 S+ 12:26PM 0:00.26 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50291 31.4 0.2 4540044 30272 s000 R+ 12:26PM 0:00.14 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50219 29.5 0.2 4541080 33540 s000 S+ 12:26PM 0:00.25 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50257 28.7 0.2 4541068 30012 s000 S+ 12:26PM 0:00.14 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50173 14.0 0.2 4540568 31668 s000 R+ 12:26PM 0:00.19 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50149 6.8 0.2 4541860 33500 s000 S+ 12:26PM 0:00.27 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50187 6.0 0.2 4540044 30028 s000 S+ 12:26PM 0:00.14 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50111 5.1 0.2 4541080 36280 s000 S+ 12:26PM 0:00.27 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50148 4.6 0.2 4541604 34936 s000 S+ 12:26PM 0:00.26 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50112 3.9 0.2 4540300 33736 s000 S+ 12:26PM 0:00.25 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50118 3.8 0.2 4542616 33380 s000 S+ 12:26PM 0:00.27 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49715 2.5 0.1 4430492 15004 s000 S+ 12:26PM 0:00.74 terraform validate -var-file=/Users/NAME/ss/infra_terraform/remote_state.tfvars
NAME 50147 1.9 0.2 4541604 32988 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50320 1.3 0.2 4537960 25716 s000 R+ 12:26PM 0:00.06 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50106 1.1 0.2 4541336 33812 s000 S+ 12:26PM 0:00.24 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50108 0.4 0.2 4542704 35452 s000 S+ 12:26PM 0:00.26 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49917 0.0 0.2 4541336 34096 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49914 0.0 0.2 4540824 34104 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49913 0.0 0.1 4395804 14280 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49912 0.0 0.2 4541604 34660 s000 S+ 12:26PM 0:00.24 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49911 0.0 0.2 4542628 34376 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49910 0.0 0.1 4395804 14360 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49909 0.0 0.1 4396920 14496 s000 S+ 12:26PM 0:00.10 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-template_v2.1.2_x4
NAME 49879 0.0 0.1 4396316 14404 s000 S+ 12:26PM 0:00.10 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49874 0.0 0.2 4540824 33048 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49873 0.0 0.1 4395624 14440 s000 S+ 12:26PM 0:00.10 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49870 0.0 0.2 4541348 33176 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49869 0.0 0.2 4540568 33344 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49868 0.0 0.2 4540824 33868 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49867 0.0 0.1 4439708 16308 s000 S+ 12:26PM 0:00.02 /usr/local/bin/terraform internal-plugin provisioner remote-exec
NAME 49866 0.0 0.1 4429968 15192 s000 S+ 12:26PM 0:00.03 /usr/local/bin/terraform internal-plugin provisioner remote-exec
NAME 50325 0.0 0.0 4268716 848 s003 R+ 12:26PM 0:00.00 grep terraform
NAME 50107 0.0 0.2 4541860 34272 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50105 0.0 0.1 4396060 14404 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50104 0.0 0.1 4395804 14408 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50103 0.0 0.1 4396332 14284 s000 S+ 12:26PM 0:00.10 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-template_v2.1.2_x4
NAME 50102 0.0 0.2 4540568 33504 s000 S+ 12:26PM 0:00.24 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50101 0.0 0.2 4541348 33596 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50100 0.0 0.1 4396316 14304 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50090 0.0 0.2 4541860 33084 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50064 0.0 0.2 4540824 33948 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50063 0.0 0.2 4540824 33576 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50061 0.0 0.1 4395804 14352 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50059 0.0 0.1 4438916 16184 s000 S+ 12:26PM 0:00.03 /usr/local/bin/terraform internal-plugin provisioner chef
NAME 50058 0.0 0.1 4429608 14780 s000 S+ 12:26PM 0:00.03 /usr/local/bin/terraform internal-plugin provisioner chef
NAME 50057 0.0 0.1 4396572 14292 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50056 0.0 0.1 4396316 14344 s000 S+ 12:26PM 0:00.08 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50055 0.0 0.1 4395804 14452 s000 S+ 12:26PM 0:00.08 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50054 0.0 0.2 4541080 33440 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50053 0.0 0.2 4541860 34168 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50052 0.0 0.1 4396060 14580 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50036 0.0 0.2 4540824 34544 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50017 0.0 0.2 4540568 34088 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50016 0.0 0.1 4439184 16312 s000 S+ 12:26PM 0:00.03 /usr/local/bin/terraform internal-plugin provisioner local-exec
NAME 50015 0.0 0.2 4541080 34316 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50014 0.0 0.1 4429700 14880 s000 S+ 12:26PM 0:00.03 /usr/local/bin/terraform internal-plugin provisioner local-exec
NAME 50011 0.0 0.2 4540044 33608 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50010 0.0 0.2 4540044 33468 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50009 0.0 0.2 4541860 34384 s000 S+ 12:26PM 0:00.24 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50008 0.0 0.2 4541092 33960 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50007 0.0 0.1 4396060 14484 s000 S+ 12:26PM 0:00.08 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 50006 0.0 0.2 4539788 34420 s000 S+ 12:26PM 0:00.24 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50005 0.0 0.2 4541604 33436 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 50004 0.0 0.1 4396572 14760 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49994 0.0 0.2 4541068 34076 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49969 0.0 0.2 4541860 33028 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49968 0.0 0.2 4540568 34572 s000 S+ 12:26PM 0:00.20 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49965 0.0 0.1 4395548 14292 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49964 0.0 0.2 4541860 33816 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49963 0.0 0.2 4541348 33660 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49962 0.0 0.2 4540568 32824 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49961 0.0 0.1 4395880 14260 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49960 0.0 0.2 4541604 34020 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49959 0.0 0.1 4396060 14396 s000 S+ 12:26PM 0:00.10 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49958 0.0 0.2 4541348 33884 s000 S+ 12:26PM 0:00.23 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49957 0.0 0.2 4541080 33644 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49956 0.0 0.1 4395880 14360 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49955 0.0 0.1 4396060 14484 s000 S+ 12:26PM 0:00.09 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49954 0.0 0.1 4396316 14540 s000 S+ 12:26PM 0:00.08 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
NAME 49953 0.0 0.2 4541080 33904 s000 S+ 12:26PM 0:00.21 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49952 0.0 0.2 4541080 34176 s000 S+ 12:26PM 0:00.22 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-aws_v2.41.0_x4
NAME 49920 0.0 0.1 4396136 14328 s000 S+ 12:26PM 0:00.10 /Users/NAME/ss/infra_terraform/.terraform/plugins/darwin_amd64/terraform-provider-null_v2.1.2_x4
Steps to Reproduce
Run plan to reproduce (we are using terragrunt to execute terraform plan)
terragrunt plan-all -lock=false
Additional Context
This is occurring on OSX.
All terraform syntax has been upgraded to terraform 12 using terraform 0.12upgrade. There was a similar bug report a few months ago here https://github.com/hashicorp/terraform/issues/21827 which was closed as duplicate of https://github.com/hashicorp/terraform/issues/21584 which has been fixed. My issue persists however.
stahs
👍6
All 9 comments
Can confirm the same issue happens when I use terraform in my production environment as well.
Terraform v0.12.20
+ provider.archive v1.3.0
+ provider.aws v2.46.0
+ provider.null v2.1.2
+ provider.random v2.2.1
+ provider.template v2.1.2
+ provider.tls v2.1.1
chapter63
on 29 Jan 2020
👍9
I can confirm this is still an issue on:
Terraform v0.12.23
- provider.aws v2.53.0
- provider.github v2.4.1
on OSX version 10.14.6
P3BKAC
on 14 Mar 2020
👍8
It's been three months since I submitted this bug report. Can someone please look into this? We're severely impacted by this problem.
stahs
on 15 Mar 2020
This is becoming a bigger and bigger issue as we cannot upgrade and are concerned we'll fall behind on updates. It also prevents us from taking advantage of all the awesome stuff in TF 12!
stantonk
on 16 Mar 2020
👍3
Same here ... to me, the problem seems to start when there are over ~160 resources defined.
Terraform v0.12.23
+ provider.github v2.6.1
OSX version 10.14.6
UPDATE: just upgraded to Terraform v0.12.24 and the problem persists.
marcelocarlos
on 7 Apr 2020
👍2
fyi, a temporary fix that has worked for me to use the terraform docker image on a mac and use that to make changes, since this issue does not happen with linux.
chapter63
on 7 Apr 2020
❤2
This is a blocker to our TF12 migration.
UncleEricB
on 7 Apr 2020
👍1
Found a solution:
ulimit -n 1024 updates the limits on open files and solves problem …. previous limit was 256 in my case (can be seen via ulimit -a)
marcelocarlos
on 8 Apr 2020
👍15
@marcelocarlos confirmed this fixed the problem we were seeing on on osx. Thanks!
stahs
on 22 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
rjinski
·
3Comments
shanmugakarna
·
3Comments
franklinwise
·
3Comments
rnowosielski
·
3Comments
sprokopiak
·
3Comments
Most helpful comment
Found a solution:
ulimit -n 1024updates the limits on open files and solves problem …. previous limit was256in my case (can be seen viaulimit -a)