Terraform-provider-aws: Crash at Aws Batch - Compute Environment Order.

Created on 26 Mar 2019  ·  7Comments  ·  Source: hashicorp/terraform-provider-aws

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.13

Affected Resource(s)

aws_batch_job_queue

Terraform Configuration Files

Can't add them, I may need to add them individually.

Debug Output

https://gist.github.com/waltercool/d138918a1875c84d930654e35f8e776b

Panic Output

aws_redshift_cluster.sales-analytics: Refreshing state... (ID: tf-sales-analytics-staging)
aws_batch_job_queue.sales_analytics_job_queue: Refreshing state... (ID: arn:aws:batch:us-east-1:636318639789:job-queue/tf_sales_analytics_staging)

Error: Error refreshing state: 2 error(s) occurred:

* aws_batch_job_queue.sales_analytics_job_queue: 1 error(s) occurred:

* aws_batch_job_queue.sales_analytics_job_queue: aws_batch_job_queue.sales_analytics_job_queue: unexpected EOF
* aws_s3_bucket.sales_analytics_bucket: 1 error(s) occurred:

* aws_s3_bucket.sales_analytics_bucket: aws_s3_bucket.sales_analytics_bucket: unexpected EOF


panic: runtime error: index out of range
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: 
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: goroutine 525 [running]:
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsBatchJobQueueRead(0xc000aec0e0, 0x2f90320, 0xc000530e00, 0xc000aec0e0, 0x0)
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_batch_job_queue.go:103 +0x4af
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc00018a9a0, 0xc0000a49b0, 0x2f90320, 0xc000530e00, 0xc000659d38, 0x4c0501, 0x29eb940)
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:352 +0x160
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc0001fad20, 0xc0000a4960, 0xc0000a49b0, 0xc00050c380, 0xc0000b4bf0, 0x7fe8d98706c0)
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x92
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc000f30160, 0xc00052f790, 0xc00052f860, 0x0, 0x0)
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: reflect.Value.call(0xc000175560, 0xc000cac8f8, 0x13, 0x3589da4, 0x4, 0xc0000b4f18, 0x3, 0x3, 0xc00008bb00, 0xc000158400, ...)
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/goenv/versions/1.11.4/src/reflect/value.go:447 +0x454
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: reflect.Value.Call(0xc000175560, 0xc000cac8f8, 0x13, 0xc000c98f18, 0x3, 0x3, 0x5d5afb, 0xc000c98f28, 0x100000001)
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/goenv/versions/1.11.4/src/reflect/value.go:308 +0xa4
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: net/rpc.(*service).call(0xc000f3c140, 0xc0003b21e0, 0xc000198070, 0xc000198080, 0xc0003aea80, 0xc0001415c0, 0x29eb900, 0xc00052f790, 0x16, 0x29eb940, ...)
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/goenv/versions/1.11.4/src/net/rpc/server.go:384 +0x14e
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4: created by net/rpc.(*Server).ServeCodec
2019-03-26T18:15:44.481-0300 [DEBUG] plugin.terraform-provider-aws_v1.55.0_x4:  /opt/goenv/versions/1.11.4/src/net/rpc/server.go:481 +0x47e
2019-03-26T18:15:44.486-0300 [DEBUG] plugin: plugin process exited: path=<HIDDEN PATH>.terraform/plugins/linux_amd64/terraform-provider-aws_v1.55.0_x4
2019/03/26 18:15:44 [ERROR] root: eval: *terraform.EvalRefresh, err: aws_batch_job_queue.sales_analytics_job_queue: unexpected EOF
2019/03/26 18:15:44 [ERROR] root: eval: *terraform.EvalSequence, err: aws_batch_job_queue.sales_analytics_job_queue: unexpected EOF
2019/03/26 18:15:44 [ERROR] root: eval: *terraform.EvalRefresh, err: aws_s3_bucket.sales_analytics_bucket: unexpected EOF
2019/03/26 18:15:44 [ERROR] root: eval: *terraform.EvalSequence, err: aws_s3_bucket.sales_analytics_bucket: unexpected EOF
2019/03/26 18:15:44 [TRACE] [walkRefresh] Exiting eval tree: aws_batch_job_queue.sales_analytics_job_queue
2019/03/26 18:15:44 [TRACE] [walkRefresh] Exiting eval tree: aws_s3_bucket.sales_analytics_bucket
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.common"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.reporting"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "output.s3_bucket"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "output.job_batch_queue"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.input"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.source"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.manual"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.transformed"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "output.s3_path_source"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "output.s3_path_input"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "output.s3_path_transformed"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "null_resource.s3_sales_analytics"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "provider.aws (close)"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "provider.null (close)"
2019/03/26 18:15:44 [TRACE] dag/walk: upstream errored, not walking "root"
2019/03/26 18:15:44 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-03-26T18:15:44.486-0300 [WARN ] plugin: error closing client during Kill: err="connection is shut down"
2019-03-26T18:15:44.487-0300 [DEBUG] plugin.terraform-provider-null_v2.1.0_x4: 2019/03/26 18:15:44 [ERR] plugin: plugin server: accept unix /tmp/plugin119020140: use of closed network connection
2019-03-26T18:15:44.488-0300 [DEBUG] plugin: plugin process exited: path=<HIDDEN PATH>.terraform/plugins/linux_amd64/terraform-provider-null_v2.1.0_x4

Expected Behavior

Work

Actual Behavior

Crash using provider.aws above 1.54.0. Works fine below or equal to 1.54.0

My darts points the problem happened after https://github.com/terraform-providers/terraform-provider-aws/pull/7079

Steps to Reproduce

  1. terraform init
    2.a terraform apply
    2.b terraform plan

Important Factoids

No

bug crash servicbatch
Source
picture waltercool
👍3

All 7 comments

Adding log with 2.3.0, previously was using 1.55.0. Bug doesn't happen with 1.54.0.

aws_batch_job_queue.sales_analytics_job_queue: Refreshing state... (ID: arn:aws:batch:us-east-1:636318639789:job-queue/tf_sales_analytics_staging)
aws_redshift_cluster.sales-analytics: Refreshing state... (ID: tf-sales-analytics-staging)

Error: Error refreshing state: 3 error(s) occurred:

* aws_redshift_cluster.sales-analytics: aws_redshift_cluster.sales-analytics: unexpected EOF
* aws_batch_job_queue.sales_analytics_job_queue: aws_batch_job_queue.sales_analytics_job_queue: unexpected EOF
* aws_s3_bucket.sales_analytics_bucket: aws_s3_bucket.sales_analytics_bucket: unexpected EOF


panic: runtime error: index out of range
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: 
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: goroutine 371 [running]:
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsBatchJobQueueRead(0xc000d15f10, 0x2ed8120, 0xc000031400, 0xc000d15f10, 0x0)
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_batch_job_queue.go:103 +0x4ac
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc0004ab260, 0xc000a8f310, 0x2ed8120, 0xc000031400, 0xc0002884e0, 0x4c1f01, 0x2edde60)
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:352 +0x160
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc0008f02a0, 0xc000a8f2c0, 0xc000a8f310, 0xc000ce4a80, 0xc0007edb60, 0x7fe0ac310000)
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x92
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc00000a0e0, 0xc0006d5f60, 0xc0007c2210, 0x0, 0x0)
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: reflect.Value.call(0xc00004ad80, 0xc00036a0c0, 0x13, 0x3c568fc, 0x4, 0xc0007edf18, 0x3, 0x3, 0xc00014a400, 0x0, ...)
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/goenv/versions/1.11.5/src/reflect/value.go:447 +0x454
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: reflect.Value.Call(0xc00004ad80, 0xc00036a0c0, 0x13, 0xc000de1f18, 0x3, 0x3, 0x0, 0x0, 0x0)
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/goenv/versions/1.11.5/src/reflect/value.go:308 +0xa4
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: net/rpc.(*service).call(0xc0006742c0, 0xc000a0a1e0, 0xc000162140, 0xc000162190, 0xc000164780, 0xc0007e6520, 0x2edde20, 0xc0006d5f60, 0x16, 0x2edde60, ...)
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/goenv/versions/1.11.5/src/net/rpc/server.go:384 +0x14e
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: created by net/rpc.(*Server).ServeCodec
2019-03-26T18:50:29.160-0300 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4:   /opt/goenv/versions/1.11.5/src/net/rpc/server.go:481 +0x47e
2019-03-26T18:50:29.166-0300 [DEBUG] plugin: plugin process exited: path=<PATH HIDDEN>.terraform/plugins/linux_amd64/terraform-provider-aws_v2.3.0_x4
2019/03/26 18:50:29 [ERROR] root: eval: *terraform.EvalRefresh, err: aws_batch_job_queue.sales_analytics_job_queue: unexpected EOF
2019/03/26 18:50:29 [ERROR] root: eval: *terraform.EvalSequence, err: aws_batch_job_queue.sales_analytics_job_queue: unexpected EOF
2019/03/26 18:50:29 [TRACE] [walkRefresh] Exiting eval tree: aws_batch_job_queue.sales_analytics_job_queue
2019/03/26 18:50:29 [ERROR] root: eval: *terraform.EvalRefresh, err: aws_s3_bucket.sales_analytics_bucket: unexpected EOF
2019/03/26 18:50:29 [ERROR] root: eval: *terraform.EvalSequence, err: aws_s3_bucket.sales_analytics_bucket: unexpected EOF
2019/03/26 18:50:29 [TRACE] [walkRefresh] Exiting eval tree: aws_s3_bucket.sales_analytics_bucket
2019/03/26 18:50:29 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.input"
2019/03/26 18:50:29 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.manual"
2019/03/26 18:50:29 [ERROR] root: eval: *terraform.EvalRefresh, err: aws_redshift_cluster.sales-analytics: unexpected EOF
2019/03/26 18:50:29 [TRACE] dag/walk: upstream errored, not walking "null_resource.s3_sales_analytics"
2019/03/26 18:50:29 [ERROR] root: eval: *terraform.EvalSequence, err: aws_redshift_cluster.sales-analytics: unexpected EOF
2019/03/26 18:50:29 [TRACE] [walkRefresh] Exiting eval tree: aws_redshift_cluster.sales-analytics
2019/03/26 18:50:29 [TRACE] dag/walk: upstream errored, not walking "aws_s3_bucket_object.reporting"
2019/03/26 18:50:29 [TRACE] dag/walk: upstream errored, not walking "provider.null (close)"
2019/03/26 18:50:29 [TRACE] dag/walk: upstream errored, not walking "provider.aws (close)"
2019/03/26 18:50:29 [TRACE] dag/walk: upstream errored, not walking "root"
2019/03/26 18:50:29 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-03-26T18:50:29.167-0300 [WARN ] plugin: error closing client during Kill: err="connection is shut down"
2019-03-26T18:50:29.167-0300 [DEBUG] plugin.terraform-provider-null_v2.1.0_x4: 2019/03/26 18:50:29 [ERR] plugin: plugin server: accept unix /tmp/plugin772945709: use of closed network connection
2019-03-26T18:50:29.169-0300 [DEBUG] plugin: plugin process exited: path=<PATH HIDDEN>.terraform/plugins/linux_amd64/terraform-provider-null_v2.1.0_x4
waltercool picture waltercool on 26 Mar 2019

Update

Today I made some new tests.

  • Using regular version 2.3.0, still failing.
  • Using own-built 2.3.0 reverting f155c2d3bf5420cef4d2929edb2abb3ccbd074c2 , works.
  • Using own-built 2.3.0 with f155c2d3bf5420cef4d2929edb2abb3ccbd074c2 , fails also.
waltercool picture waltercool on 27 Mar 2019

Fix submitted: https://github.com/terraform-providers/terraform-provider-aws/pull/12632

bflad picture bflad on 2 Apr 2020
🎉1

The fix for this has been merged and will release with version 2.60.0 of the Terraform AWS Provider, later this week. 👍

bflad picture bflad on 29 Apr 2020
🎉1

Many thanks for all your help! Fully appreciate it

waltercool picture waltercool on 29 Apr 2020

This has been released in version 2.60.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

hashibot[bot] picture hashibot[bot] on 1 May 2020

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] picture hashibot[bot] on 29 May 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ccslamstack picture ccslamstack  ·  3Comments
hashibot picture hashibot  ·  3Comments
hashibot picture hashibot  ·  3Comments
hashibot picture hashibot  ·  3Comments
joelittlejohn picture joelittlejohn  ·  3Comments