Terraform-provider-aws: Terraform failed during modify ec2 instance_type

[izaneuski]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

From time to time during EC2 instance type changing terraform fails with:

aws_instance.jupyterhub-instance: error starting instance (i-081a191c23f311c6d): InvalidParameterValue: Invalid value ‘r5.8xlarge’ for instanceType. LaunchPlan instance type does not match attribute value c5.large

Form AWS Support reques I've got:

Upon checking the information on the ticket, it looks to me that this issue is related to eventual consistency. 
From the logs we are able to see that that, the modify instance type from r5.8xlarge to c5.large and practically at same time the start-instance call  executed.

 - ModifyInstanceAttribute - 2020-11-20 12:00:38
 - StartInstances - 2020-11-20 12:00:38 - Failed with InvalidParameterValue

Then retried StartInstances call later and it worked. 

 - StartInstances   2020-11-20T20:44:03.000Z 

We also noticed that sometimes the modify-instances call + start call works fine, and when it failed, it was just tried again. This fit the behaviour of eventual consistency.  
Please, ask customer to check best process/recommendations for eventual consistency (retry/exponential backoff...). 
We would assume there is a mechanism to retry calls or delay the execution in terraform.

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/query-api-troubleshooting.html#eventual-consistency 

Is it possible to add wait for modify step somewhere after Modifying instance
https://github.com/hashicorp/terraform-provider-aws/blob/master/aws/resource_aws_instance.go#L1220
?
It could solve issue.

Terraform CLI and Terraform AWS Provider Version

Terraform v0.11.14
AWS Provider v2.70

Affected Resource(s)

• aws_instance

Terraform Configuration Files

resource "aws_instance" "resize_test_011-instance" {

  ami               = "ami-012d6279edb582477"
  instance_type     = "r5.8xlarge"
  #instance_type     = "c5.large"
  subnet_id         = "subnet-0a4e9ff530f4e290d"
  monitoring        = "true"

  vpc_security_group_ids      = ["sg-0c20237bb1d538a46"]
  ebs_optimized               = "false"

  root_block_device {
    volume_size = "40"
    volume_type = "gp2"
  }

  tags = "${map(
      "Name", "resize_test_011-instance",
    )}"

}

Expected Behavior

Terraform successfully apply changes
and
EC2 Instance is in running state with new instance_type

Actual Behavior

Terraform Failed
and
EC2 Instance is in stopped state with new instance_type

Steps to Reproduce

1. Create EC2 with instance_type = "r5.8xlarge" by running terraform apply
2. Change to instance_type = "c5.large" and run terraform apply

• #0000

[bflad]

This one is being difficult to reproduce via our existing TestAccAWSInstance_changeInstanceType acceptance test, even repeatedly running it, but the general behavior, error messaging, and the AWS Support response are in line with other eventual consistency issues we have seen in the past with EC2. Will go ahead and submit the retry logic into there.

[bflad]

Best effort fix submitted: https://github.com/hashicorp/terraform-provider-aws/pull/16443