Terraform: Modified both aws_db_parameter_group and aws_db_instance : diffs didn't match during apply

[marpada]

Hi,

I provisioned a RDS instance with MySQL 5.6 but then I had second thoughts and wanted to downgrade to 5.5 on a single step:

@@ -6,7 +6,7 @@ resource "aws_db_subnet_group" "staging" {

 resource "aws_db_parameter_group" "staging" {
   name = "staging"
-  family = "mysql5.6"
+  family = "mysql5.5"
   description = "RDS paramenters for Staging MySQL"
 }

@@ -15,7 +15,7 @@ resource "aws_db_instance" "staging" {
   identifier = "staging"
   allocated_storage = 10
   engine = "mysql"
-  engine_version = "5.6.22"
+  engine_version = "5.5.41"
   instance_class = "db.t2.micro"
   password = "${var.staging_rds_password}"
   username = "admin"

Output of terraform plan:

~ aws_db_instance.staging
    engine_version:       "5.6.22" => "5.5.41"
    parameter_group_name: "staging" => "${aws_db_parameter_group.staging.id}"

-/+ aws_db_parameter_group.staging
    description: "RDS paramenters for Staging MySQL" => "RDS paramenters for Staging MySQL"
    family:      "mysql5.6" => "mysql5.5" (forces new resource)
    name:        "staging" => "staging"

terraform apply woed though:

aws_db_parameter_group.staging: Destroying...
aws_db_parameter_group.staging: Error: 1 error(s) occurred:

* One or more database instances are still members of this parameter group staging, so the group cannot be deleted
Error applying plan:

2 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* One or more database instances are still members of this parameter group staging, so the group cannot be deleted
* 1 error(s) occurred:

* aws_db_instance.staging: diffs didn't match during apply. This is a bug with Terraform and should be reported.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[d4v3y0rk]

I am also experiencing an issue similar to this when trying to destroy an entire VPC and all related resources. I say it is related because I am unable to destroy RDS instance related resources because the RDS instance is not destroyed first.

aws_vpc.stage: Error: 1 error(s) occurred:

* Error deleting VPC: DependencyViolation: The vpc 'vpc-XXXXXXXX' has dependencies and cannot be deleted.
Error applying plan:

4 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* InvalidDBSubnetGroupStateFault: Cannot delete the subnet group 'stage-subnet-rds' because at least one database instance: stage-rds is still using it.
* 1 error(s) occurred:

* 1 error(s) occurred:

* InvalidDBParameterGroupState: One or more database instances are still members of this parameter group stage-db-parameter-group-default, so the group cannot be deleted
* 1 error(s) occurred:

* 1 error(s) occurred:

* DependencyViolation: resource sg-XXXXXXXX has a dependent object
* 1 error(s) occurred:

* 1 error(s) occurred:

* Error deleting VPC: DependencyViolation: The vpc 'vpc-XXXXXXXX' has dependencies and cannot be deleted.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[catsby]

Hey @marpada – sorry for the long silence here. A few things:

In general downgrading won't work here, the AWS API doesn't allow it. In order to do it with Terraform, you'd need to taint the entire resource:

$ terraform taint aws_db_instance.staging

That will cause the recreation of the entire thing, including your parameter group.

Regarding the parameter group, I believe you can get this working using the lifecycle block:

lifecycle {
    create_before_destroy = true
}

You'd need to add that to both the Parameter Group and the DB Instance.

Last, I wasn't able to reproduce the "Diffs don't match" thing. I suspect that's been resolved. Are you still receiving that? I realize it's been some time and again I apologize for taking so long.

[catsby]

Hey @d4v3y0rk – sorry to you too for taking so long to reply. Do you have a configuration that demonstrates this dependency issue?

[marpada]

Thanks, I was able to work around it. I didn't know about the taint command, so I think I assigned a temporary parameter group to the DB, deleted the original parameter group and then the db.

[catsby]

Thanks for following up @marpada. I'm going to close this for now.

[stuartellis]

@catsby - FWIW, I can repro the dependency issue that @d4v3y0rk describes, using Terraform 0.9.4: https://github.com/stuartellis/aws-terraform-express/blob/repro-terraform-1702/terraform/environments/test (the RDS instance and parameter group are defined in rds_instance.tf)

This configuration creates a VPC plus MySQL RDS with a parameter group. It seems to consistently give the same error on terraform destroy:

Error applying plan:

1 error(s) occurred:

• aws_db_parameter_group.app_sql (destroy): 1 error(s) occurred:

• aws_db_parameter_group.app_sql: InvalidDBParameterGroupState: One or more database instances are still members of this parameter group example1-test, so the group cannot be deleted
  status code: 400, request id: 8d012da8-341d-11e7-876f-d79c0f563dd5

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

(Old ticket, I know, but I'm new to Terraform, so only just hit the issue.)

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.