Terraform: Deposed resources are not destroyed

[henrytk]

We have used the aws_iam_server_certificate resource to rotate certificates. Our Terraform run uploads the certificate, but fails to destroy the old one as it is still in use. This issue is not to address the failure to delete the old certificate, the reason for that is understandable. However, more generally when Terraform fails to destroy resources they are marked as 'deposed' in the state file.

Is there any way Terraform can clean up these resources on subsequent runs?

Terraform Version

0.7.5

Affected Resource(s)

• aws_iam_server_certificate

Terraform Configuration Files

Here is an example of our certificate configuration.

Expected Behavior

Some kind of option to destroy deposed resources.

Actual Behavior

The resource is marked as 'deposed' in the state file and essentially ignored from then on.

[mitchellh]

Deposed resources are destroyed on subsequent terraform apply operations. There is no way currently to target or single these out unfortunately. There are other issues asking for a more expressive targeting syntax to be able to reach these, though.

[alext]

That's what I would have expected to happen, but it doesn't match what we were seeing. Subsequent terraform apply runs were ignoring them, not deleting them.

I've managed to reproduce this with a minimal test case as follows:

• Generate a SSL cert

openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout server.key -out server.crt

• Generate TF config as follows:

variable "region" {
  default     = "eu-west-1"
}

provider "aws" {
  region = "${var.region}"
}

resource "aws_iam_server_certificate" "cert" {
  name_prefix = "tf-test-cert-"
  certificate_body = "${file("server.crt")}"
  private_key = "${file("server.key")}"
  lifecycle {
    create_before_destroy = true
  }
}

• Apply

$ terraform apply .
aws_iam_server_certificate.cert: Creating...
  arn:              "" => "<computed>"
  certificate_body: "" => "41ddebfbfb3556fe5fcb7f746d0501b899fd7955"
  name:             "" => "<computed>"
  name_prefix:      "" => "tf-test-cert-"
  path:             "" => "/"
  private_key:      "" => "7ffc42b98811732210ea7e51567cdeb26cd40649"
aws_iam_server_certificate.cert: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

• Make an ELB that uses this cert (or update an existing one to use it).
• Generate a new cert/key:

openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout server.key -out server.crt

• Run terraform to replace the cert, which will create a new resource and error when deleting the old one:

$ terraform apply
aws_iam_server_certificate.cert: Refreshing state... (ID: ASCAJWPW42GENGBCJRAFU)
aws_iam_server_certificate.cert: Creating...
  arn:              "" => "<computed>"
  certificate_body: "" => "559fb3fc1ba5aa59605eced4da9a5d4a501141ce"
  name:             "" => "<computed>"
  name_prefix:      "" => "tf-test-cert-"
  path:             "" => "/"
  private_key:      "" => "df65f8e4990afa58a6138f285a958494e0eb1436"
aws_iam_server_certificate.cert: Creation complete
aws_iam_server_certificate.cert: Destroying...
aws_iam_server_certificate.cert: Still destroying... (9s elapsed)
...SNIP...
aws_iam_server_certificate.cert: Still destroying... (2m50s elapsed)
aws_iam_server_certificate.cert: Still destroying... (2m59s elapsed)
Error applying plan:

1 error(s) occurred:

* aws_iam_server_certificate.cert: DeleteConflict: Certificate: ASCAJWPW42GENGBCJRAFU is currently in use by arn:aws:elasticloadbalancing:eu-west-1:123456789012:loadbalancer/test-lb. Please remove it first before deleting it from IAM.
    status code: 409, request id: 06df9e13-a500-11e6-aab4-2bbd2523602d

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

• Update the ELB to no longer use this cert.

At this point, runing terraform again does not delete the old cert:

$ terraform apply
aws_iam_server_certificate.cert: Refreshing state... (ID: ASCAIY6CUYIBRV66XGCXG)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

The deposed cert is still listed in the tfstate file:

$ cat terraform.tfstate
{
    "version": 3,
    "terraform_version": "0.7.9",
    "serial": 1,
    "lineage": "70d7ec24-4a78-4fea-8a45-deae986e9e32",
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {
                "aws_iam_server_certificate.cert": {
                    "type": "aws_iam_server_certificate",
                    "depends_on": [],
                    "primary": {
                        "id": "ASCAIY6CUYIBRV66XGCXG",
                        "attributes": {
                            "arn": "arn:aws:iam::123456789012:server-certificate/tf-test-cert-20161107153408837174488txl",
                            "certificate_body": "559fb3fc1ba5aa59605eced4da9a5d4a501141ce",
                            "id": "ASCAIY6CUYIBRV66XGCXG",
                            "name": "tf-test-cert-20161107153408837174488txl",
                            "name_prefix": "tf-test-cert-",
                            "path": "/",
                            "private_key": "df65f8e4990afa58a6138f285a958494e0eb1436"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [
                        {
                            "id": "ASCAJWPW42GENGBCJRAFU",
                            "attributes": {
                                "arn": "arn:aws:iam::123456789012:server-certificate/tf-test-cert-20161107153209833921618mtj",
                                "certificate_body": "41ddebfbfb3556fe5fcb7f746d0501b899fd7955",
                                "id": "ASCAJWPW42GENGBCJRAFU",
                                "name": "tf-test-cert-20161107153209833921618mtj",
                                "name_prefix": "tf-test-cert-",
                                "path": "/",
                                "private_key": "7ffc42b98811732210ea7e51567cdeb26cd40649"
                            },
                            "meta": {},
                            "tainted": false
                        }
                    ],
                    "provider": ""
                }
            },
            "depends_on": []
        }
    ]
}

[dlcc]

I think I am seeing a similar issue, also with certs.

[alext]

I've tested, and this issue is now fixes (as of terraform 0.8).

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.