Terraform: S3 Lifecycle Rule Unnecessarily Updates on Each Terraform Apply

[josh-padnick]

I encountered an issue where adding a lifecycle configuration to an S3 Bucket causes terraform to keep destroying and re-adding the same lifecycle on every subsequent terraform apply.

Terraform Version

Terraform v0.7.4

Affected Resource(s)

• aws_s3_bucket
  

  Terraform Configuration Files

resource "aws_s3_bucket" "cloudtrail" {
  bucket = "gruntwork-josh-test-1"
  force_destroy = true

  versioning {
    enabled = true
  }

  # Automatically delete a log file after X days.
  lifecycle_rule {
    id = "auto-delete-after-x-days"
    prefix = ""
    enabled = true
    abort_incomplete_multipart_upload_days = 5

    expiration {
      days = 5
      expired_object_delete_marker = true
    }

    noncurrent_version_expiration {
      days = 5
    }
  }
}

Debug Output

https://gist.github.com/josh-padnick/aea607bd08cc771344e5c9063f30a495

Panic Output

N/A

Expected Behavior

After the initial apply, subsequent terraform apply or terraform plan should result in no changes.

Actual Behavior

The S3 bucket is created successfully and without error on the initial terraform apply. But without any changes to the template, running terraform plan immediately after shows this pointless change:

~ aws_s3_bucket.cloudtrail
    lifecycle_rule.0.expiration.2217964487.date:                         "" => ""
    lifecycle_rule.0.expiration.2217964487.days:                         "" => "5"
    lifecycle_rule.0.expiration.2217964487.expired_object_delete_marker: "" => "true"
    lifecycle_rule.0.expiration.4129135488.date:                         "" => ""
    lifecycle_rule.0.expiration.4129135488.days:                         "5" => "0"
    lifecycle_rule.0.expiration.4129135488.expired_object_delete_marker: "false" => "false"

Steps to Reproduce

1. terraform apply
2. terraform plan
   

   Important Factoids

Interestingly, I also discovered what is probably a race condition while preparing this issue. In about 25% of cases, doing a simple terraform apply would error out with the following:

1 error(s) occurred:

* aws_s3_bucket.cloudtrail: Error putting S3 lifecycle: NoSuchBucket: The specified bucket does not exist
    status code: 404, request id: 7B9E0FD8D09EAFDC

This is unrelated to this issue, but probably warrants a separate issue if one doesn't already exist.

[rokka-n]

Confirmed in v0.7.11 as well.

[mthoretton]

Confirmed in Terraform v0.7.12.
I found out that it's perfectly working removing expired_object_delete_marker property (but I need it 😜 )

[brikis98]

Same issue with Terraform v0.8.4 and a lifecycle_rule on an aws_s3_bucket resource. Every time I run plan, I see:

~ module.elb_access_logs_bucket.aws_s3_bucket.access_logs_with_logs_archived
    lifecycle_rule.0.noncurrent_version_expiration.#:               "0" => "1"
    lifecycle_rule.0.noncurrent_version_expiration.3678435449.days: "" => "0"

[ms1111]

I see this with v0.8.5 when expired_object_delete_marker is set to true on the lifecycle_rule. (Thanks for the tip @mthoretton!)

Looking in terraform.tfstate after the rule is created, I see:

    "lifecycle_rule.0.expiration.2421617364.expired_object_delete_marker": "false",

So the setting didn't take effect.

Looking at the rule in the S3 console UI, there is no mention of removing delete markers. Maybe it's not applicable anymore.

[wjam]

expired_object_delete_marker is not applicable when expiration has already been configured, so the reason terraform is always trying to apply the changes is that AWS is refusing to get into an incorrect state.

Below is an image of an S3 bucket with expiration enabled and note the message.

Can someone change the title of this bug to reflect the fact that expiration in lifecycle_rule shouldn't have the expired_object_delete_marker property?

[caoxp930]

why I run it and apply it on aws, fund after 5 days, the old files are not deleted?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.