Terraform: aws_s3_bucket_object fails if object is modified out of terraform

[Bowbaq]

Expected behavior: resource is marked as modified
Actual:

bowbaq@bowbaq-originate [11:17:24] [/tmp/test] 
-> % cat main.tf 
provider "aws" {
  region = "us-west-2"
}

resource "aws_s3_bucket" "bucket" {
    bucket = "tf-test-bucket-412"

    tags {
        Name = "TF test bucket"
    }
}

resource "aws_s3_bucket_object" "object" {
    bucket = "${aws_s3_bucket.bucket.id}"
    key = "foo"
    source = "foo.txt"
}
bowbaq@bowbaq-originate [11:17:30] [/tmp/test] 
-> % echo foo > foo.txt
bowbaq@bowbaq-originate [11:17:35] [/tmp/test] 
-> % terraform plan
Refreshing Terraform state prior to plan...


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ aws_s3_bucket.bucket
    acl:              "" => "private"
    arn:              "" => "<computed>"
    bucket:           "" => "tf-test-bucket-412"
    force_destroy:    "" => "0"
    hosted_zone_id:   "" => "<computed>"
    region:           "" => "<computed>"
    tags.#:           "" => "1"
    tags.Name:        "" => "TF test bucket"
    website_domain:   "" => "<computed>"
    website_endpoint: "" => "<computed>"

+ aws_s3_bucket_object.object
    bucket:       "" => "${aws_s3_bucket.bucket.id}"
    content_type: "" => "<computed>"
    etag:         "" => "<computed>"
    key:          "" => "foo"
    source:       "" => "foo.txt"


Plan: 2 to add, 0 to change, 0 to destroy.
bowbaq@bowbaq-originate [11:17:51] [/tmp/test] 
-> % terraform apply
aws_s3_bucket.bucket: Creating...
  acl:              "" => "private"
  arn:              "" => "<computed>"
  bucket:           "" => "tf-test-bucket-412"
  force_destroy:    "" => "0"
  hosted_zone_id:   "" => "<computed>"
  region:           "" => "<computed>"
  tags.#:           "" => "1"
  tags.Name:        "" => "TF test bucket"
  website_domain:   "" => "<computed>"
  website_endpoint: "" => "<computed>"
aws_s3_bucket.bucket: Creation complete
aws_s3_bucket_object.object: Creating...
  bucket:       "" => "tf-test-bucket-412"
  content_type: "" => "<computed>"
  etag:         "" => "<computed>"
  key:          "" => "foo"
  source:       "" => "foo.txt"
aws_s3_bucket_object.object: Creation complete

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
bowbaq@bowbaq-originate [11:18:09] [/tmp/test] 
-> % echo bar > foo.txt 
bowbaq@bowbaq-originate [11:18:58] [/tmp/test] 
-> % aws s3 cp foo.txt s3://tf-test-bucket-412/foo
upload: ./foo.txt to s3://tf-test-bucket-412/foo
bowbaq@bowbaq-originate [11:19:08] [/tmp/test] 
-> % terraform plan                               
Refreshing Terraform state prior to plan...

aws_s3_bucket.bucket: Refreshing state... (ID: tf-test-bucket-412)
aws_s3_bucket_object.object: Refreshing state... (ID: foo)
Error refreshing state: 1 error(s) occurred:

* aws_s3_bucket_object.object: : 
    status code: 412, request id: 9FDC9D189DCAFECA

[shinzui]

Is there a workaround?

[chiefy]

Just ran into this myself, the situation is I'm creating a exhibitor properties file bootstrap that gets put to s3, as soon as my exhibitor nodes come up, they modify this file, adding their hostname to the ensemble. When I go to destroy, I get the 412 so I have to manually delete.

Might be nice to have a force_destroy option on the resource, so it will ignore etags?

[petestorey26]

Me too, we use it to upload config the first time servers are built, but latterly update it again without re-launching them, so the file has changed. Tried ignore_changes but it didn't help.

[osterman]

Just curious... what happens if you explicitly specify the etag?

"etag - (Optional) Used to trigger updates. The only meaningful value is ${md5(file("path/to/file"))}."

https://www.terraform.io/docs/providers/aws/r/s3_bucket_object.html

[theherk]

I can't figure out why this is doing If-Match. Why not just update the state's ETag if it differs from the previous state during a state refresh. Currently, if edited in S3 (outside of TF), If-Match fails, and we get a 412.

If nobody sees a reason to fail with If-Match, I suggest we remove that check, and update the ETag in this method.

Here is further reading on the If-Match header.

[theherk]

The above pull removes the If-Match comparison. It will update the local state with the ETag of the file in the bucket if the file in the bucket has changed since the last state.

[miketheman]

I hit this today,t hanks for the awesome writeup, and looking forward to the fix!

If anyone else happens across this problem and is searching for a workaround, I found that if you want to take a bit of a risk, you can do something like this:

$ aws s3api head-object --bucket my-bucket-name --key objectname --output json | jq -r .ETag
"0f42efdc3b169c6f2c82d14e6876c41c"

And then replace the etag value in your .tfstate with this one, it'll allow you to pass the precondition until the fix is merged & released.

[catsby]

Hello friends – I just merged #8872 which should help with 412 errors, however please not that @osterman is correct above in that you need to specify the etag to detect these changes

Just curious... what happens if you explicitly specify the etag?

"etag - (Optional) Used to trigger updates. The only meaningful value is ${md5(file("path/to/file"))}."

https://www.terraform.io/docs/providers/aws/r/s3_bucket_object.html

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.