Terraform: aws_iam_server_certificate destroyed and recreated each apply

[rbowlby]

The aws_iam_server_certificate is being destroyed and recreated on each apply. Pretty self-explanatory but let me know if you need any further information.

Terraform v0.5.3

[catsby]

Hey @rbowlby – do you have a configuration that demonstrates this?
I tried with the below configuration on both 0.5.3 and the current master branch (b4c1c22), and I'm not seeing anything unusual (successfully applies, doesn't attempt to re-create)

resource "aws_iam_server_certificate" "test_cert" {
  name = "some_test_cert"
  certificate_body = "${file("key/self-ca-cert.pem")}"
  private_key = "${file("key/test-key.pem")}"
}

If you have a configuration, could you share it? Be sure to omit any secrets, and thanks!

[phoolish]

I am seeing the same behavior.

resource "aws_iam_server_certificate" "test_cert" {
  name = "some_test_cert"
  certificate_body = "${file("key/self-ca-cert.pem")}"
  certificate_chain = "${file("key/self-ca-chain.pem")}"
  private_key = "${file("key/test-key.pem")}"
}

[phoolish]

I upgraded to v0.6.0 and my problem was solved.

[phinze]

I upgraded to v0.6.0 and my problem was solved.

Hooray! Closing.

[jmound]

I was seeing the certificate being destroyed and created on each apply. I upgraded to 0.6.2 and now see:

* aws_iam_server_certificate.[redacted]: diffs didn't match during apply. This is a bug with Terraform and should be reported.

[serenitus]

@phinze Can you reopen this issue please?

I am seeing this behavior on v0.6.4 with the following output from 'terraform plan':

-/+ aws_iam_server_certificate.elb
    arn:               "arn:aws:iam::010289787041:server-certificate/magento_cert" => "<computed>"
    certificate_body:  "6548ec85937b17368c72711640b63be8f5159b15" => "73b0ffa9a0a44c67ef567ab657c8ecf065d2e33d" (forces new resource)
    certificate_chain: "bdd557ae3b7256596020e188e3176fddc50836db" => "bdd557ae3b7256596020e188e3176fddc50836db"
    name:              "magento_cert" => "magento_cert"
    path:              "/" => "/"
    private_key:       "f56315953150e2f9ec3756cd13b2c1d941ee6d3e" => "f56315953150e2f9ec3756cd13b2c1d941ee6d3e"

As you can see, it appears to have computed a difference in the certificate body when it is in fact exactly the same.

This results in it trying to destroy a certificate that's currently attached to a load balancer which in turn results in:

Error applying plan:

1 error(s) occurred:

* aws_iam_server_certificate.elb: [WARN] Error deleting server certificate: DeleteConflict: Certificate: ASCAJWH4JWIH57NNNIV7W is currently in use by arn:aws:elasticloadbalancing:us-east-1:010289787041:loadbalancer/magento. Please remove it first before deleting it from IAM.

[apparentlymart]

Reopening this since it seems like something is still a little broken here...

[zpatrick]

Seeing same behavior as @serenitus on v0.6.6

[phoolish]

I am also seeing this behavior on v0.6.6, but it is intermittent. One of my installs (A) does this every time, but another one (B) never requires it. Both installs are identical except A uses modules. I pulled the resource out of the module and into the main.tf file, but that didn't solve it either.

[catsby]

@serenitus your certificate_body is getting a new hash value there...do you have a test configuration that demonstrates this? How was the key created? I've created dozens of aws_iam_server_certificate today and have not seen this.

[jwieringa]

I'm also exeperiencing this issue with a certificate signed by DigiCert. Upon each plan/apply, the certificate body is considered changed by Terraform. I believe the issue has something to do with the file encoding. For example:

$ cat -vet server.crt

Shows that at the end of each line on the DigiCert signed certificate there is a ^M$, whereas a self signed certificate I made has a $. The following, where on a Mac the ^M is inserted with "CTRL V CTRL M", removed the character.

sed -e "s/^M//" server.crt > filtered-server.crt

Using the certificate with the removed ^M in Terraform resolved the different hashes in each plan/apply for myself.

[phoolish]

@jwieringa that was the problem for me as well. nice find!

[rbowlby]

Great work @jwieringa, I'll remove the character. Hopefully that makes the bugfix super easy too.

[elyast]

I still observe this issue (terraform 0.6.7) even tough I don't have `^M' character, in my case hashes are calculated during apply but stored in state file with totally different ones (I'm using S3 backend for state)

this may be related also to https://github.com/hashicorp/terraform/issues/3837

[JoshuaEstes]

tl;dr;

# Both crt and pem files
sed -e "s/^M//" server.crt > filtered-server.crt

Pull the certificate chain out and use certificate_chain if it still does not work.

---

Props to @jwieringa That helped with the private key, the certificate I still had issues with. I was able to eventually discover http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ssl-server-cert.html#certificate-prerequisites after reading that AWS pulls the chain out (thanks for the issue reference @elyast) I had to pull the chain out of my cert and use certificate_chain to get this working.

---

How I pulled the chain out (there might be an easier way)

# This will output your certificate, this content needs to stay
openssl x509 -inform PEM -in certificate.crt

Move everything else in the file to a certificate chain file

[jmstone617]

Using sed -e "s/^M//" server.crt > filtered-server.crt only seems to replace the very last instance of '^M' in the .crt file. I tried sed -e "s/^M//g" server.crt > filtered-server.crt, also, with the same result. Am I missing something?

[jwieringa]

@jmstone617 Perhaps a g in the sed statement? I'm probably missing a sed/unix basic here, but I can see how there can be different results:

$ echo "hello goodbye hello goodbye hello goodbye" | sed -e "s/goodbye//"
hello  hello goodbye hello goodbye
$ echo "hello goodbye hello goodbye hello goodbye" | sed -e "s/goodbye//g"
hello  hello  hello

[jmstone617]

As I mentioned in my original comment, I also tried it with g to no avail.

[jwieringa]

@jmstone617 I'm sorry, I read to fast. I don't have any recommendations, hope you find a solution soon.

[jmstone617]

I was able to use dos2unix (installed via homebrew) to create the correct file. It took a little kung-fu to update the certificate via Terraform, which included removing the entry and changing my ELBs to use non-secure ports, applying, and then undoing those changes and reapplying. In the AWS console, I noticed that the ELB's all had "Invalid certificate" listed for their certificate, even though the new certificate appeared in the drop down if I clicked on "Edit". So, I selected the cert manually, and subsequent terraform apply calls produce the same hash for the chain file, so everything is good.

[aldarund]

I have same\similar issue.
On every queue in atlas my cert body have a different hash.
certificate_body: "86b59f743a4d2f7c42660f19ba49a59b683d8f54" => "e38dd909b6eb588f3eb86e8980d559200fd4aa5f" (forces new resource)

But i just manually requeue it without anything changed at all. And when i run plan it results in Invalid-Certificate () in ELB

[aldarund]

certificate_body:  "86b59f743a4d2f7c42660f19ba49a59b683d8f54" => "e38dd909b6eb588f3eb86e8980d559200fd4aa5f" (forces new resource)

• this is same for every plan. After apply it says all success but then again with new plan same thing. Cert is actually changed on aws .
  Any workaround for this? It makes unusable terraform since its always trying to create a new cert and with conjunction of https://github.com/hashicorp/terraform/issues/4689 it just fails every time and cant really be used.

[jmstone617]

I had success installing dos2unix and reformatting the certificate chain file provided by my CA. Also make sure you break apart the cert body and cert chain.

[JoshuaEstes]

@aldarund Just make sure you have the cert(s) formatted correctly. Double check everything, check cat -vet file.crt and make sure there are no extra line endings.

[aldarund]

thanks. splitting body and chain and removed one extra line helped.

[catsby]

Hello friends – it seems the root of this issue is issues with the formatting of the cert file(s), so I'm going to close this. Let me know if there is anything else I can do here

[RykHawthorn]

Had the same issue without wrong line endings. The problem with the certificate chain was the row length in the file. My certificate was fine with a row length of 65 chars, but the intermediate certificate somehow had a row length of 77 chars. After proper setting of the row length in the intermediate certificate to 65 chars, everything was fine. It appears that "terraform" rearranges the lines and thats why it generates every time the same but different hash. Maybe this information helps someone.

[bbill87]

Shouldn't this be fixed by terraform? i.e if openssl reads the certificate as identical then we shouldn't be trying to compare line endings or how long a line is, etc. It just seems like poor usability.

[donnoman]

agreed this is 2017 and Terraform v0.8.5 and still biting me

[tuannvm]

terraform 0.9.9 and I still have this issue.

[apparentlymart]

Sorry for all the odd behaver here, everyone.

It does seem like we are missing a value normalization step here, but with the AWS provider now being developed in its own repo it may be best to open a fresh issue over there to track this. If someone has a moment to open a new issue for this in the other repo, with some up-to-date repro steps, I _think_ we should be able to make this worth more smoothly.

[howdoicomputer]

@apparentlymart I've created a new issue and linked it back to here.

[valouille]

I'm still experiencing this issue using 0.11.4. I fixed it by doing the following command :

aws iam get-server-certificate --server-certificate-name xxx

and then I noticed that in my chain files I had one more \n compared to what AWS was returning. Terraform then didn't understand I was the same certificates, and wanted to recreate them. The correct syntax when using PEM containing multiple certificates is the following :

-----BEGIN CERTIFICATE-----
XXXXX
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
XXXXX
-----END CERTIFICATE-----

without any new line after that.

[gravcat]

I split the certificate from the ca/chain certificate to stop this from continuing to happen. It seems that Terraform or AWS has some built-in splitting or data massaging happening with the previous form. As a result, every time terraform apply was invoked, it'd read my original files, notice the discrepancy, and ship configuration as declared, ultimately achieving the same result.

Now my resource reads like this:

### new:
resource "aws_iam_server_certificate" "myapp_pool_alb_cert" {
  name_prefix                  = "myapp-pool-alb-cert"
  certificate_body             = "${file("${var.secrets_dir}/certificates/mycert/mycert.pem")}"
  certificate_chain            = "${file("${var.secrets_dir}/certificates/mycert/mycert_chain.pem")}"
  private_key                  = "${file("${var.secrets_dir}/certificates/mycert/mycert.key")}"

  lifecycle {
    create_before_destroy      = true
  }
}

As opposed to:

### old:
resource "aws_iam_server_certificate" "myapp_pool_alb_cert" {
  name_prefix                  = "myapp-pool-alb-cert"
  certificate_body             = "${file("${var.secrets_dir}/certificates/mycert/mycert.pem")}"
  private_key                  = "${file("${var.secrets_dir}/certificates/mycert/mycert.key")}"

  lifecycle {
    create_before_destroy      = true
  }
}

[hernandanielg]

I'm still experiencing this issue using 0.11.4. I fixed it by doing the following command :

aws iam get-server-certificate --server-certificate-name xxx

and then I noticed that in my chain files I had one more \n compared to what AWS was returning. Terraform then didn't understand I was the same certificates, and wanted to recreate them. The correct syntax when using PEM containing multiple certificates is the following :

-----BEGIN CERTIFICATE-----
XXXXX
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
XXXXX
-----END CERTIFICATE-----

without any new line after that.

This worked for me :) I was cracking my head up! thanks @valouille

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.