Terraform-provider-aws: aws_iam_server_certificate deleted and recreated every apply regardless of cert file contents

[howdoicomputer]

Terraform Version

✗ us-east-1 (master) ✗ terraform -version
Terraform v0.9.11

Affected Resource(s)

aws_iam_server_certificate

Terraform Configuration Files

provider "aws" {
  region = "us-east-1"
}

resource "aws_iam_server_certificate" "cert" {
  certificate_body = "${file("${var.cert_path}")}"
  private_key      = "${file("${var.cert_key_path}")}"
}

Expected Behavior

When Terraform reads in a certificate, it shouldn't try to reapply that certificate if it has already been applied.

Actual Behavior

Terraform will attempt to destroy and recreate an IAM server certificate every apply phase - regardless of the certificate's contents.

Steps to Reproduce

1. Create a simple Terraform setup that loads a cert from a path using the file helper function.
2. Run terraform apply.
3. Validate that the cert has been created.
4. Run terraform apply a second time - without touching the original cert - and observe that Terraform destroys and recreates the IAM server cert.

References

This issue was originally created in the main Terraform repository and I'm recreating it here.

• https://github.com/hashicorp/terraform/issues/2625

[howdoicomputer]

This issue combined with https://github.com/terraform-providers/terraform-provider-aws/issues/739 makes things pretty hairy if you're doing SSL related ELB work.

[howdoicomputer]

As a workaround to when #739 and #1215 (this ticket) are together, I added a create_before_destroy to the cert block so that Terraform doesn't time out.

resource "aws_iam_server_certificate" "cert" {
  certificate_body = "${file("${var.cert_path}")}"
  private_key      = "${file("${var.cert_key_path}")}"

  lifecycle {
    create_before_destroy = true
  }
}

The certificate still gets deleted but at least it doesn't error out each time.

[thedarkwriter]

Same behavior on version 0.9.8

[bharathiraja29]

Any update on this?

[christopher-demarco]

Yeah, this is biting us as well.

[thedarkwriter]

I fixed this by separating the chain and not including it in the certificate body file, i.e.

resource "aws_iam_server_certificate" "wildcard.random_stuff.com"{
  name     = "wildcard.random.stuff.com"
  certificate_body = "server.crt"
  certificate_chain = "chain.crt"
  private_key = "server.private.key"

  lifecycle {
     create_before_destroy = true
  }
} 

[bharathiraja29]

Even if we have separate chain, there is not guarantee. It works differently for each of the cert files.

[mahipal-annam]

Removing extra newlines between the certs in the certs chain resolved this issue for me. Give it a try :)

good luck

[christopher-demarco]

Is anybody looking into this?

[thomasbibb]

I'm also running into this now... Its getting rather annoying.

[CpuID]

Ran into this yesterday, and feared for all of our infrastructure as I waited (due to https://github.com/terraform-providers/terraform-provider-aws/issues/739 ) to see if the delete would actually apply.

In the end I panic'ed and created a new aws_iam_server_certificate resource and updated the configs on all of our global ALBs to point at it, just incase the API call ever went through to remove the original from IAM.

I suspect one of the issues at hand is looking at the API docs there's no way to actually retrieve the Private Key (likely for security reasons), so there's no way to truly verify state on a refresh? Returning a (sufficiently strong) SHA of the private key would have been nice on the AWS side, as a comparison operator.

[olenm]

problem still exists in latest TF:

Terraform v0.12.7
+ provider.aws v2.26.0

seems to be an issue boiling down to how AWS stores the certificates.

the JSON that aws returns aws iam get-server-certificate --server-certificate-name "bla", has 64-char-witdh base64 data-text of the cert-bodies, for instance my input file has them as 76-char-width. Then TF stores a sha1sum of this data, obviously with the newline-chars in different locations, changes the sha causing the "changes" trigger.

I think a proper-solution would be to standardize the base64 data-blob cert-bodies (both input-from-file and value returned from aws). Or maybe remove newlines of the base64 components- keeping only headers/footers with appropriate newlines?

My current workaround is the following ( apologies for not formatting it ):

• env-var sets private/public keys and exposes them as var.pub, var.cert, var.chain

then, sadly, for each certificate component, I run some good-ol string-functions, same string-wrapper for pub, priv, chain:

locals {
  aws_chars_before_newline = 64

  certificate_body  = replace(replace(  replace(replace( replace(var.pub, "/\r|\n/", ""),   "/(-----+BEGIN [^-]+-----)/", "$1\n"),   "/(-----+END [^-]+-----)/", "\n$1\n"),   "/(.{${local.aws_chars_before_newline}})/", "$1\n"  ), "/\n\n/", "\n")
}

Edit: another method would be to replace all new-lines with a common replaceable-string that would never occur in the text (like _WEE_) - replace BEGIN/END with newlines instead of _WEE_(allowing a more greedy regex [^-]+-+), replace any double _WEE__WEE_ with a newline, remove all _WEE_, then finally add newlines to each line that includes 64-chars in a row

Edit 2020-09-24:
above "workaround" still works for latest terraform/aws verions (did not try to remove the regex-replacement):

$ terraform -v
Terraform v0.13.2
+ provider registry.terraform.io/-/aws v3.6.0
+ provider registry.terraform.io/hashicorp/aws v3.6.0

[ehamberg]

I just had this issue and it seems to have been caused by the local cert files having DOS line endings. Running dos2unix on the files thus solved the issue.