Terraform-provider-vault: Adding new vault_identity_entity_alias causes runtime panic

[JeroenBrinkman]

panic: runtime error: invalid memory address or nil pointer dereference
2019-08-29T15:14:42.161+0200 [DEBUG] plugin.terraform-provider-vault_v2.1.0_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1b9db64]
2019-08-29T15:14:42.161+0200 [DEBUG] plugin.terraform-provider-vault_v2.1.0_x4:
2019-08-29T15:14:42.161+0200 [DEBUG] plugin.terraform-provider-vault_v2.1.0_x4: goroutine 45 [running]:
2019-08-29T15:14:42.161+0200 [DEBUG] plugin.terraform-provider-vault_v2.1.0_x4: github.com/terraform-providers/terraform-provider-vault/vault.identityEntityAliasCreate(0xc00015b5e0, 0x1eb1bc0, 0xc00015a700, 0x2, 0x2c5fb20)

Terraform Version

I have tried all possible combinations between Terraform v0.12.3, Terraform v0.12.7, provider.vault v2.1.0 and provider.vault v2.2.0

Affected Resource(s)

• vault_identity_entity_alias

Terraform Configuration Files

Data has been slightly anonymized to get rid of server and application names.
We manage around 45-50 applications in terraform:

Setup

We are using this module (maint.tf)

resource "vault_identity_entity" "application_identity" {
    name      = "${var.application_name}"
    policies  = ["application"]
}

resource "vault_identity_entity_alias" "aliases" {
    count           = length(var.servers)
    name            = "${var.deployment_type}-${var.servers[count.index]}"
    mount_accessor  = "${var.vault_accessor_id}"
    canonical_id    = "${vault_identity_entity.application_identity.id}"
}

resource "vault_mount" "transit" {
    count = var.has_transit_engine ? 1 : 0
    path = "company/transit/apps/${var.application_name}"
    type = "transit"
}

To define identities for applications with aliases to a certificate authority which looks like this:

resource "vault_auth_backend" "applications_certificate_auth" {
    path = "company/cert/applications"
    type = "cert"
}
resource "vault_cert_auth_backend_role" "applications_certificate_auth_cert" {
    name          = "CAName"
    certificate   = "${file("./path/to/cert.pem")}"
    backend       = "${vault_auth_backend.applications_certificate_auth.path}"
    policies      = []
}

I was trying to add a new server (server13) to one application. No other changes. This was the diff on our config:
old:

module "application_s" {
    source = "./modules/application"
    application_name = "application_s"
    deployment_type = "application-staging"
    servers = ["server1", "server2", "server3", "server4", "server5", "server6", "server8", "server9", "server10", "server11", "server12"]
    vault_accessor_id = "${module.setup.applications_certificate_accessor}"
    has_transit_engine = true
}

new:

module "application_s" {
    source = "./modules/application"
    application_name = "application_s"
    deployment_type = "application-staging"
    servers = ["server1", "server2", "server3", "server4", "server5", "server6", "server8", "server9", "server10", "server11", "server12", "server13"]
    vault_accessor_id = "${module.setup.applications_certificate_accessor}"
    has_transit_engine = true
}

Panic Output

I also removed application names and servernames from this gist, it is also not the full debug log as that was a huge log and purging it of information was a lot of effort. If you need the entire log, or a bigger part of the log let me know and i'll take the time to generate it.
https://gist.github.com/JeroenBrinkman/0efb73725a1774af38e5bee5eafbb2ed

Expected Behavior

I expected the planned alias to be added to vault

Actual Behavior

Vault crashes with a runtime panic

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply old config to empty vault and no terraform state (dry run)
2. terraform apply extra server
3. planned changes are:

  # module.application_s.vault_identity_entity_alias.aliases[11] will be created
  + resource "vault_identity_entity_alias" "aliases" {
      + canonical_id   = "d957a64e-1980-f502-9bdc-e5bc88b8329e"
      + id             = (known after apply)
      + mount_accessor = "auth_cert_b4e16cb9"
      + name           = "application-staging-server13"
    }

1. confirm with yes
2. crash
3. No new alias and a broken state when combined with other changes that do apply succesfully

Important Factoids

• Vault version 1.2.2 running in HA mode
• A (assumed) fault in our config causes terraform to recreate the kv2 store every run
• The issue is 100% reproducible on this vault instance however trying the same kind of change on a different instance with about the same number of apps with different names worked perfectly fine. The other instance has less servers however, so in total will contain less aliases.

References

Not that i know about

[lawliet89]

Could you post a longer extract of the stacktrace where the panic happens? It should include the line number that caused the crash.

[JeroenBrinkman]

Would this be enough? https://gist.github.com/JeroenBrinkman/0efb73725a1774af38e5bee5eafbb2ed
2019-08-29T15:14:42.161+0200 [DEBUG] plugin.terraform-provider-vault_v2.1.0_x4: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-vault/vault/resource_identity_entity_alias.go:68 +0x4e4

I updated the link to the crash log in the first post, it should work now

[lawliet89]

What's happening is that Vault returned an empty response and the code in the provider did not expect this. Thus, it panicked when it tried to get resp.Data["id"] from the empty response.

This seems like a problem with Vault itself. The provider can add code to check that resp != nil before continuing but it can't do anything but error if Vault returns an empty response.

Trying to create an entity alias locally worked with Vault always returning something so I am not sure what is going on.

{
  "request_id": "d540de9f-ea59-5503-f45c-0f586cd2b6a5",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "canonical_id": "0dc1cf7e-c16f-8119-b9b9-b8669431a794",
    "id": "76f6a235-e22a-5e47-3c0a-13f5b5a9a0a6"
  },
  "warnings": null
}

Maybe @tyrannosaurus-becks or @kalafut would know more about Vault internals.

[kalafut]

When you try to create an alias that already exists (e.g. the name, accessor, and canonical ID match an existing alias), Vault just returns a 204 (No Content). So you can create this crash by: have TF make the alias, delete the alias from the state file, run TF Apply again.

A simple remedy for someone in this state is to just delete the alias in Vault, after which TF should be able to create it cleanly.

I do think the provider should handle this case, though it is slightly ugly (or at least I've not come up with a clean solution yet). TF needs the alias ID, but Vault is not returning it during these no-change updates. One approach would be, in the case of an empty response, to read the list of aliases for a given entity ID and match against the other parameters. I'll discuss with the team.

[JeroenBrinkman]

Thanks for the response, i'll keep an eye on this issue for the future. For now i will try to apply the proposed work around later this week.