Terraform-provider-vault: Timing issue when using dynamic secrets with Vault using "vault_generic_secret" for "azurerm" Terraform provider

[pschulz1]

Versions
Terraform 0.12.10
provider.azurerm v1.35.0
provider.vault v2.0.0

Description
I'm trying to use dynamic secrets, generated by Vault via the Azure Secret Engine, to log into Azure. However, it seems there is a timing issue, in a way that the newly generated secrets are not ready for all Azure services, which causes Terraform to run into either on of the following error messages.

Terraform plan/apply/destroy error messages
Actual IDs removed

a)

Error: Unable to list provider registration status, it is possible that this is due to invalid credentials or the service principal does not have permission to use the Resource Manager API, Azure error: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/ID/providers?api-version=2016-02-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'ID' was not found in the directory 'ID'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: f717c659-5fe0-4eaa-b6af-2c73d9159a00\r\nCorrelation ID: ID\r\nTimestamp: 2019-10-14 09:08:13Z","error_codes":[700016],"timestamp":"2019-10-14 09:08:13Z","trace_id":"f717c659-5fe0-4eaa-b6af-2c73d9159a00","correlation_id":"ID","error_uri":"https://login.microsoftonline.com/error?code=700016"}

b)

Error: Unable to list provider registration status, it is possible that this is due to invalid credentials or the service principal does not have permission to use the Resource Manager API, Azure error: resources.ProvidersClient#List: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'ID' with object id 'ID' does not have authorization to perform action 'Microsoft.Resources/subscriptions/providers/read' over scope '/subscriptions/ID' or the scope is invalid. If access was recently granted, please refresh your credentials."

The documentation for the AWS Secret Engine even mentions something that kind of matches the behaviour I'm seeing here and I tend to believe, this will not be much different with Azure:

"Unfortunately, IAM credentials are eventually consistent with respect to other Amazon services. If you are planning on using these credential in a pipeline, you may need to add a delay of 5-10 seconds (or more) after fetching credentials before they can be used successfully."

I've already successfully configured and tested the Azure Secret Engine configuration and I'm able to generate credentials (client_id and secret_id).

If I'm providing these manually generated secrets along with the corresponding subscription_id and tenant_id in my provider block, everything works fine.

But If I'm trying to use the automated way below, it is a gamble, if a Terraform plan succeeds or running into one of the two error messages mentioned above. Sometimes a back to back Terraform apply leads to different results:

➜ Azure tfa
data.vault_generic_secret.azure: Refreshing state...
...
azurerm_resource_group.main: Creating...
azurerm_resource_group.main: Creation complete after 2s [id=/subscriptions/ID/resourceGroups/pschulz-dev-temp-resources]
azurerm_virtual_network.main: Creating...
azurerm_virtual_network.main: Still creating... [10s elapsed]
azurerm_virtual_network.main: Creation complete after 12s [id=/subscriptions/ID/resourceGroups/pschulz-dev-temp-resources/providers/Microsoft.Network/virtualNetworks/pschulz-dev-network]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
➜ Azure tfa
data.vault_generic_secret.azure: Refreshing state...
Error: Unable to list provider registration status, ...

In addition to that I've cross checked the dynamically generated credentials and extracted them from the debug logs and they work just fine a couple of seconds later.

Unfortunately there is no way to configure a wait timer or build a workaround using a local provisioner as "depends_on" is reserved for future use.

Terraform Configuration Files

provider "vault" {
  address      = "https://vault:8200"
  token          = var.token
  ca_cert_file = "/Users/patrickschulz/Documents/GitHub/_DEMO/tls/vault/ca.crt.pem"
}

data "vault_generic_secret" "azure" {
  path = "azure/creds/azure-temp-creds"
}

provider "azurerm" {
  client_id     = "${data.vault_generic_secret.azure.data["client_id"]}"
  client_secret = "${data.vault_generic_secret.azure.data["client_secret"]}"
  tenant_id = var.tenant_id
  subscription_id = var.subscription_id
} 

Moved from #7717 hashicorp/vault to here, as it seems to make more sense.

[lawliet89]

Not sure if anything can be done from Vault's side. Might be better off asking the azurerm provider to have a "wait" mechanism for initialising itself.

[ttyS0]

I'm being hit by this too.

This type of issue is handled in this provider with asynchronous AWS credentials (https://github.com/terraform-providers/terraform-provider-vault/blob/master/vault/data_source_aws_access_credentials.go)

// Other types of credentials are eventually consistent. Let's check credential
// validity and slow down to give credentials time to propagate before we return
// them. We'll wait for at least 5 sequential successes before giving creds back
// to the user.

Perhaps a similar approach could be taken with Azure credentials generated by Vault?

[jmyers82]

Not sure if anything can be done from Vault's side. Might be better off asking the azurerm provider to have a "wait" mechanism for initialising itself.

I opened a ticket on the AZURERM side and they closed it and told me to come to the vault provider and open the ticket. Looks like someone already did here so just following this.

[tyrannosaurus-becks]

I agree we would need to test and retry Azure credentials until they passed, just like the AWS provider does but with slightly different logic. A colleague has actually passed on some logic that works for them a high percent of the time.

The only other question, really, is where/how to implement it. Option 1 is, in the generic secret provider, when the path contains azure, add the retry logic. Option 2 is, write an Azure-specific data source, and add it there.

I'm leaning towards Option 2 although it's more work, because it would provide a better UX. That's currently the tack I'm planning to take.

[tyrannosaurus-becks]

Working branch is located here for those following along: https://github.com/terraform-providers/terraform-provider-vault/compare/issue-580

[tyrannosaurus-becks]

Hello! A PR is now up here. @pschulz1 , @ttyS0 , @jmyers82 , please take a look and confirm it meets your use case, and feel free to give PR feedback as well if you'd like. Thanks!