Terraform-provider-azurerm: KeyVault access policy for Service Principal doesn't work as expected and doesn't provide access to secrets

[gvilarino]

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7
+ provider.azurerm v1.9.0

Affected Resource(s)

• azurerm_key_vault
• azurerm_key_vault_access_policy

Terraform Configuration Files

provider "azurerm" {
  version = "~> 1.9"

  // azure-cli not logged in, using the `ARM_*` env vars. Also tried it with the values under this line not-commented and/or with the azure-cli logged in as the Service Principal.
  //client_id       = "${var.sp_app_id}"
  //client_secret   = "${var.sp_client_secret}"
  //tenant_id       = "${var.sp_tenant_id}"
  //subscription_id = "${var.az_subscription_id}"
}

resource "azurerm_resource_group" "rg" {
  name     = "terraform-rg"
  location = "South Central US"
}

resource "azurerm_key_vault" "vault" {
  name                = "terraform-test-vault"
  location            = "${azurerm_resource_group.rg.location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"

  sku {
    name = "standard"
  }

  tenant_id = "${var.sp_tenant_id}"

  access_policy {
    tenant_id = "${var.sp_tenant_id}"
    object_id = "${var.sp_object_id}"

    key_permissions = [ ]

    secret_permissions = [ "Get", "Set" ]
  }
}

variable "az_subscription_id" {
  type = "string"
  description = "Azure subscription ID"
  default = "REDACTED"
}

variable "sp_app_id" {
  type = "string"
  description = "SP app ID"
  default = "REDACTED"
}

variable "sp_client_secret" {
  type = "string"
  description = "SP client secret"
  default = "REDACTED"
}

variable "sp_object_id" {
  type = "string"
  description = "SP object ID"
  default = "REDACTED"
}

variable "sp_tenant_id" {
  type = "string"
  description = "SP tenant ID"
  default = "REDACTED"
}

Debug Output

https://gist.github.com/gvilarino/2888944a0c62a41791f8ff3dfbacfc17

Expected Behavior

There should be a keyvault named as in the example, with an access policy for the given Service Principal, that allows me to set and get secrets, and that looks like this in the portal:

This is how the commands should work, and what is expected by following the Service Principal creation guide in the official Terraform docs:

// login to azure-cli with the Service Principal

$ az keyvault secret set --name testsecret --vault-name terraform-test-vault --value "Some test secret"
{
  "attributes": {
    // redacted
  "value": "Some test secret"
}

$ az keyvault secret show --name testsecret --vault-name terraform-test-vault          
{
  "attributes": {
  // redacted
  "value": "Some test secret"
}

Actual Behavior

The KeyVault is created, but the access policy looks like this:

It does have the Get and Set secret permissions, but is still unable to access any secrets in the KeyVault:

// create a secret named `testsecret` in the KeyVault via azure portal
// log into azure-cli as the Service Principal

$ az keyvault secret show --name "testsecret" --vault-name "terraform-test-vault"
Access denied

Steps to Reproduce

1. Create an RBAC Service Principal following the official Terraform docs
2. Configure your variables in the terraform file
3. Terraform apply
4. Log into azure cli as the Service Principal you granted the access policy to
5. Attempt to set a secret or read a secret created through the portal

Important Factoids

Factoid 1

If you add the application_id parameter to the access_policy block in the azurerm_key_vault resource, the access policy looks different in the portal:

But it still doesn't work.

Factoid 2

If I use an azurerm_key_vault_access_policy resource instead of the access_policy block inside the azurerm_key_vault resource, the result is exactly the same.

Factoid 3 - Workaround

I managed to work around this through trial and error by removing all access policy definitions and instead using a local-exec provisioner. I added the following to my azurerm_key_vault resource":

provisioner "local-exec" {
    command = "az keyvault set-policy --name ${azurerm_key_vault.vault.name} --spn ${var.sp_app_id} --secret-permissions set get"
  }

HOWEVER, for this to work, the azure-cli _must_ be logged in (I tried as the Service Principal, should work for personal account as well) as the ARM_* env vars or specifying parameters to the azurearm Terraform provider (in the file) makes the az keyvault seet-policy fail for invalid auth.

The debug output for this can be seen in this gist: https://gist.github.com/gvilarino/aa6d3a1b2bd217c1acfa606c3365dc71

References

• The most similar issue I found #627
• The most recently active, related issue I found is #656
• Related PRs I found are: #1349 and #1544

I did not find an issue that describes exactly this problem, this way.

[whytoe]

@gvilarino I have encountered this problem as well, the ICON displays the created account as a User (image 2) instead of an Application Icon (image 1) . I believe that is somehow related to the problem, possibly the wrong API function being used.

[gvilarino]

Precisely. If one adds the access policy manually via the Azure Portal UI or with the azure-cli command I mention, it does appear correctly with the Application account/icon.

@whytoe did you find any workaround other than the one I mention above?

[whytoe]

@gvilarino I have not invested too much time into it, just experienced the exact same issue, I believe.

Possibly if you add the Object ID and the AppID it might work?

```access_policy supports the following:

tenant_id - (Required) The Azure Active Directory tenant ID that should be used for authenticating requests to the key vault. Must match the tenant_id used above.

object_id - (Required) The object ID of a user, service principal or security group in the Azure Active Directory tenant for the vault. The object ID must be unique for the list of access policies.

application_id - (Optional) The object ID of an Application in Azure Active Directory.

[phekmat]

I've only run into this with an incorrect object_id, namely using that of the app rather than that of the SP (which has its own object_id)

You can get the correct id via az ad sp show. Conversely, az ad app show will get you the id of the app rather than the service principal.

[vijayakrishnarg1]

@whytoe i have added application_id, tenant_id and object_id still not able to access through SPN.

i see same behavior with keyvault like @gvilarino...

If it's Group SPN, there is no issue by adding to keyvault through terraform module, only issue if it's application SPN.

[gvilarino]

Ok, some progress, I have been able to successfully set a working access policy:

when you az ad sp create-for-rbac ..., as the documentation states, two things are created in azure:

1. a Application registration
2. an Enterprise application

These both will have the same display nameas specified in the SP creation command the same Application ID, the same Tenant ID, but different Object IDs.

I was using the Object Id of the Application regsitration instead of the one in Enterprise application. Why I was doing this has to do with the following: for the SP to be listed in the Enterprise applications blade, you must manually selectAll Applications` in the combo before typing the name. The weird thing is that there are 2 other options there (Enterprise applications and Microsoft applications), and you won't find it with any of them (which is weird, becasue I assume the third option is just a thing that covers all of the above).

In any case, I have been able to create a vault with the proper SP configuration. I still think the docs are not on-point, because they do not tell you how to get the proper Object ID for the terraform resource.

So the confusion comes from Azure not having a clear visual representation of the Service Princial figure, rather having two "application"-related entities: the Enterprise Application and the Application Registration.

However, I still can't complete my terraform project since even tho I'm able to get the resource created properly, I'm unable to access the SP as a data source since it seems to be missing required permissions. Now, here's the issue: I can't add permissions to a Service Principal. The only place where I see anything related to adding the permissions suggested in the SP authentication docs are set on the Application registration level, not the Enterprise application (as per the above, what actually is the Service Principal).

And even if I do set the SP as an subscription contributor, and set said permissions to the App registration, I still cant access the data source and get:

data.azurerm_azuread_service_principal.test: data.azurerm_azuread_service_principal.test: Error listing Service Principals: graphrbac.ServicePrincipalsClient#List: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Authorization_RequestDenied","message":{"lang":"en","value":"Insufficient privileges to complete the operation."}}}]

Any ideas?

[phekmat]

@gvilarino apologies if I missed it, but did you give the SP you're using with terraform the proper Azure AD permissions (see the note at the top of https://www.terraform.io/docs/providers/azurerm/d/azuread_service_principal.html)?

IIRC, adding those permissions requires you to be an admin, but I don't recall if it's of the subscription, the Azure AD instance, or both.

[gvilarino]

Hi @phekmat. Yes I did set those on the App (not the spn as I didn't find a way of doing that). I also added the spn as subscription Contributor, and I am Subscription owner and Global Admin. Following the docs I can't seem to find any more directions as to what I'm missing

[steve-hawkins]

@gvilarino this is way more complicated than it needs to be, I hit into this yesterday and spent way too long on it as well

The permissions @phekmat mentioned is to the Service Principal that you are accessing Azure API via the Terraform azurerm provider as mentioned on the Data Source: azurerm_azuread_service_principal:-

NOTE: If you're authenticating using a Service Principal then it must have permissions to both Read and write all applications and Sign in and read user profile within the Windows Azure Active Directory API.

when you have given those permissions you also need to Grant permissions, looks like the following via the portal:-

Word of warning as I am now hitting an issue with the data source as it cannot always find the Service Principal ID for an Application Registration even though you can see it with az ad sp show --id

Issue raised here #1844

[gvilarino]

@steve-hawkins wow thanks for the heads-up.

This whole Azure IAM is so... obscure... It really makes me feel like I know nothing about software engineering :(

[squasta]

Hi, I got the same issue with Terraform v0.11.8 and azure provider 1.14.0. I created my Keyvault & secret using Terraform and a SPN. When I want, later, to get a secret using datasource and the same SPN, I got a 403 error. Problem is a bad creation of access policy with a SPN

[BlondeBurrito]

@steve-hawkins Long time no see!
The only way around the original issue I've found is to create two identical access policies for the service principal. The first one specifying the SPs object_id. The second specifying the object_id and the application_id.

Two policies will apear (with the same name), one for the SP and one for the App Registration. Once set all my 403s magically went away and I could access/create secrets

[fluffy-cakes]

I'll leave my 2 cents here for anyone that might care to be in the same position as me.

I have 2 different TF templates; one to build the keyvault with perms, another to deploy a resource and add a secret to the keyvault at the same time. I had pre-setup a service principal to access/deploy to the subscription which had been given contributor rights. I used that same principal whilst deploying my TF templates set in the environment variables of bash. I added the object_id of the service principal to the TF templates by grabbing it from App Registrations in Active Directory (taking the object ID, not the app ID). No matter what I did, I received the same 403 error as everyone has described here.

I found the issue to be using the wrong object ID of the service principal. The working ID I grabbed from Enterprise Applications -> All Applications -> *my-spn* -> Properties -> Object ID. Using this ID in the policy definition of both TF template deployments succeeded.

I also noticed that using the correct object ID shows the correct BLUE application icon for the spn in Access Policies (under the keyvault), not the GREY human icon (as mentioned above).

[vijayakrishnarg1]

@paulmackinnon-adv365

i think by providing object id from (Enterprise Applications -> All Applications -> my-spn -> Properties -> Object ID), it works fine. (Application ID not needed)

initially i was taking object id from (Azure AD ---> App Registrations --> my-spn --> Object ID & Application ID), it was not working.

SPN is same but Object id is different when we get from Enterprise Applications.

@gvilarino i would suggest you to follow same procedure hopefully it fixes your issue as well.

[whytoe]

@vijayakrishnarg1 This works, 100% for me

SPN is same but Object id is different when we get from Enterprise Applications

This creates an application instead of creating a user which the "Application" Object ID creates
@tombuildsstuff @katbyte maybe the documentation can just be updated to ensure the correct ObjectID is used

[bpoland]

I ran into this problem too, and was able to resolve it by using a data source to grab the right ID.

data "azurerm_azuread_service_principal" "sp" {
  application_id = "${var.sp_id}"
}

And then in the keyvault:

access_policy {
    tenant_id = "${var.tenant_id}"
    object_id = "${data.azurerm_azuread_service_principal.sp.id}"
    ...
}

[jamesbannan]

I had this same issue. Using the data source approach as mentioned by @bpoland worked, but only using the access_policy block in the azurerm_key_vault resource. If I used the same policy configuration but using the azurerm_key_vault_access_policy resource, the Key Vault Access Policies were created successfully, but it seemed like the permissions had not taken hold by the time terraform attempted to create the secrets using the service principal which it had just assigned permissions to. Re-running the plan after the first failed attempt succeeded, as the Access Policies were already in place.

[rohitiscancerian]

Check this one out. It works for me.

https://github.com/terraform-providers/terraform-provider-azurerm/issues/1034

[siobam]

Hello,

I guess you use wrong ObjectId.
The right object id belong to service principal and you can get it in your active directory:

So if here is how you can automate this:
We create azure SP, generate output, put it to module.
resource definition

module "key_vault" {
  source = "../../modules/keyVault/keyvault"

  name                = "${var.key_vault_name}"
  resource_group_name = "${module.resource_group_rg01.name}"
  location            = "${module.resource_group_rg01.location}"
  AADSecret           = "${var.client_secret}"

  access_policy_sp_securestorage_id = **"${module.azure_service_principal__secure_storage.id}"**
  access_policy_sp_developers_id    = "${var.key_vault_access_policy_object_id["developers"]}"
  access_policy_sp_app_service_id   = "${var.key_vault_access_policy_object_id["app_service"]}"
}

Service principal
_Module definition with output variable_

data "azurerm_client_config" "current" {}

resource "azurerm_azuread_application" "azuread_application" {
  name                       = "${var.app_display_name}"
  homepage                   = "${var.app_home_page}"
  identifier_uris            = ["${var.identifier_uris}"] # app id url, and this is also service principal name in powershell cmdlets
  reply_urls                 = ["${var.reply_urls}"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = true
}

resource "azurerm_azuread_service_principal" "azuread_service_principal" {
  application_id = "${azurerm_azuread_application.azuread_application.application_id}"
}

resource "random_string" "password" {
  length  = 16
  special = true
}

locals {
  appid_password = "${random_string.password.result}"
}

resource "azurerm_azuread_service_principal_password" "azuread_service_principal_password" {
  service_principal_id = "${azurerm_azuread_service_principal.azuread_service_principal.id}"
  value                = "${local.appid_password}"
  end_date             = "2020-01-01T01:02:03Z"
}

Output variable

output "id" {
  value = "${azurerm_azuread_service_principal.azuread_service_principal.id}"
}

Key vault:
_Also module_

resource "azurerm_key_vault" "key_vault" {
  name                = "${var.name}"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"
  tenant_id           = "${data.azurerm_client_config.current.tenant_id}"

  sku {
    name = "premium"
  }

  tenant_id = "${data.azurerm_client_config.current.tenant_id}"

  # Microsoft Azure App Service
  access_policy {
    tenant_id = "${data.azurerm_client_config.current.tenant_id}"
    object_id = "${var.access_policy_sp_app_service_id}"

    key_permissions = [
      "get",
    ]

    secret_permissions = [
      "get",
    ]
  }

  access_policy {
    tenant_id = "${data.azurerm_client_config.current.tenant_id}"
    object_id = "${data.azurerm_client_config.current.service_principal_object_id}"

    certificate_permissions = [
      "create",
      "delete",
      "deleteissuers",
      "get",
      "getissuers",
      "import",
      "list",
      "listissuers",
      "managecontacts",
      "manageissuers",
      "setissuers",
      "update",
    ]

    key_permissions = [
      "backup",
      "create",
      "decrypt",
      "delete",
      "encrypt",
      "get",
      "import",
      "list",
      "purge",
      "recover",
      "restore",
      "sign",
      "unwrapKey",
      "update",
      "verify",
      "wrapKey",
    ]

    secret_permissions = [
      "backup",
      "delete",
      "get",
      "list",
      "purge",
      "recover",
      "restore",
      "set",
    ]
  }

  # permission for developers permissions
  access_policy {
    tenant_id = "${data.azurerm_client_config.current.tenant_id}"
    object_id = "${var.access_policy_sp_developers_id}"

    certificate_permissions = [
      "create",
      "delete",
      "get",
      "import",
      "list",
      "update"
    ]

    key_permissions = [
      "backup",
      "create",
      "decrypt",
      "delete",
      "encrypt",
      "get",
      "import",
      "list",
      "purge",
      "recover",
      "restore",
      "sign",
      "unwrapKey",
      "update",
      "verify",
      "wrapKey",
    ]

    secret_permissions = [
      "backup",
      "delete",
      "get",
      "list",
      "set",
    ]
  }

  access_policy {
    tenant_id = "${data.azurerm_client_config.current.tenant_id}"
    object_id = **"${var.access_policy_sp_securestorage_id}"**


    certificate_permissions = []

    key_permissions = []

    secret_permissions = [
      "delete",
      "get",
      "list",
      "set",
    ]
  }

  enabled_for_disk_encryption = true

  tags {
    environment = "${var.resource_group_name}-${var.location}"
  }
}

[tombuildsstuff]

👋

Taking a look through here since this appears to have been fixed by updating the Terraform Configuration from using the Application ID to using the Object ID - as such I'm going to close this issue for the moment, but if your still seeing this please let us know and we'll take another look.

Thanks!