Azure-docs: Azure KeyVault: where is the secret that is retrieved through environment variables cached?

[mslot]

I have an environment variable that references a secret in Azure KeyVault:

{ "name": "SECRET", "value": "@Microsoft.KeyVault(SecretUri=https://keyvault_name.vault.azure.net/secrets/secret_name/)", "slotSetting": false }
This is loaded when on Startup.cs in my web api solution:

public void ConfigureServices(IServiceCollection services)
{
    services.AddOptions<Secret>().Configure(o => o.ClearText = Configuration["SECRET"]);

    services.AddControllers();
}

How is the secret resolved? Is it resolved on every option call? Eg everytime it gets injected into my constructor, or is it resolved when the environment variable is resoved and loaded into the Configuration object?

I don't think the docs is clear about this, and it is important for us, because we need to be sure that we dont hammer to much on KeyVault. It costs money in the end.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 7feaa3d0-7233-a3cb-cb9b-a9a55b1f0d7a
• Version Independent ID: f41f9d51-52f8-1f45-1043-1344a3b74458
• Content: Use Key Vault references - Azure App Service
• Content Source: articles/app-service/app-service-key-vault-references.md
• Service: app-service
• GitHub Login: @mattchenderson
• Microsoft Alias: mahender

[mimckitt]

@mslot Please provide us with the link to the documentation you are referring to so we can better assist.

[mslot]

@MicahMcKittrick-MSFT sorry, I forgot: https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references

[mimckitt]

Thanks for that! I have assigned it to the correct engineer to take a look.

[SnehaAgrawal-MSFT]

@mslot Thanks again for the feedback! You may refer to this link: https://docs.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-3.1#reload-secrets
Also, I have assigned the issue to the content author to review further and update the document as appropriate.

[mslot]

@mslot Thanks again for the feedback! You may refer to this link: https://docs.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-3.1#reload-secrets
Also, I have assigned the issue to the content author to review further and update the document as appropriate.

Thanks for pointing this out. Although, it is not quite enough for me. I can see that the configuration in https://docs.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-3.1#reload-secrets is done by adding keyvault to the configuration on startup. What I am refering to is environment variable references, on the form

{
  "name": "SECRET",
  "value": "@Microsoft.KeyVault(SecretUri=https://keyvault_name.vault.azure.net/secrets/secret_name/)",
  "slotSetting": false
}

Could it be that it adds the keyvault to the config on webapp startup, if it senses a @Microsoft.KeyVault value in the environment variables?

[mslot]

Any update to my latest answer? Am I right about my assumption?

[Zimmergren]

I'll concur that this could be better explained in the docs to be more transparent about how the secrets are fetched, and in what scenarios.

I built a PoC recently where I am triggering a Function to execute more than 1M times, but my Key Vault is hit only in the lower hundreds, which to me indicates that they may not be immediately fetched on every request, but I'd like that confirmed in the docs so we can properly design our systems at scale, as it could incur quite some costs if the vault was fetching secrets on all requests, and I expect 200-400 million function executions per month.

[mslot]

I'll concur that this could be better explained in the docs to be more transparent about how the secrets are fetched, and in what scenarios.

I built a PoC recently where I am triggering a Function to execute more than 1M times, but my Key Vault is hit only in the lower hundreds, which to me indicates that they may not be immediately fetched on every request, but I'd like that confirmed in the docs so we can properly design our systems at scale, as it could incur quite some costs if the vault was fetching secrets on all requests, and I expect 200-400 million function executions per month.

This is exactly what I am seeing as well! And I have difficult implementing this, because the price impact can be huge! My client cant afford paying 1000 of dollars each month.

[Zimmergren]

As an update to this, I received an update from Jeff Hollan on Twitter, confirming our suspicions. The secrets are loaded when the Function app is booted up.

https://twitter.com/jeffhollan/status/1205885304038510592?s=19

Hope this helps!

[mslot]

As an update to this, I received an update from Jeff Hollan on Twitter, confirming our suspicions. The secrets are loaded when the Function app is booted up.

https://twitter.com/jeffhollan/status/1205885304038510592?s=19

Hope this helps!

This is great! Then we can safely reference the loaded config and sure that the secrets is only loaded when the app cold starts :) Thanks!

Can we get this into the docs for others to use, Microsoft? Jeff Hollan is a pretty solid source :)

[brandonh-msft]

Agreed, this should be in the docs @Zimmergren

[click2install]

This doesn't particularly help with rolling key updates if that is the case. Does this mean I have to restart all my functions when I rotate keys, which I do regularly?

[mslot]

This doesn't particularly help with rolling key updates if that is the case. Does this mean I have to restart all my functions when I rotate keys, which I do regularly?

No. You have to reload the config as explained here, which doesn't require a restart: https://docs.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-3.1#reload-secrets. You have to have some kind of retry logic, or messaging logic that either:

1. tries to use the secret, if it fails: try to update
2. have some kind of custom app where you update the secret through. After updating you could post a message to a service bus, that is picked up on all that is listening. All listeners then gets the key from Vault

Specifically for database, because Azure SQL dont support the rolling identity like MariaDB and MySQL, your best solution would be to allow both both users+password, access to the database for an amount of time _before_ deleting the old user. This is best security wise, but it is the only known solution to the problem. This also implies that you have to have some kind of logic, to actually delete users from the database. I would "simply" have a listener to the "update secret", but delay the message for a day or two.

[brandonh-msft]

regarding secret rotation, please have a look at this Azure Sample I wrote: https://github.com/Azure-Samples/serverless-keyvault-secret-rotation-handling

This process will get easier in the future - perhaps automatic - but for now yes a "soft reset" of the Function App is necessary to get the new values.

[click2install]

Thank you @brandonh-msft that was how I was thinking of implementing it, I didn't find this already in my searches, it will serve as a good reference point.

@mslot thank you, I have resilient KV clients using Polly for other Azure services. I am yet to try this but I don't imagine you can retry or detect a rolled key for a running function that is using triggers to say EventHub or storage. Your second approach sounds similar to @brandonh-msft's approach which was the way I was leaning for the implementation.

[garyapps]

@brandonh-msft Does this information apply to web apps too? I am not too familiar with Azure functions, so I wasn't sure if it is safe to assume this.

For my web app I currently have the connection string as well as some other passwords and api keys stored in Application Settings page of the web app on the Azure portal. These can be accessed as environment variables in my code. I would like to instead store these values in Key Vault as secrets, and then update each of the variables on the Application Settings page to point to the corresponding secret value, using reference syntax (@Microsoft.KeyVault(SecretUri=https://xxxxx)).

If I do this, will Azure fetch all of the secrets from the Key Vault and put them in the web app's environment only when the instance is started/rebooted? Or will this happen whenever my code refers to the environment variables?

[mslot]

@garyapps. There is two ways of doing this.

Through environment variables referencing
Here you have to:

1. update the environment variable, which in turn will restart the resource

You now have the updated secret.

Through code
You can add this to your startup:

var azureServiceTokenProvider = new AzureServiceTokenProvider();
            var keyVaultClient = new KeyVaultClient(
                new KeyVaultClient.AuthenticationCallback(
                    azureServiceTokenProvider.KeyVaultTokenCallback));

            var keyvaultName = builtConfig["KeyVaultName"];

            config.AddAzureKeyVault(
                $"https://{keyvaultName}.vault.azure.net/",
                keyVaultClient,
                new DefaultKeyVaultSecretManager());

You can now reference your secret this way:

            builder.Services.AddOptions<Secret>()
                    .Configure<IConfiguration>((secret, configuration) =>
                    {
                        var s = configuration["secret"];
                        secret.ClearText = s;
                    });

If you reload the Configuration you will get the latest secret.

Please note
Technically speaking you can reference the secret from the environment variable with @Microsoft.KeyVault(SecretUri=https://keyvault.com/secret/secret_name/) without the version (remember the slash at the end). This will work, but is not officially supported from Azure, so use it with care. The same can be achieved by using the above code-first approach. This works both in Azure Functions (from version 3 with IoC: https://docs.microsoft.com/en-us/azure/azure-functions/functions-dotnet-dependency-injection) and Web API.

[garyapps]

@mslot I understand this part, and I am planning to use the first method (environment variable key vault referencing).

My question was this - Are those secrets read from the key vault and loaded into the web app's environment only one time, when the application starts, and then they are permanently available to the app (without additional trips to the key vault)? Or, is the key vault accessed multiple times, with a new trip to the key vault every time my web app needs the value?

[mslot]

@garyapps the runtime will cache it until next start. Notice that:

1. webapps with always on will in theory never restart
2. azure functions on consumption plan will cold start after 20 minutes of idle

Read this: https://github.com/MicrosoftDocs/azure-docs/issues/44064#issuecomment-567791890 and the Twitter comment.

[garyapps]

@mslot Thanks, sounds like this will work great.