Azure-functions-host: listSecrets broken for function keys in V2 (Runtime version: 2.0.12050.0)

[mjul]

I am trying to get the default function key for a function.
Both the Azure PowerShell tools and the listSecrets function in an ARM template provide an invalid key.

Investigative information

Please provide the following:

• Timestamp: "2018-09-07T12:52:54.8606523Z" ("eventTimestamp")
• Function App version (1.0 or 2.0-beta): 2.0-beta
• Function App name: functionapp39284732
• Function name(s) (as appropriate): GetKommuner
• Invocation ID:
• Region: West Europe

We are using the new "breaking changes" V2 beta (Runtime version: 2.0.12050.0)

Repro steps

Provide the steps required to reproduce the problem:

First, make sure you have set up a resource group and function app with a function.

I have observed the same problem with all the resource groups/apps/functions I have tested it on.

The names used below are for an example that no longer exists, so you need supply your own values in the relevant places.

Via PowerShell

With the Azure PowerShell module you can reproduce the problem like this:

Invoke-AzureRmResourceAction -ApiVersion '2018-02-01'  -ResourceGroupName 'sandbox39284732'   -ResourceType "Microsoft.Web/sites/functions" -ResourceName 'functionapp39284732/GetKommuner' -Action listsecrets -Force

This will return a key and trigger_url.

Via ARM templates

You can use the following ARM template.

ARM template:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "functionApp": {"type": "string", "defaultValue": "functionapp39284732"},
        "functionName": {"type": "string", "defaultValue": "GetKommuner"}
    },
    "variables": {
        "functionAppId": "[resourceId('Microsoft.Web/sites', parameters('functionApp'))]",
        "functionId": "[resourceId('Microsoft.Web/sites/functions', parameters('functionApp'), parameters('functionName'))]"
    },
    "resources": [
    ],
    "outputs": {
        "functionAppId": {
            "type": "string",
            "value": "[variables('functionAppId')]"
        },
        "functionId": {
            "type": "string",
            "value": "[variables('functionId')]"
        },
        "functionkeys": {
            "type": "object",
            "value": "[listSecrets(variables('functionId'), '2018-02-01')]"
        }
    }
}

You can deploy the ARM template with this command (assuming it is saved in GetKeys.json) and you supply your own -ResourceGroupName

New-AzureRmResourceGroupDeployment -name foo  -TemplateFile .\GetKeys.json -ResourceGroupName nameofyourresourcegroup -Verbose

The outputs contain the function key and the related trigger_url with code. They do not match the value seen in the portal.

Expected behavior

Provide a description of the expected behavior.

I would expect the default function key for the function to be returned.
This was the behaviour last week, before the breaking changes.

Actual behavior

Provide a description of the actual behavior observed.

Both ARM and PowerShell return a key that is not recognized as a function key.

I have verified the following:

• it is NOT the _master host key
• it is NOT the default host key
• it is NOT the default function key for the function (as expected)

Perhaps the new release is still reading keys from the file system rather than the associated StorageAccount?

Also note that the Azure web portal shows a valid value.

The behaviour is the same if I add additional function keys.

Known workarounds

Provide a description of any known workarounds.

None.

Related information

None.

[mjul]

For general 2.0.12050 issues see #3363

[paulbatum]

@mjul I wrote a wiki page to try to explain the current state of things. I dont have a great solution for you right this second, but we are working to get things into a good state. Take a read, and give me your feedback here so I can augment the document with additional answers as required:
https://github.com/Azure/azure-functions-host/wiki/Changes-to-Key-Management-in-Functions-V2

[ejizba]

This announcement said:

if you relied on direct file access or SCM APIs to manage those secrets, that will no longer work.

And your doc said this:

The end result is that working with function keys over ARM when running on functions 2.0.12050.0-alpha or later is currently problematic. We are working on a solution to the issue.

So does that mean SCM API is deprecated whereas the ARM API is just temporarily not working?

Also I read the article @justinyoo mentioned here: https://github.com/Azure/app-service-announcements-discussions/issues/64#issuecomment-417977791. Should we be using the admin api instead of the ARM api? I don't see secrets listed in the docs...but @justinyoo said it was working for him

[paulbatum]

Some quick answers:

1. We don't expect to fix the SCM API, beyond making sure it returns the appropriate error responses.
2. We intend to provide a working ARM API. It might be a new API. Details TBD.
3. You should get good behavior if you use the admin API. I'm not tracking any issues regarding the usage of that API. I suspect the wiki page is out of date for the V2 changes.

[justinyoo]

In order to elaborate @EricJizbaMSFT 's comment, I'd rather make something a bit clear.

• my blog post doesn't rely on KUDU API as @fabiocav has merged #1334 to remove KUDU API sometime soon
• It is possible to call the admin API through the Invoke-RestMethod cmdlet (in a generic way) but not through Invoke-AzureRmResourceAction as it doesn't seem to be implemented or exposed.
• But I do found #3381 to get the individual function key through Invoke-AzureRmResourceAction, which I'm expecting @paulbatum 's second statement, in terms of "TBD" ;-)
• My current workaround is to run a PowerShell script that finds all function endpoint, get the individual key through the admin API and apply the correct keys after the ARM template deployment.

Hope this helps.

[paulbatum]

@justinyoo To be clear, #3381 is not going to be merged as is. We're still working to find the right solution.

[mjul]

@paulbatum Thanks for the useful wiki description, it would be helpful with an example of how to use the recommended admin API to get the correct keys for now.

[irajbalakrish]

@mjul I wrote a wiki page to try to explain the current state of things. I dont have a great solution for you right this second, but we are working to get things into a good state. Take a read, and give me your feedback here so I can augment the document with additional answers as required:
https://github.com/Azure/azure-functions-host/wiki/Changes-to-Key-Management-in-Functions-V2

Do you know if there's a newer API for ARM Templates that support fetching keys from the blobs yet ?

[paulbatum]

@irajbalakrish not yet but we are actively working on this. Its part of an effort to address several issues with working with function apps through ARM. The scope is somewhat large, so I expect its still a few months away.

[annadante]

@paulbatum Any news on this?
I'm trying to create Azure Event Grid subscription with ARM template, but this blocks it:(

[mathewc]

Addressed via https://github.com/Azure/azure-functions-host/issues/3994 - the new key management APIs are now live. Rather than the old "listsecrets" API if you now use the new "listkeys" APIs as described in the issue above,