Azure-functions-host: ARM deployment fails when using WEBSITE_RUN_FROM_PACKAGE and listkey function

[mjisaak]

I have an Azure ARM template that deploys two Azure Functions to an Azure App Service Premium Plan (EP1). The templates add the default host key of the first function to the app settings of the first function using the listkey function:

"FunctionOneKey": "[listkeys(concat(variables('functionOneppId'), '/host/default/'),'2016-08-01').functionKeys.default]"

If I deploy this template to a new resource group, it works the first time (at least most of the time). Every subsequent deployment fails with a Bad Request Error:

{
    "Code": "BadRequest",
    "Message": "Encountered an error (ServiceUnavailable) from host runtime.",
    "Target": null,
    "Details": [
        {
            "Message": "Encountered an error (ServiceUnavailable) from host runtime."
        },
        {
            "Code": "BadRequest"
        },
        {
            "ErrorEntity": {
                "Code": "BadRequest",
                "Message": "Encountered an error (ServiceUnavailable) from host runtime."
            }
        }
    ],
    "Innererror": null
}

The reason for the ServiceUnavailable error is probably because Kudu adds a web.config with a rewrite rule (because I use WEBSITE_RUN_FROM_PACKAGE but don't have deployed the function).

Investigative information

Please provide the following:

• Function App version: 3.0 / 2.0
• Region: Tested with North Europe, Germany West Central

Repro steps

The templates is available here.

You can reproduce this issue by deploying this template at least two times:
using e. g. New-AzResourceGroupDeployment -ResourceGroupName 'my-resource-group' -TemplateFile "c:\Path\to\azuredeploy.json" -name "azuredeploy-$(New-Guid)"

Expected behavior

I should be able to run the ARM deployment multiple times without an error.

Known workarounds

The deployment works if I disable WEBSITE_RUN_FROM_PACKAGE.

Related information

I posted a related question on StackOverflow.

[DK-Dennis]

Same issue here, I cannot use listkeys, and I get the same error message, which really does not say anything meaning- or helpful.

My template is deploying an EventGrid Topic and some EventTopic Subscribers, I only reference the Function App to get the key, in order to use it for the Subscribers. The Function App, however is Run From package, since C# code is deployed to it in an earlier stage.

[older]

I have exactly the same issue but I use neither App Service Premium Plan nor WEBSITE_RUN_FROM_PACKAGE. In my case it is key vault references which seem to trigger this error. I updated template to use key vault references and get this error. If I deploy previous version of the template - it works fine. If I delete app service - both versions of the template work fine.

It is reproducible 100% of the time for me.

Also, I don't know if this is relevant but I'm using implicit consumption hosting plan:

A hosting plan is only required when you choose to run your function app on a Premium plan (in preview) or on an App Service plan.

[DK-Dennis]

Possible solution: I've changed the json template's schema to:
"$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json" (same as the one in the listkeys() function)
This is also the newest, supported schema, as far as I reckon

Now, even many, subsequent deployments seem to work consistantly. I'll be back if that changes...

[mjisaak]

Possible solution: I've changed the json template's schema to:
"$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json" (same as the one in the listkeys() function)
This is also the newest, supported schema, as far as I reckon

Now, even many, subsequent deployments seem to work consistantly. I'll be back if that changes...

Thanks for your answer @DK-Dennis ! I will give it a try.

[DK-Dennis]

Thanks for your answer @DK-Dennis ! I will give it a try. <

De nada, Martin, I'm excited to hear about your results. If this fix works, please give my answer a vote on stackoverflow, will ya?

[mjisaak]

Thanks for your answer @DK-Dennis ! I will give it a try. <

De nada, Martin, I'm excited to hear about your results. If this fix works, please give my answer a vote on stackoverflow, will ya?

sure, give you a vote anywhere and will accept your answer if it works. cheers

[mjisaak]

Thanks for your answer @DK-Dennis ! I will give it a try. <

De nada, Martin, I'm excited to hear about your results. If this fix works, please give my answer a vote on stackoverflow, will ya?

@DK-Dennis: Unfortunately I still get the error after a second deployment :-(. Thanks anyway

[DK-Dennis]

I'll keep poking around a bit, as I would really like to be able to trust the way the template works

[guderkar]

I have exactly the same issue. For me this error occurs when I deploy to empty RG.

When I deploy the template again when the functionapp is already present in RG the problem disappears.

{
      "type": "Microsoft.KeyVault/vaults/secrets",
      "name": "[concat(parameters('keyvault').name, '/', 'FUNCTION-APP-KEY')]",
      "apiVersion": "2018-02-14",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.KeyVault/vaults', parameters('keyvault').name)]",
        "[resourceId('Microsoft.Web/sites', parameters('functionapp').name)]"
      ],
      "properties": {
        "value": "[listkeys(resourceId('Microsoft.Web/sites/host', parameters('functionapp').name, 'default'), '2019-08-01').functionKeys.default]"
      }
}

To me it seems like a dependency issue. The function app seems like created and OK, but the listKeys api call proves otherwise.

[balag0]

@guderkar Can you share the function app name you tried this with?

[guderkar]

whoops sorry the function app name is hvxxbpfunctionsdev

[balag0]

Can you check if you are setting AzureWebjobsStorage appsetting in your template?

I found the following exception for this app.

System.InvalidOperationException : Secret initialization from Blob storage failed due to missing both an Azure Storage connection string and a SAS connection uri. For Blob Storage, please provide at least one of these. If you intend to use files for secrets, add an App Setting key 'AzureWebJobsSecretStorageType' with value 'Files'.

I would have also expected this error to be available in your appinsights.

[jeromekol]

We have the same issue. Creating a vanilla app on the Azure Portal and setting WEBSITE_RUN_FROM_PACKAGE to 1 leads to a 502. Pretty sure, that was working before.

[balag0]

@jeromekol Are you creating a linux function app in consumption plan by any chance? In that case WEBSITE_RUN_FROM_PACKAGE needs to point to a sas-uri.

[guderkar]

@balag0 Thanks for the suggestion, in my template I was first doing listkeys and then associating appsettings with the function. It was beacause I needed secrets to be in keyvault first. Setting AzureWebJobsSecretStorageType: Files on functionapp creation helped. Thanks a lot

[jeromekol]

@balag0 no, its a windows app.

[balag0]

@guderkar Please note AzureWebJobsSecretStorageType=Files is not supported on Linux consumption plans yet , the recommendation is not to set that AppSetting at all.

@jeromekol If you can share the functionapp name, I can take a closer look.

[DK-Dennis]

Now, all of a sudden it has stopped working again. Multiple, identical, repeated attempts have now resulted in it working the last time of 6 attempts (fails 5 times out of 6). This is when running the template deployment through a release pipeline. When repeating the deployment in azure portal, it fails roughly half of the time. So my solution above is really not a solution, I'm afraid.
I'm still sure, that listkeys() is the culprit, since omitting the function makes repeated attempts succeed 10 out of 10.
My thoughts along the difference between pipeline deployments and portal deployments are, that they probably have a different (but still too low) API timeout value, when listkeys() is invoked, but I dunno ...

[cytron201]

Further to this, I've had similar experience. Running it from powershell locally works sometimes (mostly fails), where running it from the pipeline fails every time.

Worth noting: After doing a deployment, when going to the function blade in Azure (using the new preview view), and selecting the App keys option, there is at least a 10 second delay before the keys are displayed. If it's connected, it would make sense. There's a definite issue retrieving the host keys.

[smokedlinq]

I opened a case with support this week on a similar issue. I was trying to use a Linux based function app, however the container wouldn't spin up before the "create" of the function app was done so it would fail. Moving to a Windows function app this problem does not seem to exist.

I did experience failures consistently when using a consumption plan. Normal app service plan worked fairly reliably.

Also, I had a similar issue with listKeys on a function app slot, but this was because I was missing a dependency on the creation of the slot. Even if you aren't using a slot, double/triple check your dependencies. I actually used a dependency on the config/appsettings resource of the function app which so far has been working reliably.

[balag0]

If some one can share their app name and the template, we can investigate.
It looks like there are multiple different failures being reported on this thread across different skus , os types, different version of templates making it very difficult to investigate.

[smokedlinq]

@balag0 I sent you an email with some info I have from recent experiences.

[balag0]

@smokedlinq Thanks, I am looking into it.

[DK-Dennis]

@balag0 We've gone with another solution, in stead of listing the keys, we create function apps with a (known) host key to use. I don't particularly like that, but there you go.
Are you still interested in me trying to reproduce the problems I've experienced?

[balag0]

Thanks for the help @DK-Dennis. I got a repro from smokedlinq and hopefully we can root cause the issue with that. I would like to make sure the fix we identify works for your scenario also. I will provide more details a soon as I know more. For now I think we have enough info to investigate.

[balag0]

@smokedlinq It looks like the template you shared is for a windows app and you mentioned you were seeing the failures on linux. Specifically I was looking for
"reserved": true,
in the serverfarm properties.

Is there a different template you were using earlier?

[balag0]

@ alI am closing this thread since there are too many different issues being conflated and it is very difficult to see what needs to be investigated / what is failing.

Please open a new (separate) issue and feel free to @ me and I will follow up individually.

[mjisaak]

@balag0 Seriously? The issue I described still exists and is 100% reproducible. I also spend multiple days tracking down the issue from a very large deployment to this small template in order to help you fix the issue.

[balag0]

@mjisaak I tried the template a few times and deployment went through fine.

Can you clarify if it is the deployment that is failing or the sites don't start up and fail with 503?

[mjisaak]

@balag0
It is the deployment that fails. Sometimes, I get the error on the first (initial) deployment. At least the second deployment will fail 100% -
I tried it on multiple different subscriptions and locations.

To make it easier for you, I adopt the template to use unique names for the resources so just provide a resource group name and execute the following deployment twice:

az deployment group create -g github-issue-5722-arm-rg --template-uri https://gist.githubusercontent.com/mjisaak/d335447be4b6e25ce718f8fcc461afcf/raw/856876868041e827abcc5ad50793d04ba1baedbd/azuredeploy.json

This is how the error looks like in the portal:

And here the error message:

{
"Code": "BadRequest",
"Message": "Encountered an error (ServiceUnavailable) from host runtime.",
"Target": null,
"Details": [
{
"Message": "Encountered an error (ServiceUnavailable) from host runtime."
},
{
"Code": "BadRequest"
},
{
"ErrorEntity": {
"Code": "BadRequest",
"Message": "Encountered an error (ServiceUnavailable) from host runtime."
}
}
],
"Innererror": null
}

Also, if you have access, feel free to check the Azure resources yourself:
Subscription: mbr_mvp_12k
Resource group: github-issue-5722-arm-rg

[balag0]

Thanks, this was helpful.

Since the template has WEBSITE_RUN_FROM_PACKAGE set but there is no actual content deployed , the file system check fails.
https://github.com/Azure/azure-functions-host/blob/9fb8b40dbeb1b5a902094bc88e9766fa8945a6b8/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L162

Removing WEBSITE_RUN_FROM_PACKAGE = 1 should work. If you are looking to deploy content using arm template then consider zip deploy
https://github.com/projectkudu/kudu/wiki/MSDeploy-VS.-ZipDeploy

Hope this resolves this issue. Feel free to open a new issue for any other questions.