Azure-cli: ARM template should not delete and replace application settings for a function app

[ajklotz]

I created an ARM template to deploy and manage two Azure Functions across multiple environments (CI, DEV, QA, PROD, etc). My goal was to have a single ARM template that deploys to each using a parameters.json file. In my parameters.json file, I created a JSON structure where I could reference each environment and all of the application settings. My function app has the siteConfig.appSettings object included in properties so I could use template functions to apply dynamically generated application settings, like a appinsights instrumentation key, and other things. Within the function app resource, I defined another resource to apply static application settings from a parameters.json file.

Unfortunately, when the ARM template deployed it deleted any application settings that were not defined in the ARM template. This means that any application settings that were defined as part of the initial function app were deleted and replaced by the parameters.json. Also, all of the dynamically generated application settings generated from the template functions were removed and replaced by the application settings defined in the parameters.json. The two resources do not work together at all.

I went into great detail with my struggles on how to handle environment-specific ARM template deployment of a function app with dynamic and static application settings, on a stackoverflow post.

Describe the solution you'd like
ARM template deployment of application settings should NOT delete existing application settings. The PowerShell Azure CLI cmdlet does not delete anything and merges nicely.
az functionapp config appsettings set
The ARM template should behave similarly.

Describe alternatives you've considered
The alternative is for me to put everything in a key vault and then download the key vault secrets to get application settings and then use the az functionapp config appsettings set command. But a lot of my application settings have characters that are not allowed as names for the secrets, so a bunch of annoying renaming has to occur.

Another alternative is to create deployment templates for each environment (dev, qa, prod) that contain all of the application settings. This alternative goes against the idea of having deployment templates that can be reused (destructured). I feel that MSFT didn't design ARM templates to be used in this manner. ARM templates should be able to support multiple environments from a single deployment template using parameter templates.

_I feel that this is more of a bug and not a feature request_.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure

[ajklotz]

Any chance this might get fixed soon?

[Juliehzl]

@Tiano2017 do you have any idea about this feature?

[Tiano2017]

@ajklotz did you use the "complete" mode when you submit the template deployment? A "complete" mode deployment deletes all resources that were not specified in the template from the resource group. The "Incremental" mode would prevent that.

[ajklotz]

@ajklotz did you use the "complete" mode when you submit the template deployment? A "complete" mode deployment deletes all resources that were not specified in the template from the resource group. The "Incremental" mode would prevent that.

I did not use complete mode.

[ajklotz]

Please consider getting a fix in a upcoming milestone, this issue is blocking us from using ARM templates to deploy resources to Azure in our CI/CD pipeline. I started creating ARM templates for our Azure Functions and planned to have the entire infrastructure deployed via ARM template. It is discouraging to see that my first ARM template is running into this issue and I'm worried that every other resource I try to automate with ARM templates will run into similar problems where things just arent idempotent. We have several more App Services with application settings and I imagine we will be running into this problem.

[clawrenceks]

Same issue here. App Settings in an ARM template are not respecting the incremental nature of ARM template deployments.

[clawrenceks]

The same goes for connection strings.

[kutsyk-neocles]

Hey guys, is this bug getting fixed?

[alex-frankel]

Incremental, unfortunately, is a misleading term.

Incremental mode does not apply to properties of a resource, it applies to the resource itself. If a deployment is done in complete mode, that means it will delete resources not declared in the template, but it does not affect how the properties within a resource are handled. That is up to the RP itself (in this case the web RP).

@bim-msft or @Juliehzl - is there someone from the web team we can bring into this?

[zhoxing-ms]

@alex-frankel I would like to confirm with you the key issues:
What you mean is that the incremental mode and complete mode of the ARM template only affects whether the undefined resources in template are deleted, but they do not take effect on the property settings in resources defined in the template.
These original property settings will be directly overwritten by new properties in the parameter file when deployed in either mode, right?

[alex-frankel]

These original property settings will be directly overwritten by new properties in the parameter file when deployed in either mode, right?

In most cases that is correct, but some RPs will treat this as a PATCH request which will do a strategic merge instead of overwriting. This is totally up to the RP teams though.

The rest of what you said is correct.

[zhoxing-ms]

In most cases that is correct, but some RPs will treat this as a PATCH request which will do a strategic merge instead of overwriting. This is totally up to the RP teams though.

@Tiano2017 Could you please help to see whether the original property settings should be directly overridden or merged by strategy during deployment ARM template?

[alex-frankel]

Here is the RPC for ARM PUTs: https://github.com/Azure/azure-resource-manager-rpc/blob/master/v1.0/resource-api-reference.md#put-resource

It's supposed to overwrite by design because ARM does not distinguish between an update or create. However, certain RP teams have chosen to treat PUTs on existing resources as a PATCH. That's why I recommend bringing in someone from the Web RP.

[ajklotz]

I'm observing back and forth conversation. Can you define "RP"? Thanks

[alex-frankel]

Resource Provider - the first part of the resourceType (e.g. Web, Compute, Sql, etc.).

[johnwc]

Did this get raised with the Web RP team?

[MauricioZa]

Hey @ajklotz, do you think this is how the product works by design?

See this:
https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-modes#incremental-mode

In the note there you can read that:

..."When redeploying an existing resource in incremental mode, all properties are reapplied. The properties aren't incrementally added. A common misunderstanding is to think properties that aren't specified in the template are left unchanged. If you don't specify certain properties, Resource Manager interprets the deployment as overwriting those values. Properties that aren't included in the template are reset to the default values. Specify all non-default values for the resource, not just the ones you're updating. The resource definition in the template always contains the final state of the resource. It can't represent a partial update to an existing resource.

In rare cases, properties that you specify for a resource are actually implemented as a child resource. For example, when you provide site configuration values for a web app, those values are implemented in the child resource type Microsoft.Web/sites/config. If you redeploy the web app and specify an empty object for the site configuration values, the child resource isn't updated. However, if you provide new site configuration values, the child resource type is updated..."

Please let me know if my statement above reflects the issue you are having or if I am confused.
Thanks.

[frankzo]

To work a bit around this you can use the 'union' function in ARM to combine sets of configuration settings.
First a param to provide an array with settings:

      "appProperties": { 
            "type": "Array",
            "metadata": {
                "description": "Settings."
            }
        }

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.1",
    "parameters": {
        "appProperties": {
            "value": [
                {
                    "name": "Setting3",
                    "value": "Value for setting3"
                },
                {
                    "name": "Setting4",
                    "value": "Value for setting4"
                }
            ]
        }
    }
}

These params can be passed without a param file as well.

Next: in variables the default section and use the 'union' function to combine the param settings and the variable settings.

"variables": {
        "defaultProperties": [
            {
                "name": "Setting1",
                "value": "[parameters('Setting1')]"
            },
            {
                "name": "Setting2",
                "value": "Value for Setting2"
            }
        ],
        "appProperties": "[union(variables('defaultProperties'), parameters('appProperties'))]"

Deploy :

"siteConfig": {
                    "appSettings": "[variables('appProperties')]"

[dgard1981]

I've started to look at Bicep - very cool - and it's so nearly very easy to do this, but not quite.

The plan was to deploy a Function App (parent) and also some App Settings (child) as a separate resource in one Bicep file. I was then going to reference parent.properties from the child resource and use the union() function to merge in some default App Settings, but unfortunately the appSettings from the parent are always null.

Also, while I agree with the sentiment on here around the PUT nature of App Settings being a bad thing, that could be negated if we had a way of pulling down existing App Settings from within an ARM template. I don't think that is possible, but I'd be very glad if I was wrong.

A possible solution to pull down App Settings - The reference() function could accept an HTTP verb to allow for POST requests to Microsoft.Web/sites/config. For example - "[reference('parent/appsettings', '2020-06-01', 'Full', 'POST')]"

Or, better still, it would be good if they existing within parent.properties.

[alex-frankel]

I think the issue in this particular case @dgard1981 is that the Web RP treats appSettings as a secret, so it doesn't support simple GETs (which is what the reference() function is doing).

There may be a list* POST API for retrieving this, but I don't know off hand. Worth looking more closely here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions-resource?tabs=json#list

@bmoore-msft may also know a better way to deal with this

[dgard1981]

Good shout @alex-frankel. There is a list endpoint for Microsoft.Web/sites/config that I can use to get App Settings.

It works in an ARM template -

"appSettings": {
  "type": "object",
  "value": "[list(format('{0}/config/appsettings', resourceId('Microsoft.Web/sites', parameters('functionApp').name)), '2020-06-01')]"
}

But currently fails in Bicep. I created an issue in that repo.

Error BCP082: The name "list" does not exist in the current context. Did you mean "last"?

output appSettings object = list(format('{0}/config/appsettings', functionAppResource.id), functionAppResource.apiVersion)

[bmoore-msft]

@dgard1981 - can you help me understand your scenario a bit more? For context, PUT is declarative so that's why it behaves the way it does...

I'm trying to wrap my head around the case where I have a running functionApp and then want to modify only part of it in the declarative model (i.e. a template). Seems like you risk breaking things that way, but that might be the part I'm missing...

(fwiw, I'm asking because we're trying to see if there needs to be a platform wide fix for this)

[dgard1981]

@bmoore-msft - in this case my main frustration is around slots.

I have separate pipelines to deploy infrastructure and code (which includes App Settings). When the infrastructure pipeline creates a Function App with no App Settings FUNCTIONS_EXTENSION_VERSION is set to ~1. However when the code pipeline carries out a slot swap the value of FUNCTIONS_EXTENSION_VERSION doesn't swap.

So this means that I need to set App Settings for the same Function App in two separate pipelines.

At the moment the infrastructure pipeline includes tasks to set that value through a PowerShell script, as the PowerShell cmdlet will PATCH the App Settings. So it's doable, but more work.

Going forward the dream is to be able to deploy the infrastructure pipeline with some App Settings in the ARM template that are unlikely to change (e.g. APPINSIGHTS_INSTRUMENTATIONKEY, FUNCTIONS_EXTENSION_VERSION, etc.). And then with the code pipeline I can include an ARM template for any other App Settings.

Hope that makes sense.

[bmoore-msft]

@dgard1981 - I think I'm with you... Right now you have 2 pipelines and don't set appSettings in both (which is good IMHO)... but the swapping is not working? If the swap worked then your model is working as expected. Is that right?

Is this "swap not working" recent? I think I saw another user with a similar problem recently - couldn't get slots to swap as expected...

re: your going fwd path, ack... though that's less declarative IMO, but I get it.

[dgard1981]

@bmoore-msft - We have two pipelines and MUST set some App Settings in both due to our use of slots. Slot swapping is working, but using slots complicates our procedure because some App Settings are treated as slot-settings even thought they are not identified as such, e.g. FUNCTIONS_EXTENSION_VERSION.

(Apologies to anyone who falls asleep attempting to read this...)

How we do things at the moment

1. We run a pipeline to deploy our Function App resource (production and slot), and we MUST set FUNCTIONS_EXTENSION_VERSION to ~3 for production. If we don't, it's set to ~1 by default, and as mentioned above, a slot swap does not swap the value of that setting even though we don't identify it as a slot-setting.
2. We run a pipeline to deploy our code to the Function App slot. At the same time we update all App Setting in the slot. Finally we swap production and slot.

The issue with this is that in step 1, we only really need to set FUNCTIONS_EXTENSION_VERSION in production on the first deployment, but as there is no way to test if a resource exists through an ARM template that was always unknown and so we always had to set FUNCTIONS_EXTENSION_VERSION just in case. This mean that all other App Settings were wiped out due to the PUT nature of appSettings in an ARM template, necessitating a deployment of the code after any infrastructure update.

How we can (hopefully) now do things

Now that I know of the list() function, we can make things simpler. In both steps we can use list() to pull back the existing App Settings and then using union() to merge in new/updated settings. This means that an update of our infrastructure won't necessitate a redeployment of our code.

How things can be even easier

If we could chose to PATCH rather than PUT App Settings, we wouldn't even have to worry about the list() and union() functions.

Hope that helps.

[johnwc]

@dgard1981 Do you have the app setting FUNCTIONS_EXTENSION_VERSION set as a "slot setting" so that it stays with the slot and not get swapped? If you do, it should not get updated when you do a swap, ours does not. One thing we did notice though, if that "slot-setting" is applied in both prod slot and dev/other slot, they cancel each other out and get swapped anyways. We have to make sure to only apply the "slot-setting" option for the prod app service resource.

[dgard1981]

@johnwc - With regards to FUNCTIONS_EXTENSION_VERSION from my previous post.

a slot swap does not swap the value of that setting even though we don't identify it as a slot-setting.

I did find a GitHub issue about it once upon a time. Can't remember the full conversation, but the gist was MS know that happens, didn't seem to be by design, but why it was happening was understood so the issue wouldn't be fixed.

[johnwc]

@dgard1981 Ok, so what is the issue then? Are you wanting it to be swapped?

Technically, it shouldn't ever swap that setting, as it belongs to the infrastructure and not any code base.

[dgard1981]

@johnwc - to my mind, if it's an App Setting and as a user I have not specifief it as a slot-setting, then it makes sense that it should be swapped. I get where you are coming from with regards to it being an infrastructure setting but if that's the case then should it an App Setting at all?

Basically, as a user, being unable to PATCH App Settings is very inconvenient and adds complexity where it shouldn't necessary exist.

Another example of this complexity is when developers need to update App Settings along with a code deployment. Not being able to PATCH through an ARM template means that every App Setting has be be set in it. This requires developers to include App Settings around Application Insights and Storage Settings (for example) when they needn't be required if we could PATCH, adding risk that those setting may be incorrectly configured.

[johnwc]

@dgard1981 I agree that there needs to be a way to do incremental updates without wiping the existing settings. But, as for the example you give for FUNCTIONS_EXTENSION_VERSION, it should never be swapped. We have to go the extra mile for our deployments as well and make sure to include it on our config settings in our ARM template. It really should be a property within Microsoft.Web/sites/properties/siteConfig rather an appSetting.