Azure-functions-host: Allow External Startup to update the config when Runtime Scale Monitoring is enabled

[shibayan]

What problem would the feature you're requesting solve? Please describe.

I would like to be able to use Key Vault to manage secure secrets in applications using Premium Plan and VNET Integration / Private Endpoint.

In the current version (3.0.14287.0), the host fails to start with the following error, even if Runtime Scale Monitoring is enabled

Microsoft.Azure.WebJobs.Script: The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class. Function 'Function2' uses the modified key(s): DefaultCosmosConnection

Describe the solution you'd like

I understand that the Scale Controller is not restricted in the Function App with Runtime Scale Monitoring enabled, so please change the behavior to the same as when you use the App Service Plan.

https://github.com/Azure/azure-functions-host/blob/dcdd1c251fb041d6828d7b2cdcad99a311518cbe/src/WebJobs.Script/ScriptHostBuilderExtensions.cs#L165

Describe alternatives you've considered

The App Service Key Vault Reference cannot be restricted using Service Endpoint / Private Endpoint, so I want to use External Startup.

[brettsam]

I'll take a look and see if we can relax the restriction.

[fabiocav]

Moving this to sprint 85

[rhythmnewt]

I'm having the same problem. If sprint 85 is complete by 9/30, when will the update be actually available?

[fabiocav]

Full global deployment takes approximately 10 days after sprint completion. This is, of course, dependent on validation results.

The announcements repo here would be the best place to watch: https://github.com/Azure/app-service-announcements/issues

[rhythmnewt]

Will the change affect the regular consumption plan as well? I took a look again and my problem with this is on consumption plan, not premium - sorry!

[jaliyaudagedara]

Hi Guys, any update on this.

Now, we can include appsettings by doing,

override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
{
    FunctionsHostBuilderContext context = builder.GetContext();

    builder.ConfigurationBuilder
        .AddJsonFile(Path.Combine(context.ApplicationRootPath, "appsettings.json"), optional: false, reloadOnChange: false)
        .AddJsonFile(Path.Combine(context.ApplicationRootPath, $"appsettings.{context.EnvironmentName}.json"), optional: true, reloadOnChange: false)
        .AddEnvironmentVariables();
}

and we should be able to use those settings when doing a binding, something like below, isn't it?

[SignalR(HubName = "%SignalR:HubName%", ConnectionStringSetting = "SignalR:ConnectionString")]

But we aren't able to do that.

[Ruud-cb]

Just sharing here that I have similar problems when running in Azure.

  <ItemGroup>
    <PackageReference Include="AutoMapper" Version="9.0.0" />
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="7.0.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="4.0.3" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.9" />
  </ItemGroup>

I recently upgraded this, I didn't face the above issue when I had these versions:

  <ItemGroup>
    <PackageReference Include="AutoMapper" Version="9.0.0" />
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="7.0.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.10" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.9" />
  </ItemGroup>

I actually upgraded because I had trouble that my queue triggers were not synced, hoped that when upgrading them it would solve my problem but it actually created more problems.. It did work last week though, even a restart in Azure doesn't help.

My startup class:

public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            var executioncontextoptions = builder.Services.BuildServiceProvider().GetService<IOptions<ExecutionContextOptions>>().Value;
            var currentDirectory = executioncontextoptions.AppDirectory;
            var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
            IConfigurationRoot config; // using different appsettings for multiple environments, basically ignoring the other appsettings files.
            if (environment == CularBytesEnvironmentEnum.W.ToString()
                || environment == CularBytesEnvironmentEnum.X.ToString()
                || environment == CularBytesEnvironmentEnum.Y.ToString()
                || environment == CularBytesEnvironmentEnum.Z.ToString())
            {
                config = new ConfigurationBuilder()
               .SetBasePath(currentDirectory)
               .AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: false)
               .AddEnvironmentVariables()
               .Build();
            }
            else
            {
               // else use default
                config = new ConfigurationBuilder()
               .SetBasePath(currentDirectory)
               .AddJsonFile("appsettings.json", optional: true, reloadOnChange: false)
               .AddEnvironmentVariables()
               .Build();
            }

            builder.Services.AddAutoMapper(typeof(CoreModelProfile));
            DependencyFunctionInjectionSetup.Setup(builder.Services, config);// => Some more DI config setp

            var connectionString = Environment.GetEnvironmentVariable("ConnectionStrings:DatabaseContext") ?? Environment.GetEnvironmentVariable("DatabaseContext");

            builder.Services.AddDbContext<DatabaseContext>(options =>
                options.UseSqlServer(connectionString));

            builder.Services.AddLogging();

            builder.Services.Replace(ServiceDescriptor.Singleton(typeof(IConfiguration), config));

        }
    }

Found something else in AppInsights:

Breaking change analysis operation failed

System.NotSupportedException:
   at System.Reflection.Emit.InternalAssemblyBuilder.get_CodeBase (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Script.ChangeAnalysis.ChangeAnalysisService.LogBreakingChangeReport (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\BreakingChangeAnalysis\ChangeAnalysisService.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 140)
   at Microsoft.Azure.WebJobs.Script.ChangeAnalysis.ChangeAnalysisService+<TryLogBreakingChangeReportAsync>d__12.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\BreakingChangeAnalysis\ChangeAnalysisService.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 103)

Thereafter it stops by Webhost and stops listening for storage queue updates:

Stopping the listener 'Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener' for function ...

Now trying to revert Microsoft.Azure.Functions.Extensions to 1.0.0, hope that helps, following this issue: https://github.com/Azure/azure-webjobs-sdk/issues/2586

EDIT: Downgrade did solve the problem, still having issues regarding trigger syncing with azure infrastructure, which I am now writing a issue for.

[paulhickman-a365]

I'm also suffering this issue since updating a working production application from version 1.0.0 of Microsoft.Azure.Functions.Extensions.

Do you have any ETA updates on when this will be looked at? The sprint its currently in should have finished 7 weeks ago.

[ahahn95]

I'm getting this issue as well when trying to use queue name bindings for functions that are set from azure app configuration from startup

[stevesage]

We have the same issue with some Timer and Cosmos triggers.

We have a Functions app running dotnet 2.2 on v2 of the functions runtime and we can set timer schedules and Cosmos attributes from app settings configured in startup (in a similar way as described in this comment above: https://github.com/Azure/azure-functions-host/issues/6542#issuecomment-714775515).

If we upgrade the project to dotnet 3.1 and v3 of the functions runtime we get a message like the following:

_The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class.
Function 'Timer1' uses the modified key(s): %TimerSchedules:Timer1%_

[jasonp123]

Hi. We are having a similar issue. We tried creating a simple example:
csproj

 <Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="4.1.0" />
    <PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" Version="3.1.10" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.7" />
  </ItemGroup>
  <ItemGroup>
    <None Update="appsettings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="appsettings.local.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

startup

[assembly: Microsoft.Azure.WebJobs.Hosting.WebJobsStartup(typeof(DemoFunction.OtherStartup))]
namespace DemoFunction
{
    using Microsoft.Azure.Functions.Extensions.DependencyInjection;
    using Microsoft.Extensions.Configuration;
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Text;
    public class OtherStartup : FunctionsStartup
    {
        private bool ApplyWorkaround = true;
        private void ConfigureSettings(IConfigurationBuilder builder, FunctionsHostBuilderContext context)
        {
            builder.
                AddJsonFile(Path.Combine(context.ApplicationRootPath, "appsettings.json"), optional: false, reloadOnChange: false)
                .AddJsonFile(Path.Combine(context.ApplicationRootPath, $"appsettings.local.json"), optional: true, reloadOnChange: false)
                .AddEnvironmentVariables();
        }
        public override void Configure(IFunctionsHostBuilder builder)
        {
            if (ApplyWorkaround)
            {
                OverwriteConfigurationValues(builder);
            }
        }
        public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
        {
            if (!ApplyWorkaround)
            {
                FunctionsHostBuilderContext context = builder.GetContext();
                ConfigureSettings(builder.ConfigurationBuilder, context);
            }
        }
        private void OverwriteConfigurationValues(IFunctionsHostBuilder builder)
        {
            var config = builder.GetContext().Configuration;
            foreach (var item in GetConfigurationValuesForElasticPlan(builder.GetContext()))
            {
                config[item.Key] = item.Value;
            }
        }
        private IEnumerable<KeyValuePair<string,string>> GetConfigurationValuesForElasticPlan(FunctionsHostBuilderContext context)
        {
            var configurationBuilder = new ConfigurationBuilder();
            ConfigureSettings(configurationBuilder, context);
            var configuration = configurationBuilder.Build();
            var values =  configuration.AsEnumerable();
            return values;
        }
    }
}

function1

public static class Function1
    {
        [FunctionName("Function1")]
        public static void Run([ServiceBusTrigger("%myqueue%", Connection = "pubsubconnection")]string myQueueItem, ILogger log)
        {
            log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
        }
    }

function2

public static class Function2
    {
        [FunctionName("Function2")]
        public static void Run([TimerTrigger("%cronpattern%")]TimerInfo myTimer, ILogger log)
        {
            log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
        }
    }

appsettings.json

{
  "pubsubconnection": "[Endpoint=X]",
  "myqueue": "some-queue",
  "cronpattern": "0 */5 * * * *"
}

[patcout26]

I'm getting this issue as well when trying to use queue name bindings for functions that are set from azure app configuration from startup

Finally the function app work only on a app service plan ... not consumption plan

[NSouth]

We are having the same issue. Our TimerTriggers worked fine in v2, but we get this error after upgrading to v3.

[fabiocav]

@brettsam looks like this wasn't closed. Should this move to 92 for investigation?

[sriksmachi]

We are also facing the issue while loading the service bus queue connection name from KeyVault - what is the suggested alternative until the patch is released?

Microsoft.Azure.WebJobs.Script: The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class. Function 'fallout-save' uses the modified key(s): ServiceBusConnectionString .

[fabiocav]

Other items are taking priority over this work, so moving back to _triaged_ until we're able to reassign it.

[fabiocav]

Due to active support cases on this, moving back into the sprint and assigning. Trying to get this in for Sprint 94.

[paulhickman-a365]

Although it will fix the issue for the OP, I don't think this patch will resole the issue for my application and may not fix it for other's who have being experiencing this issue. I am running on elastic premium, but I do not have (or want) Runtime Scaling enabled.

Why is the ability to read configuration from KeyVault (or any other external provider) being linked to runtime scaling being enabled? I don't see why the ability to read sensitive configuration from a secure source should be tied to allowing the functions to scale out to more servers.

[jackbond]

We had to switch from consumption to premium due to this issue. Will all hosting plans be able to override ConfigureAppConfiguration in the future? Seems odd that AzureAppConfiguration is incompatible with any of the function plans.

[ahahn95]

I'm getting this issue as well when trying to use queue name bindings for functions that are set from azure app configuration from startup

Finally the function app work only on a app service plan ... not consumption plan

Are you saying this is fixed now?

[briandunnington]

I know this is closed, but it can still bite people in various scenarios. One fix that worked for us was to change our Startup.cs to use IWebJobsStartup instead of FunctionsStartup - by doing that, it seems to bypass the extra run-time scaler checks. YMMV but it was the simplest fix for us.

[pradeepgururani]

I am using Azure App Configuration service to inject my configuration. I am facing the same problem with consumption plan.

Why are we being forced to use Premium plan?

[munizig]

I'm facing the same problem. I have two functions dealing with Event Hub triggers. One is output and the other one uses an input binder. The output one works perfectly. But the input one throws the same error as this topic.

Microsoft.Azure.WebJobs.Script: The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class.

The function is 3.x version and we are using these package versions.

The configuration is being added this way on Startup.cs:

The function works only when we include the KeyVault reference directly to the Function App Configuration tab.
@Microsoft.KeyVault(SecretUri=full_url)

[TLA020]

Overriding Configuration still gives all kind of issues.

```C#
public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
{
// build a conf to read vault
var config = new ConfigurationBuilder()
.AddEnvironmentVariables()
.Build();

builder.ConfigurationBuilder
    .AddEnvironmentVariables()
    .AddAzureKeyVault(
        config["Vault:Url"],
        config["Vault:ClientId"],
        config["Vault:ClientSecret"]);

}
```

Everything works on a localmachine, but also issues on azure. Functions won't run or log anything (yes loglevel set based on namespace)

Functions runtime error
Microsoft.Azure.WebJobs.Script: The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class. Function 'ImportRssQueueTriggerFunction' uses the modified key(s): ServiceBus:Connection .

[shibayan]

This problem will be resolved when the Function Runtime version is 3.0.15411 or later. Most of the versions are now 3.0.15371, so wait for the latest Function Runtime to be deployed.

[TLA020]

This problem will be resolved when the Function Runtime version is 3.0.15411 or later. Most of the versions are now 3.0.15371, so wait for the latest Function Runtime to be deployed.

When will the latest runtime be deployed?

Do you have a source for this information?

Edit: thanks

[shibayan]

https://github.com/Azure/app-service-announcements/issues/323

[nhwilly]

I'm having a bit of trouble following this. I'm just not familiar with all the flavors things can come in. I'm just running the consumption model on pretty much everything until this app starts to scale.

I just switched over to Azure App Configuration Service which is pretty great. But I also have this problem of not being able to supply values to CosmosDbTrigger attributes and TimerAttributes.

Am I to understand that this latest release corrects the problem for me, too?

And a seriously noob question here: How do I update to the latest version so I can have the fix?

TIA

[jjcv2304]

@shibayan Is there a way to use the new runtime version locally without waiting? I've tried installing the Azure functions core tools pre-release and it doesn't change the runtime version. I also tried adding the variable FUNCTIONS_EXTENSION_VERSION on my local.settings but same result, no change.

[MTCMarkFranco]

This problem will be resolved when the Function Runtime version is 3.0.15411 or later. Most of the versions are now 3.0.15371, so wait for the latest Function Runtime to be deployed.

How do I move to this version? is it in platform or a change in the VS Project?

[jackbond]

@MTCMarkFranco I haven't seen 3.0.15411, but I did see an update to 3.0.15405.0. How much longer would you expect?

[shibayan]

Some regions (West US 2 in my case) had 3.0.15417 deployed, so I was able to confirm that it works correctly. Thank you.

[fabiocav]

@shibayan thank you for confirming. Global deployment should complete this week. It has now completed in all of the Linux stamps and most of Windows.

[TLA020]

Noticed the runtime has been updated, will test soon and let you guys know.

[jjcv2304]

I've updated to the Runtime version 3.0.15417.0 but I still receive a message that says:

The Functions scale controller may not scale the following functions correctly because some configuration values were modified in an external startup class.

In my case I have a ServiceBusTrigger, who's Connection is injected from Application Configuration.
Everything seems to work correctly, but since I moved from Application Settings to Application Configuration I get that warning.
Should this be resolved?

[CSymes]

Similarly to @nhwilly, I'm trying to offload my settings and secret storage to App Configuration while using that data to set up my Function triggers, but am running into this issue as well.
I can confirm I'm running on runtime 3.0.15417.0, on a consumption plan.

[shibayan]

It is expected by design that it will not work on Consumption Plan. Function Premium Plan and Runtime Scale Monitoring need to be enabled.

https://docs.microsoft.com/en-us/azure/azure-functions/functions-networking-options#premium-plan-with-virtual-network-triggers