Appconfiguration: Not seeing reliability of enabling a feature flag (Azure Functions)

[kirk-marple]

I've configured my Azure Functions to look for a 'circuit breaker' feature flag, and I'm occasionally seeing it flow through, but in general, it doesn't seem to take effect.

I've read through many samples and the docs, but I can't see anything I'm doing wrong. I even lowered cache expiration to 5sec, with no change.

My feature flag name is "CircuitBreaker" with a label of "Development". You can see it was flipped at 4:47pm, and it's now 5:12pm.

I see the first reference to seeing the flag turned on at 4:53pm, but

Recognized feature [CircuitBreaker] is enabled

But even at 5:08pm, I see:

Recognized feature [CircuitBreaker] is not enabled.

`
private static readonly TimeSpan DEFAULT_REFRESH = TimeSpan.FromSeconds(5);

        var config = new ConfigurationBuilder()
                .SetBasePath(currentDirectory)
                .AddJsonFile("appSettings.json", optional: false, reloadOnChange: true)
                .AddJsonFile($"appsettings.{environmentName}.json", optional: true, reloadOnChange: true)
                .AddEnvironmentVariables();

        this.Configuration = config.Build();

        string connectionString = Configuration[AZURE_APP_CONFIGURATION];

        if (!String.IsNullOrEmpty(connectionString))
        {
            config.AddAzureAppConfiguration(options =>
            {
                options
                    .Connect(connectionString)
                    .Select(KeyFilter.Any, LabelFilter.Null)
                    .Select(KeyFilter.Any, environmentName)
                    .UseFeatureFlags(featureFlagOptions =>
                    {
                        featureFlagOptions.CacheExpirationInterval = DEFAULT_REFRESH;
                    });
                this.Refresher = options.GetRefresher();
            });

            this.Configuration = config.Build();
        }

        providers.AddRange(config.Build().Providers);

        services.AddSingleton<IConfiguration>(new ConfigurationRoot(providers));

        services.AddFeatureManagement(this.Configuration);
        services.AddSingleton<IConfigurationRefresher>(this.Refresher);

`

Here's my helper function which checks if a flag is enabled; I'm seeing all these logs in App Insights. (I tried both IFeatureManager and IFeatureManagerSnapshot.)

`
public async static Task IsEnabledAsync(this IFeatureManagerSnapshot manager, ILogger logger, IConfigurationRefresher refresher, string feature, string category = null)
{
if (String.IsNullOrEmpty(feature))
throw new ArgumentNullException(nameof(feature));

        if (!String.IsNullOrEmpty(category))
            feature = $"{category}.{feature}";

        logger?.LogDebug($"Checking for feature [{feature}].");

        try
        {
            if (refresher != null)
            {
                logger?.LogDebug($"Refreshing feature flags.");

                await refresher.RefreshAsync();
            }

            bool enabled = await manager.IsEnabledAsync(feature);

            if (enabled)
            {
                logger?.LogDebug($"Recognized feature [{feature}] is enabled.");
                return true;
            }

            logger?.LogDebug($"Recognized feature [{feature}] *is not* enabled.");
            return false;
        }
        catch (Exception e)
        {
            logger?.LogException($"Failed to check for feature [{feature}].", e);
            return null;
        }
    }

`
Also, I'm not seeing any throttling or anything, and I'm on Standard plan:

[jimmyca15]

@kirk-marple

If your feature flag has a label of "Development" can you add that option in our UseFeatureFlags call like below?

                    .UseFeatureFlags(featureFlagOptions =>
                    {
                        featureFlagOptions.Label = {the label of your feature flag here};
                        featureFlagOptions.CacheExpirationInterval = DEFAULT_REFRESH;
                    });

[kirk-marple]

@jimmyca15 Thanks for the reply; unfortunately not seeing any difference.

I just flipped my circuit breaker flag off at 12:08:52, and since I have a 5sec cache expiration, it should take effect quickly.

I re-ran my test a minute later, and it still sees it as 'on':

[jimmyca15]

@kirk-marple

Also, make sure not to use IFeatureManagementSnapshot for this case (Azure Function). Unless being used in a framework that supports built in Dependency Injection with scoped contexts, using this interface will result in stale features.

I didn't see before that you mentioned you were using it in your helper function IsEnabledAsync

[kirk-marple]

Thanks, I'd originally used IFeatureManager, but when things weren't working, I tried snapshot, after seeing it in some Azure Function code samples. I'll go back to IFeatureManager now and try this again.

[jimmyca15]

I see, thanks for mentioning that. If you saw IFeatureManagerSnapshot in some of our Azure Function examples then we need to fix them. Were they any kind of Microsoft doc set?

[kirk-marple]

It was here:
https://github.com/Azure/AppConfiguration/blob/e08606a40cc893a5c8bea0f5a7712a7c68d06fc0/examples/DotNetCore/AzureFunction/FunctionApp/ShowBetaFeature.cs

[jimmyca15]

@kirk-marple thank you!

[jimmyca15]

@kirk-marple

In that example that you linked, it does indeed use the version of Azure Functions that supports Dependency Injection and scoped services, so IFeatureManagerSnapshot should not be a problem.

I just cloned that example and ran it to verify everything is working as expected. If you are also using the same Azure Function version as the example you linked and adding feature management to the IFunctionHostBuilder like the example, then IFeatureManagerSnapshot will work. If you are just copying code snippets but using an older version of Azure Function then it could be the problem I mentioned where the features are stale in the IFeatureManagerSnapshot.

[kirk-marple]

@jimmyca15 It's odd, I went back to IFeatureManager, and still not seeing the effect of flipping the flag in the Azure Portal.

environmentName is 'Development', which matches the label.

It seems it's connected properly to App Configuration, since I am initially seeing the flag enabled. It's just when I turn the flag off, I never see that change when checking if it's enabled.

I'm seeing "Refreshing feature flags" and "Recognized feature [CircuitBreaker] is enabled." in the logs.

            if (refresher != null)
            {
                logger?.LogDebug($"Refreshing feature flags.");

                await refresher.RefreshAsync();
            }

            bool enabled = await manager.IsEnabledAsync(feature);

            if (enabled)
            {
                logger?.LogDebug($"Recognized feature [{feature}] is enabled.");
                return true;
            }

And here's how I'm configuring things (and I'm using v3 of Azure Functions):

        var config = new ConfigurationBuilder()
                .SetBasePath(currentDirectory)
                .AddJsonFile("appSettings.json", optional: false, reloadOnChange: true)
                .AddJsonFile($"appsettings.{environmentName}.json", optional: true, reloadOnChange: true)
                .AddEnvironmentVariables();

        this.Configuration = config.Build();

        string connectionString = Configuration[AZURE_APP_CONFIGURATION];

        if (!String.IsNullOrEmpty(connectionString))
        {
            config.AddAzureAppConfiguration(options =>
            {
                options
                    .Connect(connectionString)
                    .Select(KeyFilter.Any, LabelFilter.Null)
                    .Select(KeyFilter.Any, environmentName)
                    .UseFeatureFlags(featureFlagOptions =>
                    {
                        featureFlagOptions.Label = environmentName;
                        featureFlagOptions.CacheExpirationInterval = DEFAULT_REFRESH;
                    });
                this.Refresher = options.GetRefresher();
            });

            this.Configuration = config.Build();
        }

        services.AddFeatureManagement(this.Configuration);
        services.AddSingleton<IConfigurationRefresher>(this.Refresher);

[jimmyca15]

You don't have the feature also defined in your json configuration file do you?

[kirk-marple]

I doublechecked, and I don't. I'd never used a local feature mgmt flag initially.

One note, if I restart my Function App Service, it picks up the proper state of the flag. But I'm still not seeing further changes, if I flip the flag after the app service is running.

Also, in my testing this weekend, I was seeing a mix of flags across instances; almost like it picked up the flag when the instance was spun up, but wouldn't ever get refreshed after that.

Seems like the configuration refresher isn't doing anything, since the initial flag is being read.

[zhenlan]

@kirk-marple did you try to run the example you mentioned earlier? Does it work for you, ie., feature flags are updated properly?
https://github.com/Azure/AppConfiguration/tree/main/examples/DotNetCore/AzureFunction

If the example works for you, we can then see what's different between the example and your app.

[kirk-marple]

I haven't tried that yet, but I did just add this to a command-line app (netcoreapp3.1), with this code below, and it reproduced.

Basically, it is picking up the enabled value when I run the app, but if I flip the flag in Azure Portal (either on or off), it never shows up in the console.

I flipped the flag at: 9/21/2020, 8:43:50 PM, and nothing changes (I've tried letting it run for 5+ min with no changes.

Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:43].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:44].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:45].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:47].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:48].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:49].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:50].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:51].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:52].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:53].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:54].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:55].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:56].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:57].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:58].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:43:59].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:00].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:01].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:02].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:03].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:04].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:05].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:06].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:07].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:08].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T20:44:09].

        private async Task GetFeatureAsync(GetOptions options)
        {
            while (true)
            {
                bool? enabled = await FeatureHelpers.IsEnabledAsync(this.FeatureManager, this.Logger, this.Refresher, options.Name);

                if (enabled != null)
                    Console.WriteLine($"Recognized feature [{options.Name}] {(enabled.Value ? "is" : "*is not*")} enabled [{Helpers.FormatLocalDate(DateTime.UtcNow)}].");
                else
                    Console.WriteLine($"Recognized feature management not configured [{Helpers.FormatLocalDate(DateTime.UtcNow)}].");

                Thread.Sleep(TimeSpan.FromSeconds(1));
            }
        }

            string connectionString = Configuration.GetSection("AzureAppConfiguration")?["ConnectionString"];

            if (!String.IsNullOrEmpty(connectionString))
            {
                string env = GetEnvironmentVariable(ASPNETCORE_ENVIRONMENT);

                if (String.IsNullOrEmpty(env))
                    env = DEVELOPMENT_ENVIRONMENT;

                this.Configuration = new ConfigurationBuilder()
                    .SetBasePath(PlatformServices.Default.Application.ApplicationBasePath)
                    .AddJsonFile("appsettings.json", optional: true)
                    .AddJsonFile($"appsettings.{env}.json", optional: true)
                    .AddEnvironmentVariables()
                    .AddAzureAppConfiguration(options =>
                    {
                        options
                            .Connect(connectionString)
                            .Select(KeyFilter.Any, LabelFilter.Null)
                            .Select(KeyFilter.Any, env)
                           .UseFeatureFlags(featureFlagOptions =>
                           {
                               featureFlagOptions.Label = env;
                               featureFlagOptions.CacheExpirationInterval = TimeSpan.FromSeconds(5);
                           });
                        this.Refresher = options.GetRefresher();
                    })
                    .Build();

                var services = new ServiceCollection();

                services.AddSingleton<IConfiguration>(this.Configuration).AddFeatureManagement();

                using var serviceProvider = services.BuildServiceProvider();

                this.FeatureManager = serviceProvider.GetRequiredService<IFeatureManager>();
            }



md5-e3d3ed866ea578819e5de5b4e4a22234



        public static async Task<bool?> IsEnabledAsync(this IFeatureManager manager, ILogger logger, IConfigurationRefresher refresher, string feature, string category = null)
        {
            if (String.IsNullOrEmpty(feature))
                throw new ArgumentNullException(nameof(feature));

            if (!String.IsNullOrEmpty(category))
                feature = $"{category}.{feature}";

            logger?.LogDebug($"Checking for feature [{feature}].");

            try
            {
                if (refresher != null)
                {
                    logger?.LogDebug($"Refreshing feature flags.");

                    await refresher.RefreshAsync();
                }

                bool enabled = await manager.IsEnabledAsync(feature);

                if (enabled)
                {
                    logger?.LogDebug($"Recognized feature [{feature}] is enabled.");
                    return true;
                }

                logger?.LogDebug($"Recognized feature [{feature}] *is not* enabled.");
                return false;
            }
            catch (Exception e)
            {
                logger?.LogException($"Failed to check for feature [{feature}].", e);
                return null;
            }
        }

[jimmyca15]

@kirk-marple

The problem is this line using var serviceProvider = services.BuildServiceProvider();.

That line is disposing the ServiceProvider that the IFeatureManager is taken out of. The ServiceProvider that is built should not be disposed while any of the services retrieved from it are in use. So can't dispose it as long as the feature manager that is taken from it is being used.

Just remove the word using and your sample will work.

For an azure function you should be fine to keep that service provider around for the lifetime of your function. No need to create + dispose on every function call.

[jimmyca15]

Thanks for the full example by the way, that's the only way I would've caught that issue.

[kirk-marple]

Awesome, thanks all for the help. Now I can see it changing in my CLI.

Recognized feature [CircuitBreaker] is not enabled [2020-09-21T21:39:21].
Recognized feature [CircuitBreaker] is not enabled [2020-09-21T21:39:22].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T21:39:23].
Recognized feature [CircuitBreaker] is enabled [2020-09-21T21:39:24].

I'd had a 'using' in my Azure Function startup class where I was using an extra service provider to grab the service (registered with AddFeatureManagement) so I could register it with Autofac. I haven't tested the Azure Function yet, but should be easy to fix.

[zhenlan]

Closing as the issue seems resolved.