Appconfiguration: ConfigurationRefresher Refresh throws unhanded exception when called & hosted within a Service Fabric .Net Core stateful microservice

[MedAnd]

Actual:

Calling ConfigurationRefresher.Refresh() throws unhanded exceptions when hosted in a stateful Service Fabric .Net Core microservice.

Expected:

Calling ConfigurationRefresher.Refresh() within Service Fabric .Net Core microservices (both stateless and stateful) is fully supported.

Exception:

RunAsync reported Warning for property 'RunAsyncUnhandledException'.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.AppConfiguration.Azconfig.AzconfigHandler.d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.AppConfiguration.Azconfig.AzconfigMessageInvoker.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.AppConfiguration.Azconfig.AzconfigClient.d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Extensions.Configuration.AzureAppConfiguration.Extensions.AzconfigClientExtensions.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationRefresher.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at AspNetCore.DataProtection.ServiceFabric.DataProtectionService.d__4.MoveNext() in C:ProjectsAspNetCore.DataProtection.ServiceFabricAspNetCore.DataProtection.ServiceFabricDataProtectionService.cs:line 63
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceReplicaAdapter.d__23.Mov

Example code:

Library versions:

Microsoft.Azure.AppConfiguration.AspNetCore" Version="2.0.0-preview-009470001-12"
Microsoft.ServiceFabric" Version="6.5.664"
Microsoft.ServiceFabric.Services" Version="3.4.664"
Microsoft.ServiceFabric.Services.Remoting" Version="3.4.664"

Links / Related:

https://github.com/Azure/AppConfiguration/issues/100

[abhilasharora]

@MedAnd I just tried creating a stateful service fabric application and ConfigurationRefresher.Refresh() works fine. The unhandled exception above is thrown when we receive a retriable error from the server and all our retry attempts are exhausted. I have a few questions to better understand the scenario here.

1. Could you share the code snippet where the Azure App Configuration provider is initialized?
2. How frequently is the above initialization code invoked in the application?
3. How many key-values does the configuration store contain?

[MedAnd]

@abhilasharora I initially observed the refresh working once but then I started getting this exception... not sure as to the root cause. As a general rule wondering if the method should throw unhandled exceptions, if so this would be good to document both at the code / interface level and also in the docs so that developers can follow a safe pattern, for example should we use a Polly to handle / re-try etc?

There are only half a dozen keys in my App Configuration Service. I am also only updating & using one of those keys in my code. The default refresh interval of 30 secs should apply?

In case I am doing something unsupported would appreciate clarification. As in once I get the simple connection string method working I am also testing the ConnectWithManagedIdentity method after having configured all required Managed Identity / Service Fabric components.

I initially started testing the ConnectWithManagedIdentity method however I received other exceptions and thought best to get the Connect working first.

Have you tested ConnectWithManagedIdentity working in a Service Fabric cluster? My initialization code is provided below:

[abhilasharora]

As a general rule wondering if the method should throw unhandled exceptions, if so this would be good to document both at the code / interface level and also in the docs so that developers can follow a safe pattern, for example should we use a Polly to handle / re-try etc?

IConfigurationRefresher.Refresh() currently throws when the HTTP response code received from the server is 429 (Too Many Requests) or 503 (Internal Server Error). However, these exceptions are currently not exposed and we throw a generic TaskCancelledException for these cases. We plan to move to the official .NET Core SDK (currently in progress) in our next stable release of the configuration provider, where the underlying exceptions would be exposed and included in the public documentation.

There are only half a dozen keys in my App Configuration Service. I am also only updating & using one of those keys in my code.

When the specific keys to be retrieved from the configuration store are not specified using options.Use(), the default query is used that retrieves all key-values from the configuration store, irrespective of the specific key-values referenced in the ConfigureRefresh method.

The default refresh interval of 30 secs should apply?

Yes, the default cache expiration time of 30 seconds is used in the above example.

Have you tested ConnectWithManagedIdentity working in a Service Fabric cluster? My initialization code is provided below:

The managed identity support for service fabric is still in preview. I would suggest creating an issue with the service fabric team for a quicker response on this issue.

[MedAnd]

Has this been fixed, in which release? Thx.

[abhilasharora]

This issue is currently tracked here.

[MedAnd]

These seem to be two different and separate issues. This one is specifically for refresh throwing unhandled exception. Wonder why close this without an incoming fix?

[abhilasharora]

By design, refresh will continue to throw unhandled exception in the scenarios I specified earlier. The type of the exception might change in the first stable release but we don't expect any changes in the behavior. This allows developers to handle cases when all attempts to update the configuration fail instead of failing silently. Could you clarify what is the expectation from this issue?

[MedAnd]

As this is by design, documentation should guide developers on how to handle the exception, assuming it will always be just the one type of exception?

For example should we re-try, with back-off then if still not working should we revert to a cached (stored) last known good configuration, or maybe use a secondary etc?

Please also annotate the code so that intellisense and go to definition contain the type of exception as currently this is not the case, hence why I logged the issue.

Hope above clarifies.... happy to engage further if needed.

PS. I think this will be a great service to use with a microservice architecture!

[zhenlan]

This issue should be resolved with the introduction of TryRefreshAsync API by the latest release of v3.0.0.

[abhilasharora]

Also, we have added documentation to include the exception contract for RefreshAsync method, and it should show up as a part of intellisense. The source code can be found here.