Azure-sdk-for-net: Transient error in key storage refresh at Azure blob every 24h

[jaaant]

Describe the bug
I'm trying to set up data protection using Azure blobs for key storage. After encountering an issue with the Microsoft.AspNetCore.DataProtection.AzureStorage package (https://github.com/dotnet/aspnetcore/issues/21569), I was directed here to try the Azure.AspNetCore.DataProtection.Blobs and Azure.AspNetCore.DataProtection.Keys packages. They mostly work.

However, every 24 hours, when the service tries to refresh the keys, it gets an error "Root element is missing." from handling the blob. After that, on the next attempt the problem disappears, logs show that calls to storage account and key vault succeed, and everything works without problems for the next 24 hours.

(Apologies for poor code formatting in this ticket, but the insert code syntax seems to throw away all line changes, which is worse.)

Expected behavior
Key refresh works without errors every time.

Actual behavior (include Exception or Stack Trace)
The first key refresh attempt every 24 hours fails, with details below. An immediately repeated attempt succeeds and everything works until the next refresh.

Message: Root element is missing.
Exception type: System.Xml.XmlException
Failed method: Azure.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.CreateDocumentFromBlob
FormattedMessage: An error occurred while refreshing the key ring. Will try again in 2 minutes.
AspNetCoreEnvironment: Production
CategoryName: Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider

System.Xml.XmlException:
at System.Xml.XmlTextReaderImpl.Throw (System.Private.Xml, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.Xml.XmlTextReaderImpl.ParseDocumentContent (System.Private.Xml, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.Xml.XmlTextReaderImpl.Read (System.Private.Xml, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.Xml.Linq.XDocument.Load (System.Private.Xml.Linq, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.Xml.Linq.XDocument.Load (System.Private.Xml.Linq, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Azure.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.CreateDocumentFromBlob (Azure.AspNetCore.DataProtection.Blobs, Version=1.0.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at Azure.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository+d__14.MoveNext (Azure.AspNetCore.DataProtection.Blobs, Version=1.0.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Azure.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.StoreElement (Azure.AspNetCore.DataProtection.Blobs, Version=1.0.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.CreateNewKey (Microsoft.AspNetCore.DataProtection, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.CreateNewKey (Microsoft.AspNetCore.DataProtection, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore (Microsoft.AspNetCore.DataProtection, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.ICacheableKeyRingProvider.GetCacheableKeyRing (Microsoft.AspNetCore.DataProtection, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore (Microsoft.AspNetCore.DataProtection, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

To Reproduce
The application has been given a system assigned managed identity, with Storage Blob Data Contributor role in the storage account. Data protection is set up as follows:

string blobUri = $"https://{storageAccountName}.blob.core.windows.net/{containerName}/{blobName}";
string keyId = $"https://{keyVaultName}.vault.azure.net/keys/{keyName}/";
services.AddDataProtection()
.SetApplicationName( $"{applicationName}" )
.ProtectKeysWithAzureKeyVault( keyId, new DefaultAzureCredential() )
.PersistKeysToAzureBlobStorage( new Uri( blobUri ), new DefaultAzureCredential() );

Environment:
Library packages: Azure.AspNetCore.DataProtection.Blobs and Azure.AspNetCore.DataProtection.Keys, both versions 1.0.0-preview.1

IDE: Microsoft Visual Studio Professional 2019, Version 16.4.5

dotnet --info prints the following:

.NET Core SDK (reflecting any global.json):
Version: 3.1.101
Commit: b377529961

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18363
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.101\

Host (useful for support):
Version: 3.1.1
Commit: a1388f194c

.NET Core SDKs installed:
1.0.0-preview2-003131 [C:\Program Files\dotnet\sdk]
2.1.101 [C:\Program Files\dotnet\sdk]
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.508 [C:\Program Files\dotnet\sdk]
2.1.801 [C:\Program Files\dotnet\sdk]
2.1.802 [C:\Program Files\dotnet\sdk]
2.2.207 [C:\Program Files\dotnet\sdk]
3.1.101 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[amnguye]

Hi, Just read through the issue. This look like it's not related to the Storage Client SDK libraries, I'm going to tag ASPNET Extensions and see what they think.

If this is related to the Storage Client Libraries, please add the Storage tag back. Also please include the customer's version of the Azure.Storage.Blobs package and Azure.Core package that are being used. Thanks in advance.

[amishra-dev]

@kurtzeborn do you know how this should be redirected. The bug is in the Microsoft.AspNetCore.DataProtection.AzureStorage package

[jaaant]

Azure package versions in the service are as follows:

Azure.AspNetCore.DataProtection.Blobs: 1.0.0-preview.1
Azure.AspNetCore.DataProtection.Keys: 1.0.0-preview.1
Azure.Core: 1.0.2
Azure.Identity: 1.1.1
Azure.Security.KeyVault.Keys: 4.0.1
Azure.Storage.Blobs: 12.0.0
Azure.Storage.Common: 12.0.0

[masterjs]

I've also been struggling with this error for a few weeks that I see in my logs that makes a request fail, but only once in a while.

[jsquire]

//cc: @tg-msft, @pakrym. Any assistance with helping route to the appropriate owner would be appreciated.

[jaaant]

Any news on this? It would be good to know if a fix is coming, or if I need to look for a different solution or workaround.

[dbruning]

Looking through App Insights logs, this error always seems to follow a 304 response from BlobBaseClient.Download()

Just in case that's useful.

[tg-msft]

I think the IfNoneMatch condition is causing a problem because the DownloadTo effectively ignores the 304 and the rest of the code doesn't hit Response.Value.

Assigning to @pakrym to investigate adding a check similar to IsUnavailable before proceeding, instead of trying to catch a 304.

[pakrym]

the DownloadTo effectively ignores the 304

Ugh, that's pretty unexpected.

Thank you for the investigation!