Appcenter-sdk-dotnet: Infinite StorageFullException produce high CPU usage

[YZahringer]

Description

While analyzing a performance issue with our Xamarin Forms UWP applications on production, I detected high CPU usage by
Microsoft.AppCenter SDK. This one produces the StorageFullException in a loop. This uses more than 25% of the CPU continuously.

This happens at startup, without sending any logs or events yet.

The maximum size of the storage has not been defined, the default value is used (10MiB). This one is effectively full, but this should not have any impact on the performance? In which case it can happen?

Repro Steps

Set a full Microsoft.AppCenter.Storage database on UWP and start app center:
cs Microsoft.AppCenter.AppCenter.LogLevel = LogLevel.Verbose; Microsoft.AppCenter.AppCenter.Start( "android=***;ios=***;uwp=***;", typeof(Microsoft.AppCenter.Analytics.Analytics), typeof(Microsoft.AppCenter.Crashes.Crashes));

Details

1. What is your app platform (Xamarin.Android or Xamarin.iOS or UWP)?
   
   
   
   • Xamarin.Forms UWP
   
   
2. If using Xamarin.Forms or if using portable/shared code to call our SDK APIs, are you using shared project, PCL code or .NET standard code for the application? Which .NET standard version or which PCL profile?
   
   
   
   • .NET standard 2.0
   
   
3. Which SDK version are you using?
   
   
   
   • 4.2.0
   
   
4. Which OS version did you experience the issue on?
   
   
   
   • Windows 10 20H2
   
   
5. What device version did you see this error on? Were you using an emulator or a physical device?
   
   
   
   • Physical computer
   
   
6. What third party libraries are you using?
7. Please enable verbose logging for your app using AppCenter.LogLevel = LogLevel.Verbose before your call to AppCenter.Start(...) and include the logs here:

2021-05-23 10:49:04.544 [AppCenter] DEBUG: Creating database at: C:\Users\***\AppData\Local\Packages\***\LocalState\Microsoft.AppCenter.Storage
2021-05-23 10:49:04.546 [AppCenter] DEBUG: AddChannel(core)
2021-05-23 10:49:04.567 [AppCenter] INFO: App Center SDK configured successfully.
2021-05-23 10:49:04.568 [AppCenter] DEBUG: Set max storage size.
2021-05-23 10:49:04.571 [AppCenter] INFO: Found 2560 pages in the database.
2021-05-23 10:49:04.572 [AppCenter] DEBUG: AddChannel(analytics)
2021-05-23 10:49:04.574 [AppCenter] INFO: Changed maximum database size to 10485760 bytes.
2021-05-23 10:49:04.585 [AppCenter] INFO: 'Analytics' service started.
2021-05-23 10:49:04.588 [AppCenter] DEBUG: AddChannel(crashes)
2021-05-23 10:49:04.600 [AppCenter] INFO: 'Crashes' service started.
2021-05-23 10:49:04.605 [AppCenter] DEBUG: Invoke EnqueuingLog event for channel 'core'
2021-05-23 10:49:04.923 [AppCenterAnalytics] DEBUG: SessionTracker.Resume
2021-05-23 10:49:04.925 [AppCenter] DEBUG: Invoke EnqueuingLog event for channel 'analytics'
2021-05-23 10:49:06.839 [AppCenter] DEBUG: Invoke FilteringLog event for channel 'analytics'
2021-05-23 10:49:06.840 [AppCenter] DEBUG: Invoke FilteringLog event for channel 'core'
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:06.935 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:06.954 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:06.978 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.005 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.030 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.066 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.091 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.114 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.132 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.151 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.189 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.210 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.231 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.250 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.271 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.289 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.308 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.328 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.347 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.366 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.385 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.404 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.424 [AppCenter] ERROR: Failed to finalize statement, result=13
Exception thrown: 'Microsoft.AppCenter.Storage.StorageFullException' in Microsoft.AppCenter.dll
2021-05-23 10:49:07.443 [AppCenter] ERROR: Failed to finalize statement, result=13

[YZahringer]

At startup, a startSession is sent on analytics channel. Then, If a StorageFullException is generated, the oldestLog of this channel is retrieved, deleted and then a retry insert on the storage is executed.

The problem occurs when no oldestLog from the channel analytics is present, only logs of crashes channel are present in the full storage for example. In this case, an infinite loop occurs, it tries again to insert the startSession endlessly.
https://github.com/microsoft/appcenter-sdk-dotnet/blob/5a5ec309e8d4900e3f21e619382372e207d3277e/SDK/AppCenter/Microsoft.AppCenter.Windows.Shared/Storage/Storage.cs#L100-L119

A max-retry-attempts must be implemented? try to delete oldestLog in another channel? re-throw StorageFullException when no oldestLog to delete?

[DmitriyKirakosyan]

Hi @YZahringer

Thank you for reporting this issue! We are really appreciate your time and effort in investigating it.

We will try to reproduce it and get back to you with an update.

[DmitriyKirakosyan]

hi @YZahringer

The bug is confirmed.
I prepared a fix and will create a PR soon.
The fix will be available in the next release, which is happening soon, this quarter.

[AnastasiaKubova]

The fix has been released in the 4.3.0 version of App Center SDK. Please update the App Center SDK to the latest version and let me know if this issue still persists.