Aws-sdk-android: AWSKeyValueStore - Failed to communicate with keystore service

[rbarbish]

Describe the bug
Occasionally we receive the following exception (see Stacktrace) on calling AWSMobileClient.getInstance().getTokens(), which eventually causes the Cognito user to be signed out (SIGNED_OUT_USER_POOLS_TOKENS_INVALID event) and the "token" and "provider" to be nulled out in the AWSKeyValueStore. It seems to be related to fetching the keys from SharedPreferences, read/write locking on the KeyStore, or ciphers. This is causing users to be randomly logged out of the app.

Stacktrace:

E/AWSKeyValueStore: Error in decrypting data. 
    android.security.keystore.KeyStoreConnectException: Failed to communicate with keystore service
        at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:256)
        at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:148)
        at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:2980)
        at javax.crypto.Cipher.tryCombinations(Cipher.java:2891)
        at javax.crypto.Cipher$SpiAndProviderUpdater.updateAndGetSpiAndProvider(Cipher.java:2796)
        at javax.crypto.Cipher.chooseProvider(Cipher.java:773)
        at javax.crypto.Cipher.init(Cipher.java:1288)
        at javax.crypto.Cipher.init(Cipher.java:1223)
        at com.amazonaws.internal.keyvaluestore.AWSKeyValueStore.decrypt(AWSKeyValueStore.java:435)
        at com.amazonaws.internal.keyvaluestore.AWSKeyValueStore.get(AWSKeyValueStore.java:255)
        at com.amazonaws.mobile.client.AWSMobileClientStore.get(AWSMobileClient.java:3395)
        at com.amazonaws.mobile.client.AWSMobileClient.getSignInDetailsMap(AWSMobileClient.java:937)
        at com.amazonaws.mobile.client.AWSMobileClient.getUserStateDetails(AWSMobileClient.java:952)
        at com.amazonaws.mobile.client.AWSMobileClient.waitForSignIn(AWSMobileClient.java:903)
        at com.amazonaws.mobile.client.AWSMobileClient$11.run(AWSMobileClient.java:1700)
        at com.amazonaws.mobile.client.internal.InternalCallback.await(InternalCallback.java:115)
        at com.amazonaws.mobile.client.AWSMobileClient.getTokens(AWSMobileClient.java:1666)
        at com.drivecentric.drivecentric.utility.DriveApiNg.run(DriveApiNg.java:194)
        at com.drivecentric.drivecentric.activity.SearchCustomersActivity$QuickSearchTask.doInBackground(SearchCustomersActivity.java:415)
        at com.drivecentric.drivecentric.activity.SearchCustomersActivity$QuickSearchTask.doInBackground(SearchCustomersActivity.java:395)
        at android.os.AsyncTask$3.call(AsyncTask.java:378)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:289)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:919)

This is another exception I encounter occasionally that seems to be related:

com.amazonaws.internal.keyvaluestore.KeyNotFoundException: Error occurred while accessing AndroidKeyStore to retrieve the key for keyAlias: com.amazonaws.mobile.client.aesKeyStoreAlias

This exception occurs here - https://github.com/aws-amplify/aws-sdk-android/blob/develop/aws-android-sdk-core/src/main/java/com/amazonaws/internal/keyvaluestore/AWSKeyValueStore.java#L485

This line of code seems to be where "provider" and "token" are coming back as null. https://github.com/aws-amplify/aws-sdk-android/blob/master/aws-android-sdk-mobile-client/src/main/java/com/amazonaws/mobile/client/AWSMobileClient.java#L3395 . At some point "provider" and "token" seem to be getting cleared from the KeyStore.

To Reproduce
Create a while loop with 1000 runs and call getTokens() synchronously. Add user state changed event for SIGNED_OUT_USER_POOLS_TOKENS_INVALID. Set a break point. View the stacktrace.

Which AWS service(s) are affected?
Android AWSMobileClient SDK, Cognito User Pools

Expected behavior
The AWSMobileClient should have no difficulties retrieving the idToken from the KeyStore at any time.

Environment Information (please complete the following information):

• AWS Android SDK Version: 2.16.3, 2.15.1, and 2.12.5 (attempted all)
• Device: Pixel 3a XL
• Android Version: Android 10, API 29
• Specific to simulators: No, many users experiencing this in production, Pixels, Samsungs, Notes

[prechtelm]

Same error here when using the SDK higher than 2.14.2. We need a fix for that asap!!

[dokutoshi]

This is a known issue and our team communicated with AWS that the planned release would not solve the issue.

[kvasukib]

@rbarbish @prechtelm @dokutoshi Sorry for the inconvenience caused.

Whenever there is an issue with the Android KeyStore, AWSKeyValueStore will catch all exceptions and handle them appropriately. In this case, when decrypt fails, the get call will return null and the attributes map in the AWSMobileClient will have null for the key specified.

This is expected behavior given the implementation of Android KeyStore and it's working on different devices and OS levels. I would recommend checking for the event SIGNED_OUT_USER_POOLS_TOKENS_INVALID and presenting a login screen to the user to log them again.

[rbarbish]

@kvasukib - Any plans for an alternate implementation that successfully fetches the cached token 100% of the time? Possibly circumventing the Android KeyStore? This makes for an extremely poor app experience for any implementation of this SDK.

[prechtelm]

@kvasukib couldn't agree more with @rbarbish
This needs to work 100% of the time. Letting the user sign in again is no solution.