Aws-sdk-android: DynamoDB is consistently slow when AWSMobileClient instance is used as credentials provider

[markorakita]

Describe the bug
When AmazonDynamoDBClient is created using AWSMobileClient.getInstance() as credentials provider, and then you build DynamoDBMapper from it, every call from DynamoDBMapper takes around 2 seconds.

To Reproduce
On the activity start I call:

• AWSMobileClient.getInstance().initialize() to initialize AWSMobileClient
• AWSMobileClient.getInstance().signIn() to sign in with my credentials

I initialize DDB client and mapper like this:

AmazonDynamoDBClient dynamoDBClient = new AmazonDynamoDBClient(AWSMobileClient.getInstance());
m_dynamoDBMapper = DynamoDBMapper.builder()
                                    .dynamoDBClient(dynamoDBClient)
                                    .awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
                                    .build();

On button click I do:

final long startTime = System.nanoTime();
PaginatedList<ShopItemsDO> models = dynamoDBMapper.scan(ShopItemsDO.class, new DynamoDBScanExpression());
models.loadAllResults();
double durationInMs = (System.nanoTime() - startTime) / 1000000.0;
Log.d(DDB_LOG_TAG, "Scan took: " + durationInMs + "ms");

It consistently prints duration around 1.5s - 2s every time I click the button. Table I use is very small (just 3 items), and duration is same no matter what DDB operation I invoke (scan, query, load one item, save one item, etc). While debugging, I can see that most of that time is spent on getCredentials() call inside invoke() function in AmazonDynamoDBClient, rest of network call to DDB is fast as expected.

Important note: Both Cognito and DynamoDB are set up for eu-central-1 region, and I am also based there. See my comment below for results for us-east-1 region.

Which AWS service(s) are affected?
DynamoDB database, called via DynamoDBMapper from Android AWS SDK.

Expected behavior
I expect DDB calls to take less than 100ms.

Environment Information (please complete the following information):

• SDK Version: 2.16.9 (also every version before that, I see this issue for more than one year...)

[markorakita]

I've written about this in May 2019, and it still happens. This is unusable as it is, 1.5s is extremely high latency.

If you create CognitoCachingCredentialsProvider and use it as credentials provider to construct AmazonDynamoDBClient then everything works fast, but when credentials get stale it crashes.

When you use AWSMobileClient as credentials provider then it updates credentials when they get stale, but it apparently does some very long operation every time getCredentials() is called.

951

[jamesonwilliams]

Hi @markorakita, thanks for this report, and for taking the time to trace the bottleneck into the AWSMobileClient. We will try to investigate this as soon as we are able.

[markorakita]

Sure, thank you for your response Jameson!

In the meantime, I mitigated this issue by constructing AmazonDynamoDBClient using static credentials like this:
AmazonDynamoDBClient dynamoDBClient = new AmazonDynamoDBClient(AWSMobileClient.getInstance().getCredentials());

That way it doesn't have to call getCredentials() from AWSMobileClient every time I make some DDB query. And when some query crashes because credentials got stale, I just recreate AmazonDynamoDBClient with current credentials from AWSMobileClient and do the query again.

But it's definitely a sub-optimal fix, so I am looking forward to your investigation.

[markorakita]

Out of curiosity, I just tested this in different region. For my previous measurement, both Cognito and DynamoDB were based in eu-central-1 region. Now I've tested on project where both Cognito and DynamoDB are based in us-east-1 region, and these are the results for couple successive clicks:

03-17 18:33:49.798 1938-2062/com.markor.awsproba D/DdbPerformance: Scan took: 852.204232ms
03-17 18:33:52.254 1938-2089/com.markor.awsproba D/DdbPerformance: Scan took: 272.249895ms
03-17 18:33:55.163 1938-2093/com.markor.awsproba D/DdbPerformance: Scan took: 311.033385ms
03-17 18:33:57.943 1938-2114/com.markor.awsproba D/DdbPerformance: Scan took: 432.27349ms
03-17 18:34:02.579 1938-2048/com.markor.awsproba D/DdbPerformance: Scan took: 303.087598ms
03-17 18:34:18.506 1938-2062/com.markor.awsproba D/DdbPerformance: Scan took: 354.093333ms

Still far from perfect, but much better than for eu-central-1 region. Also, note how first (warm up) DDB call took much more time than the others. For eu-central-1 region, all calls take 1.5s-2s, not just the first one.

[maxflores]

This is an issue for me as well. I found the AWSMobileClient to be unusably slow even with CognitoCachingCredentialsProvider backing it. I ended up switching to just using CognitoCachingCredentialsProvider but now the credentials become stale after some number of minutes (maybe an hour).

What is the proper usage of CognitoCachingCredentialsProvider to prevent credentials from going stale or detect and handle the case where they do become stale?