Aws-sdk-java: Gzip compression doesn't ALWAYS work with DynamoDB (crc32 errors)

[pschmechel]

Generally it works, but once in a while (can't say how often - every 30 seconds or so) I'm getting crc32 errors for DynamoDBMapper.save method (other sdk methods work without problems).
JAVA: 1.8.0_31
SDK: aws-java-sdk-core-1.11.86

Stacktraces:

Caused by: com.amazonaws.internal.CRC32MismatchException: Client calculated crc32 checksum didn't match that calculated by server side
at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:125) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:46) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1501) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1222) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035) ~[aws-java-sdk-core-1.11.86.jar:na]


at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_31]
com.amazonaws.SdkClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1043) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:747) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518) ~[aws-java-sdk-core-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1831) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1807) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1657) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper$SaveObjectHandler.doUpdateItem(DynamoDBMapper.java:854) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper$2.executeLowLevelRequest(DynamoDBMapper.java:594) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper$SaveObjectHandler.execute(DynamoDBMapper.java:733) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.save(DynamoDBMapper.java:623) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]
at com.amazonaws.services.dynamodbv2.datamodeling.AbstractDynamoDBMapper.save(AbstractDynamoDBMapper.java:123) ~[aws-java-sdk-dynamodb-1.11.86.jar:na]

[mezzatto]

Already opened a technical issue about this a fews weeks ago. The response:

Thank you for your inquiry. As of today, you cannot use the AWS SDKs with DynamoDB with compression enabled. We recommend disabling compression when making DynamoDB requests using the AWS SDKs. If compression is critical for your use case, you can bypass the AWS SDKs and use the HTTP APIs to make requests to DynamoDB. We are currently investigating this issue, but cannot comment on the timing for a fix at this time.

[shorea]

Is it possible to capture the wire log when this happens?

http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html

[mezzatto]

Do you have access to private cases? Please refer to https://console.aws.amazon.com/support/home?region=sa-east-1#/case/?displayId=2022632251&language=en

[shorea]

I don't believe so. Can you ask your support contact to open a ticket against the AWS SDK for Java team? They should be able to provide me the details.

[mezzatto]

@shorea done.

[shorea]

Thanks @mezzatto, they reached out to me. I have a suspicion of what may be causing this, setting up a repro case now.

[shorea]

@mezzatto, I was able to reproduce an issue but I'm not sure if it's the same issue you're seeing. Would you be willing to try a hotfix branch in your test environment to see if it fixes those transient crc32 mismatches?

[mezzatto]

@shorea sure! What you need me to do?

[shorea]

Can you try to reproduce this in your environment with this change? https://github.com/aws/aws-sdk-java/tree/issue-1018-test-fix

Should just be able to update your dependencies to 1.11.93 of the SDK and do a mvn install of the aws-java-sdk-core module from this branch.

[mezzatto]

Hi @shorea. Sorry for taking so long to test you patch. I confirm that I'm no longer seeing the CRC32 exception rise. Looks like the code on your branch solved it! 👍

[shorea]

Great to hear! I'll submit an official fix for it this week. I'll update this issue when it's out.

[mezzatto]

@shorea a 1.11.94 version was released with a "Major refactoring of marshallers" (https://github.com/aws/aws-sdk-java/blame/master/CHANGELOG.md#L21). Does that affect the issues we were dealing with in here?

[shorea]

Sorry for the delay @mezzatto. I pushed the fix earlier today so it will go out with today's release. The refactoring is unrelated.

[shorea]

This is available in 1.11.96. Thanks for reporting! Feel free to reopen if you still have issues.