Aws-sdk-ruby: Aws::S3::Errors::XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed. (JRuby threaded)

[krystofspl]

Issue description

Hello, we are using v2 with JRuby (9.1.5.0) and uploading in multiple threads simulataneously.

With v2, in some cases we started to obtain "Aws::S3::Errors::XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.".

The code runs fine if the threads count is reduced to one. Therefore we suspect some part is thread-unsafe...

Gem name

• aws-sdk (2.10.37)
• aws-sdk-core (2.10.37)
• aws-sdk-resources (2.10.37)

Version of Ruby, OS environment

JRuby 9.1.5.0, Mac OS Sierra 10.12.6

Code snippets

Might be relevant, cut from http_wire_trace, subsequent requests (some values are the same across requests):

<ClientComputedContentSHA256>da6b687c2c652a50a6bc0dfd534afa1f32e568a22a1cdd1e1bb961886b0c46e1</ClientComputedContentSHA256><S3ComputedContentSHA256>3ccacf33d63de6a5f672c48907100c7584f576242908b9f5b3c2ca0c5ce95b47</S3ComputedContentSHA256><RequestId>76637B4B5535BBB1</RequestId>

<ClientComputedContentSHA256>3ccacf33d63de6a5f672c48907100c7584f576242908b9f5b3c2ca0c5ce95b47</ClientComputedContentSHA256><S3ComputedContentSHA256>49e6a07d1e7f698c6c3ab2298919a5a09b32fe8504b9123245134a740ea5103e</S3ComputedContentSHA256><RequestId>BB793E70171A2637</RequestId>

<ClientComputedContentSHA256>7e9c8a41a7ad09e5631d1f8466197aa96871f2ab58d2b542362616f904f39678</ClientComputedContentSHA256><S3ComputedContentSHA256>0deac75e6ecf10c6d19d89e88d1f9410fd5f25f069c3843af6d16c39b068858c</S3ComputedContentSHA256><RequestId>A79F833A9777BDE5</RequestId>

<ClientComputedContentSHA256>a0492f77ed5e64d60e33b225980f828bf67a873edb8c740e8ea76aa6d3273929</ClientComputedContentSHA256><S3ComputedContentSHA256>7e9c8a41a7ad09e5631d1f8466197aa96871f2ab58d2b542362616f904f39678</S3ComputedContentSHA256><RequestId>FF52E1C16424B1DC</RequestId>

[cjyclaire]

@krystofspl Thanks for the info, taking a look. Curious if this has to do with autoload, could you try our V3 SDK (which gets rid of auto_load) to see if makes any difference? You can even just try the aws-sdk-s3 gem :)

Another question is, is this error happens every time or occasionally?

[krystofspl]

Unfortunately v3 causes other problems for us and we cannot debug it at the moment. We'll try to look into the v3 compatibility in the future.
It seems to happen every time with mentioned JRuby and multi-threaded use.

[cjyclaire]

@krystofspl Sorry to hear that, curious to know what problems has v3 caused for you?

Adding to original issue, could you provide a code snippet that can help me reproduce the error in a jruby environment? Or are you just calling object.upload and it's causing problem for you?

[krystofspl]

I'm very sorry about the delay, couldn't find the time to get into it.
V3 hangs indefinitely when connecting, but again that might be something with our app, it's quite complex so it's time consuming to debug. Upgrade to v3 is planned in the future for sure, but we need to make it work with v2 for the time being.
Basically we are just calling object.upload_file(...) with :server_side_encryption => 'AES256' with peach (jruby concurrency gem) and >1 threads... Not helpful a lot :-(

[cjyclaire]

Soft ping checking, does this still happens? Any luck in finding more information for reproducing the issue? : )

[krystofspl]

I wasn't able to reproduce the issue since. You can probably close this for now. If something happens, I will report :-) Thanks!

[cjyclaire]

For sure, closing this now. Feel free to reopen if you have further info :)

[ashu106]

I am facing the same issue with aws-sdk-3. It happens occasionally. So, having a really hard time debugging what exactly is the issue

[Divine11]

I am also facing the same issue with aws-sdk-3

[jerryclinesmith]

Once I made this change to how I created the S3::Resource, I haven't seen the error again:
Aws::S3::Resource.new(signature_version: 'v4')

[Divine11]

@jerryclinesmith I tried your suggestion s3 = Aws::S3::Resource.new(signature_version: 'v4') but still I am getting this error. I am using sidekiq to upload file to S3

[mullermp]

Do either of you have any minimal code snippets to reproduce? I am assuming that you both are using JRuby as well as > 1 threads?

[Divine11]

def self.upload_to_s3(file, file_name)
return nil if file.blank? || file_name.blank?
s3 = Aws::S3::Resource.new(signature_version: 'v4')
obj = s3.bucket(BUCKET_NAME).object(file_name)
obj.upload_file(file)
end

This is the function I am using to upload file on S3. Yes, I am using >1 threads.