Aws-sdk-java: org.apache.httpcomponents:httpclient:4.5.7 breaks fetching S3 objects with consecutive slashes in the key

[akhaku]

org.apache.httpcomponents:httpclient:4.5.7 made a change[1] to url rewriting (used when executing a request) where it normalizes the provided urls by removing empty path segments (effectively collapsing consecutive slashes into a single slash).

S3 supports objects with consecutive slashes, but because of this behaviour change in httpclient, you can no longer fetch those objects with the AWS SDK, since the url httpclient makes a request to is different. On a side note, the _manifestation_ of this issue is a signature mismatch, since the signature calculated by the SDK uses double slashes but when httpclient hits the path without double slashes, the calculated signature is different.

It looks like there's some debate on it now and they're actually working on making normalization optional[2], so once that's released we can update the RequestConfig[3] to disable normalization and revert to the previous behaviour[3].

Example stack trace showing the code path to slash collapsing:

rewriteURI:158, URIUtils (org.apache.http.client.utils)
rewriteURIForRoute:231, URIUtils (org.apache.http.client.utils)
rewriteRequestURI:95, ProtocolExec (org.apache.http.impl.execchain)
execute:132, ProtocolExec (org.apache.http.impl.execchain)
doExecute:185, InternalHttpClient (org.apache.http.impl.client)
execute:83, CloseableHttpClient (org.apache.http.impl.client)
execute:56, CloseableHttpClient (org.apache.http.impl.client)
execute:72, SdkHttpClient (com.amazonaws.http.apache.client.impl)
executeOneRequest:1285, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
executeHelper:1101, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
doExecute:758, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
executeWithTimer:732, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
execute:714, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
access$500:674, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
execute:656, AmazonHttpClient$RequestExecutionBuilderImpl (com.amazonaws.http)
execute:520, AmazonHttpClient (com.amazonaws.http)
invoke:4705, AmazonS3Client (com.amazonaws.services.s3)
invoke:4652, AmazonS3Client (com.amazonaws.services.s3)
getObject:1457, AmazonS3Client (com.amazonaws.services.s3)
getObject:1316, AmazonS3Client (com.amazonaws.services.s3)

1: https://issues.apache.org/jira/browse/HTTPCLIENT-1960, https://github.com/apache/httpcomponents-client/commit/8c04c6ae5e5ba1432e40684428338ce68431766b
2: https://issues.apache.org/jira/browse/HTTPCLIENT-1968, https://github.com/apache/httpcomponents-client/commit/4093a3015d6b789888077e317e535df4c8102e5d
3: https://github.com/aws/aws-sdk-java/blob/1.11.502/aws-java-sdk-core/src/main/java/com/amazonaws/http/AmazonHttpClient.java#L1282

[debora-ito]

@akhaku Thank you for the detailed report!

Using org.apache.httpcomponents:httpclient:4.5.7 indeed removed one of the two consecutive slashes in my tests.

Based on the resolution of [2] https://issues.apache.org/jira/browse/HTTPCLIENT-1968, we'll keep an eye on the next httpclient release to see what is the new behaviour of the fix. I'll keep this issue open in the meantime.

[valters]

I am seeing this issue ("AmazonS3Exception: The request signature we calculated does not match the signature you provided.") on S3 key that contains "+" character (that seems the only thing out of ordinary) - like "2019/a+b/item". Could it be related? The regression is caused by upgrading to Spring Boot 2.1.3 which brings in (version manages) httpclient:jar:4.5.7 (from 4.5.6).

[debora-ito]

@valters yes, it looks like keys containing special characters like + were also affected by 4.5.7. We are investigating the whole impact.

[CatalinaMoisuc]

The problem persists with org.apache.httpcomponents:httpclient#4.5.8

[akhaku]

Indeed - #1966 will fix it, just waiting for them to take a look.

[swetavkamal]

This error also comes up while we we call getBootstrapBrokers on AWS MSK(managed streaming kafka).
This error comes up with httpclient 5.5.7 and above while works fine for 5.5.6 or lower.

[debora-ito]

Status update:

We opened a ticket with Apache asking to change the normalization default value to false to reduce the impact on the SDK, but they won't make the change.

Then, while working on the solution, @varunnvs92 found a bug in httpclient 4.5.8 in the URIBuilder#isPathEmpty method.

This bug was fixed and merged, but now the solution on the SDK side is blocked by the next release of httpclient.

We are actively working to fix this. Thank you for your patience.

[debora-ito]

Hey everyone, the SDK version 1.11.596 was released, updating Apache httpclient to 4.5.9.

Please try it out and feel free to reach out if you're still experiencing any issue.

Again, thank you for your patience!

[manoj-pathak]

I am seeing this issue ("AmazonS3Exception: The request signature we calculated does not match the signature you provided.") on S3 key that contains "+" character (that seems the only thing out of ordinary) - like "2019/a+b/item". Could it be related? The regression is caused by upgrading to Spring Boot 2.1.3 which brings in (version manages) httpclient:jar:4.5.7 (from 4.5.6).

I have also faced the exact same issue once I upgraded or downgraded the sprint boot version it worked like a charm.

[efenderbosch]

Was this reverted? I'm seeing com.amazonaws:aws-java-sdk-core:1.11.602 has a dependency on org.apache.httpcomponents:httpclient:4.5.6.

[debora-ito]

@efenderbosch the pom.xml shows 4.5.9
https://github.com/aws/aws-sdk-java/blob/fd13608116c7e257f6c9bccf4dec8d2afa6bedd6/pom.xml#L256-L259

Where are you seeing 4.5.6?

[efenderbosch]

edit: never mind. we had overriden the value in a property due to other conflicts. I swear I searched for that before posting. Apologies.

[mircoianese]

Hey everyone, the SDK version 1.11.596 was released, updating Apache httpclient to 4.5.9.

Please try it out and feel free to reach out if you're still experiencing any issue.

Again, thank you for your patience!

I'm still receiving the warning with version 1.11.603 of aws-java-sdk-s3

[ArielSoftimize]

Hey everyone, the SDK version 1.11.596 was released, updating Apache httpclient to 4.5.9.
Please try it out and feel free to reach out if you're still experiencing any issue.
Again, thank you for your patience!

I'm still receiving the warning with version 1.11.603 of aws-java-sdk-s3

+1

[debora-ito]

@ZombieTheBest @ArielSoftimize

What warning specifically?
Can you give us more details about what you're trying to do, providing a code sample if possible?

[mircoianese]

@ZombieTheBest @ArielSoftimize

What warning specifically?
Can you give us more details about what you're trying to do, providing a code sample if possible?

I've solved it by manually adding the httpclient 4.5.9 dependency on my pom.xml file.
The error was about an old version of httpclient being used. But I had no httpclient dependency on my pom.xml, so I guess that for aws-java-sdk-s3 the library have not been updated yet.

[volphy]

It looks like the same issue as https://github.com/aws/aws-sdk-java-v2/issues/1372

[ArielSoftimize]

@ZombieTheBest @ArielSoftimize

What warning specifically?
Can you give us more details about what you're trying to do, providing a code sample if possible?

In my case, I was trying to send custom metrics to CloudWatch with 'aws-java-sdk-cloudwatch':

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-cloudwatch</artifactId>
    <version>1.11.603</version>
</dependency>

And getting this warning on every request:
com.amazonaws.http.apache.utils.ApacheUtils - NoSuchMethodError was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information

[volphy]

The same warning (see my latest comments to aws/aws-sdk-java-v2#1372)

[imriss]

For me resolved as below:

                <dependency>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-java-sdk</artifactId>
                    <version>1.11.612</version>
                </dependency>

                <dependency>
                    <groupId>org.apache.httpcomponents</groupId>
                    <artifactId>httpclient</artifactId>
                    <version>4.5.9</version>
                </dependency>

[ursNiemi]

What is the last version of AWS SDK that does not have this issue? I have a setup, where upgrading Apache httpclient is not possible, but downgrading AWS SDK is.

[debora-ito]

@ursNiemi What version of httpclient are you using? If you're using httpclient-4.5.9 or newer, it will work with SDK version 1.11.596 or newer.

And just to be clear, the issue was not in SDK but in the changed behavior in Apache httpclient-4.5.7.