Aws-sdk-java: No way to enforce V2 / ceph

[bornmw]

There is ENFORCE_S3_SIGV4_SYSTEM_PROPERTY that enforces V4 signatures.
But there is no way to enforce V2 signatures.

By default S3 SDK uses V4 for "non-standard" endpoints. This practically makes it impossible to use S3 SDK to talk to Ceph since Ceph doesn't support v4 signatures.

It seems like a straightforward fix.

[manikandanrs]

Not all regions support v2 signatures. SDK uses this configuration to decide on the signature version to be used and by default it uses v4.

However for GetObject requests that uses a non standard endpoint, SDK upgrade's to v4. This is because requests that fetch AWS KMS encrypted objects needs to be v4 signed and there is no way SDK can now differentiate a request that fetches a AWS KMS encrypted objects and other objects.

[bornmw]

All that is valid and understood. But makes it incompatible with Ceph, and probably some other use cases. Do you think there's any possibility to add a flag to enforce V2?

[hansonchar]

IIRC, there is a programmatic way to override the signer algorithm via ClientConfiguration:

    ClientConfiguration.setSignerOverride(final String value);

For example, you can do something like:

    ClientConfiguration.setSignerOverride("S3SignerType");

[bornmw]

Unfortunately setSignerOverride doesn't override anything in case of a non-standard endpoint and GET request :(

com.amazonaws.services.s3.AmazonS3Client.createSigner gets signer first (that's when signerOverride comes into play) but then since it is non-standard endpoint it returns v4Signer:

    Signer signer = getSigner();
    ...
    final boolean isStandardEnpoint = getEndpoint().getHost().equals(
            Constants.S3_HOSTNAME);
    ...
    final boolean sigv4ForGetRequests = ((originalRequest instanceof GetObjectRequest) && !isStandardEnpoint);
    ...
    if (
            !(signer instanceof AWSS3V4Signer) &&
            (upgradeToSigV4() || sigv4ForGetRequests)
       ) {
            ...
            return v4Signer;

(it returns in the middle of method body)

Also, (upgradeToSigV4() || sigv4ForGetRequests) calls upgradeToSigV4 method on each GET request irrespective of sigv4ForGetRequests, it would be nice to change it to (sigv4ForGetRequests || upgradeToSigV4()) to skip method call to save some cpu cycles.

[manikandanrs]

@bornmw

A couple of questions

1. What is the endpoint that you are using ?
2. What SDK version are you using ? The latest SDK first derives the signer based on SignerOverride and the endpoint. It skips the upgrade if the signerOverride is set by the user.

The code that you have pasted above is from an older version of the SDK. Can you try upgrading to the latest version ?

[bornmw]

I was on 1.9.9.
Switched to 1.9.23 and it has that logic rewritten, fixing the issue.

Thanks!

[jamshid]

So to sum up, the way to make aws-sdk-jar 1.10.15 work with an S3-compatible service that does not implement V4 signatures is to:

        ClientConfiguration opts = new ClientConfiguration();
        opts.setSignerOverride("S3SignerType");  // NOT "AWS3SignerType"
        AmazonS3Client s3 = new AmazonS3Client(opts);
        s3.setEndpoint("https://storage.example.com");

It's tricky because as mentioned earlier, the SDK really wants to use V4 signatures for GetObject, so most of the operations in https://github.com/awslabs/aws-java-sample work without the override.

At one point in stepping through and trying things that did not work (eg, a custom Region loaded from a resource file with <SignatureVersionOverride>2</SignatureVersionOverride>), I thought SignerFactory.registerSigner("S3SignerType", S3Signer.class); was also necessary, but it seems to work without that.

[wangboak]

Hi,i don't understand why calculateStringToSignV2 use default 'POST',not use request.getMethod()? at com.amazonaws.auth.QueryStringSigner.

[wuseal]

Hi, Everyone, When setting this

        ClientConfiguration opts = new ClientConfiguration();
        opts.setSignerOverride("S3SignerType");  // NOT "AWS3SignerType"
        AmazonS3Client s3 = new AmazonS3Client(getCredProvider(context.getApplicationContext()),opts);

standpoint, It will work ok when upload file below 5 MB ,and when upload file bigger then 5MB .It will throw an exception like this:

com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: AFBCE47F09747899), S3 Extended Request ID: b2LJLc/1GjyflgfrUk90XdPjEZIKUR8EYt8EGHvP/qeBWDXpZAH9atPR2UGb6i/kw+y2wxz1NHI=
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:730)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:405)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:212)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4654)
        at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3485)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.initiateMultipartUpload(UploadTask.java:270)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.uploadMultipartAndWaitForCompletion(UploadTask.java:103)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:83)
        at com.amazonaws.mobileconnectors.s3.transferutility.UploadTask.call(UploadTask.java:47)
        at java.util.concurrent.FutureTask.run(FutureTask.java:237)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:761)

Help! Is there anything i missed?