Aws-sdk-android: [S3][TransferUtility] Key names with special characters are not supported

[rickymohk]

To help us solve your problem better, please answer the following list of questions.

• What service are you using?
  AWS S3
• In what version of SDK are you facing the problem?
  2.6.30
• Is the issue limited to Simulators / Actual Devices?
  no
• Can your problem be resolved if you bump to a higher version of SDK?
  no
• Is this problem related to specific Android/OS version?
  no

• Can you give us steps to reproduce with a minimal, complete, and verifiable example? Please include any specific network conditions that might be required to reproduce the problem.
  Upload a file with key name "[email protected]" using TransferUtility.
  With sdk version 2.2.18, the key name appears correctly as "[email protected]" on s3.
  With sdk version 2.6.30, the key name becomes "abc%40def.com" on s3.

• Please include a stacktrace if applicable.

I want the key name not get automatically url encoded. Is staying in lower sdk version the only way to accomplish? However, I have to bump to 2.6.30 due to API level 28 Apache Commons Logging issue. Please help.
If you need help with understanding how to implement something in particular then we suggest that you first look into our developer guide. You can also simplify your process of creating an application by using Mobile Hub.

[kvasukib]

@rickymohk Sorry for the inconvenience caused. This issue has been fixed in 2.7.4. Please update your app and see if it fixes the issue.

[Huolju]

@kvasukib Actually, in a putObject the key is still encoded in AmazonS3Client.resolveRequestEndpoint and in HttpUtils.appendUri so the problem is still there.

[kvasukib]

@Huolju Sorry for the inconvenience caused. We are looking into fixing it. Currently the SDK uses the URLEncoder to encode the URL with the key name in it and un-encodes (escapes) certain characters. What is the key you are trying to upload?

[Huolju]

@kvasukib Hi thanks for the quick answer, workgroup/[email protected].

[ufoq]

Also you can try any athena path like year=2018/month=09/etc url is encoded once in URLEncoder, then some characters like '/' are decoded back, but some like '=' are not, then they are again encoded in HTTP client in appendUri, what makes some part of paths to be double encoded

[ufoq]

Also for workaround I use now something like:

class FixedS3Client extends AmazonS3Client {

    FixedS3Client(AWSCredentialsProvider credentialsProvider) {
        super(credentialsProvider);
    }

    @Override
    public void resolveRequestEndpoint(Request<?> request, String bucketName, String key, URI endpoint) {
        super.resolveRequestEndpoint(request, bucketName, key, endpoint);
        // just write path as it is here because it's going to be encoded later anyway
        // what results in double encode in case of default aws implementation
        request.setResourcePath(this.getHostStyleResourcePath(key));
    }

    private String getHostStyleResourcePath(String key) {
        String resourcePath = key;
        if (key != null && key.startsWith("/")) {
            resourcePath = "/" + key;
        }

        return resourcePath;
    }
}

It makes S3Client#putObject work ok, virtual path with '=' characters is correct in aws console

[kvasukib]

@mUfoq Thank you for the analysis. The SDK is currently doing double encoding which is the cause for the issue. We are looking into fixing it. I will post an update once the fix is out.

[Daxito]

Also facing the same issue here, I hope there is a fix soon

[kvasukib]

@rickymohk @Huolju @mUfoq @Daxito We have fixed this issue in 2.11.0 version of the SDK. The fix includes removal of the additional layer of encoding and escaping done by the AmazonS3Client. Hence this change requires testing on your end before upgrading your app with the new SDK.

Please read our CHANGELOG and Documentation for detailed information on the change.

Please let us know if you have any further questions/comments on the fix.

Tagging the folks who raised the issue:

issue #321 @maxbax @defaultbr @kwong93 @driskell @BenCactus @ioannist @kwong93
issue #360 @TakkuMattsu
issue #545 @depak330
issue #597 @mvievardassystem

[rickymohk]

The fix works. Thank you very much.

[frankmuellr]

@rickymohk, excellent, thank you for confirming. We'll leave this issue open for a few more days, so others can respond also.

[frankmuellr]

@rickymohk @Huolju @mUfoq @Daxito - I am going ahead and closing this issue as @rickymohk confirmed that the fix worked for him. Please open another issue if you are still experiencing problems.