Azure-sdk-for-java: [BUG] Listing blobs with a special character in the blob name fails with 403 AuthenticationFailed

[somanshreddy]

Describe the bug
Listing blobs with a special character in the blob name fails. If we encode the prefix then there is no exception but the listing returns 0 objects. Authentication is via SharedKeyCredential

Exception or Stack Trace

Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:3850e592-601e-00a7-45f2-bbbb36000000
Time:2020-11-16T08:25:51.2139806Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'hMA5oVOIWCMUj/qjM0es8z9ALvkUM1MFC/Dg/htyHvo=' is not the same as any computed signature. Server used following string to sign: 'GET
Mon, 16 Nov 2020 08:25:51 GMT
x-ms-client-request-id:cc000ea8-05b7-4380-aaea-de2ac67c41c4
x-ms-version:2020-02-10
/kompriseqalrs/komprise-somansh
comp:list
delimiter:/
prefix:tmp/ks/1059/Special Files/files with period/weird/JPC/Grants, reports etc/etcfile.txt
restype:container'.</AuthenticationErrorDetail></Error>"

To Reproduce
List Blobs with a prefix that has a special non ASCII character like tmp/ks/1059/Special Files/files with period/weird/JPC/Grants, reports etc/etcfile.txt

Code Snippet

ListBlobsOptions options = new ListBlobsOptions().setPrefix(prefix);
PagedIterable<BlobItem> pagedIterable =  recursive ? container.listBlobs(options, TIMEOUT)
                                                   : container.listBlobsByHierarchy(/*delimiter*/ "/", options, TIMEOUT);
pagedIterable.forEach(action);  // action is a consumer which just adds blobs to a list

Expected behavior
List one blob which is the current blob itself.

This works in v4.0.0

for (ListBlobItem blobItem : container.listBlobs(keyPrefix, recursive, EnumSet.noneOf(BlobListingDetails.class), /*options*/ null, operationContext))

Screenshots
None

Setup:

• OS: Ubuntu 18.04
• IDE : IntelliJ 19.1.4
• Version of the Library used - azure-storage-blob v12.7.0 ( issue seen even in azure-storage-blob v12.9.0-beta.1 and 12.9.0 but would like a workaround for 12.7.0 if possible )

Additional context
On a side issue, creating a blob client for blob names with % fails. I presume this is due to the fact that the Azure SDK does a Url decode internally like Utility.urlEncode(Utility.urlDecode(blobName))) which gives the below stack trace for blob name
Special Files/SingleSpecialCharactersInFileName/.TestFile_%636940838567587622_1%.md

Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: ".m"
    at java.net.URLDecoder.decode(URLDecoder.java:194) ~[na:1.8.0_252]
    at com.azure.storage.common.Utility.decode(Utility.java:99) ~[kcfscloud-3.2.0.jar:na]
    at com.azure.storage.common.Utility.urlDecode(Utility.java:90) ~[kcfscloud-3.2.0.jar:na]
    at com.azure.storage.blob.BlobContainerAsyncClient.getBlobAsyncClient(BlobContainerAsyncClient.java:170) ~[kcfscloud-3.2.0.jar:na]
    at com.azure.storage.blob.BlobContainerAsyncClient.getBlobAsyncClient(BlobContainerAsyncClient.java:151) ~[kcfscloud-3.2.0.jar:na]
    at com.azure.storage.blob.BlobContainerClient.getBlobClient(BlobContainerClient.java:80) ~[kcfscloud-3.2.0.jar:na]

A workaround for this was to encode the % before passing it to the SDK. So we re-used the SDK's built in Url encode
Utility.urlEncode()

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [x] Bug Description Added
• [x] Repro Steps Added
• [x] Setup information Added

[gapra-msft]

Hi @somanshreddy

Thank you for posting this issue. I will take a look at the issue and see if I can repro it.

As for your additional context part, yes the SDK expects a user to encode their URL if it has any special characters.

[gapra-msft]

@somanshreddy

I'm having some difficulty reproing the issue you are seeing. Here is the code I am using to repro

        def b = cc.getBlobClient(generateBlobName() + "ÿ").getBlockBlobClient()
        b.upload(defaultInputStream.get(), defaultData.remaining())

        ListBlobsOptions options = new ListBlobsOptions().setPrefix(b.getBlobName())
        def blob = cc.listBlobs(options, null).iterator().next()

Would it be possible for you to provide some more detailed sample code that I can run to see if I can repro your issue?

[somanshreddy]

@gapra-msft The snippet attached above already covers the area of code that gives an exception. I can post a more detailed sample code that reproduces this issue. But have you tested this in Java? I probably should have added that explicitly in the issue. I have a slight feeling that this might be due to the Java SDK in particular. So would you be able to try that?

Also, regarding the encoding the blob name, I just took a look at the docs. Looks like it was added in the documentation after we used the SDK for the first time. Thanks for that. It will help future users of the SDK avoid the same error.

[somanshreddy]

@gapra-msft Here is the MWE. Would you be able to try this in the Java SDK? Perhaps with the same object key ( though I think the failure is for any special character )

    public void listBlobs() {
        StorageSharedKeyCredential credential = new StorageSharedKeyCredential(accountName, accountKey);
        String endpoint = String.format(Locale.ROOT, "https://%s.blob.core.windows.net", accountName);

        BlobServiceClient serviceClient =  new BlobServiceClientBuilder().endpoint(endpoint).credential(credential).buildClient();
        BlobContainerClient containerClient = serviceClient.getBlobContainerClient(containerName);

        String prefix = "tmp/ks/1059/Special Files/files with period/weird/JPC\uF028/Grants, reports etc\uF029";
        ListBlobsOptions options = new ListBlobsOptions().setPrefix(prefix);
        containerClient.listBlobs(options, Duration.ofSeconds(30)).forEach(System.out::println);
    }

[gapra-msft]

@somanshreddy Thanks for providing the repro code.

Sorry, the code I shared was testing the Java SDK - we use a framework called Groovy to run tests, so it looks a little different.

I will work on reproducing with the code snippet you provided.

[somanshreddy]

@gapra-msft Thanks. Let me know if you need further help in reproducing the issue.

[gapra-msft]

@somanshreddy Thanks for following up. I was able to repro the issue and I am looking into the best way to fix it.

[somanshreddy]

@gapra-msft Thanks. Please let me know if there is something that can be done from the client side when passing this blob name to the SDK. If it is an internal SDK change for which we have to wait for a new release, then we would plan our production deployment accordingly. Would be great if we can get an answer soon.

[gapra-msft]

@somanshreddy
It's definitely a bug internal to the SDK so you will have to wait until our next release (likely next month)

[somanshreddy]

@gapra-msft Is this coming in the Feb release of the SDK?

[gapra-msft]

Yes, this should be released soon. I will update this post as soon as it goes out!

[somanshreddy]

Thanks @gapra-msft, I appreciate it

[gapra-msft]

@somanshreddy

The updated SDK should be out now! Should be azure-storage-blob 12.10.0

[somanshreddy]

Thanks @gapra-msft . Will check it out and let you know :+1:

[timrobinson33]

As for your additional context part, yes the SDK expects a user to encode their URL if it has any special characters.

Just to be clear for anyone else experiencing this problem, from my tests I have concluded you should URL encode the blob name when creating a blob, but you must _not_ URL encode the prefix when using listBlobsByHiererchy or it will return nothing as @somanshreddy pointed out initially.

Also, when URL encoding the blob name on create, you cannot use java.net.URLEncoder because it encodes a space to "+". Azure requires that a space be URL encoded to "%20". This is a not specific to Java because the exact same thing happens in the .Net Azure SDK.