Azure-sdk-for-net: [BUG][Storage] Blob with encoded space characters in file name cannot be found

[DennisJantos]

Describe the bug
When trying to download a blob, e.g. with blobClient.DownloadAsync() a RequestFailedException occurs with status code 404 if the blob name contains the character %20.

Examples:

• This blob name works: 2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-XXXXXX.json
• This blob name does not work: 2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%20%20XXXXX.json

Expected behavior
Blobs that conain the character %20 in its name can be downloaded successfully.

Actual behavior (include Exception or Stack Trace)
The blob cannot be found and a RequestFailedException with status code 404 (The specified blob does not exist.) and ErrorCode "BlobNotFound" is thrown.

To Reproduce

1. Uploade a json file with at least one encoded space character %20 in the file name.
2. Try to download the blob file with the BlobClient from the Azure SDK.

Environment:

• Azure.Storage.Blobs 12.4.2
• Windows 10, .NET Core 3.1
• Visual Studio 16.5.2

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[seanmcc-msft]

Hi @DennisJantos, unfortunately I'm not able to reproduce your issue.

using Azure.Storage.Blobs;
using System;
using System.IO;
using System.Threading.Tasks;

namespace ConsoleApp2
{
    class Program
    {

        static async Task Main(string[] args)
        {
            string connectionString = "";

            BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
            BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("mycontainer45");
            try
            {
                await containerClient.CreateAsync();
                BlobClient blobClient = containerClient.GetBlobClient("2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%20%20XXXXX.json");
                byte[] buffer = GetRandomBuffer(1024);
                using MemoryStream stream = new MemoryStream(buffer);
                await blobClient.UploadAsync(stream);
                await blobClient.DownloadAsync();
            }
            finally
            {
                await containerClient.DeleteAsync();
            }
        }

        public static byte[] GetRandomBuffer(long size)
        {
            Random random = new Random(Environment.TickCount);
            var buffer = new byte[size];
            random.NextBytes(buffer);
            return buffer;
        }
    }
}

The upload and download requests and responses in Fiddler:

PUT https://seanmcccanary.blob.core.windows.net/mycontainer45/2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%20%20XXXXX.json HTTP/1.1
Host: seanmcccanary.blob.core.windows.net
x-ms-blob-type: BlockBlob
x-ms-version: 2019-07-07
If-None-Match: *
x-ms-client-request-id: 3e3c902e-1e13-4827-91cc-99beb1a8f6cf
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.4.2 (.NET Core 3.1.3; Microsoft Windows 10.0.18362)
x-ms-date: Tue, 12 May 2020 19:35:52 GMT
Authorization: SharedKey ***
Request-Id: |1f67e84c-4ccd9e4638144662.
Content-Length: 1024
[content]

HTTP/1.1 201 Created
Content-Length: 0
Content-MD5: MobWcUk/iAybMNdda2cP1g==
Last-Modified: Tue, 12 May 2020 19:35:52 GMT
ETag: "0x8D7F6ABAE6CE496"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: fd6eb9a2-e01e-0021-7794-2869db000000
x-ms-client-request-id: 3e3c902e-1e13-4827-91cc-99beb1a8f6cf
x-ms-version: 2019-07-07
x-ms-content-crc64: r88hRqSuw4o=
x-ms-request-server-encrypted: true
Date: Tue, 12 May 2020 19:35:51 GMT

GET https://seanmcccanary.blob.core.windows.net/mycontainer45/2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%20%20XXXXX.json HTTP/1.1
Host: seanmcccanary.blob.core.windows.net
x-ms-version: 2019-07-07
x-ms-client-request-id: 81e71912-6265-449d-9ae4-a6be319e3d7d
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.4.2 (.NET Core 3.1.3; Microsoft Windows 10.0.18362)
x-ms-date: Tue, 12 May 2020 19:35:52 GMT
Authorization: ***
Request-Id: |1f67e84d-4ccd9e4638144662.

HTTP/1.1 200 OK
Content-Length: 1024
Content-Type: application/octet-stream
Content-MD5: MobWcUk/iAybMNdda2cP1g==
Last-Modified: Tue, 12 May 2020 19:35:52 GMT
Accept-Ranges: bytes
ETag: "0x8D7F6ABAE6CE496"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: fd6eb9b5-e01e-0021-0994-2869db000000
x-ms-client-request-id: 81e71912-6265-449d-9ae4-a6be319e3d7d
x-ms-version: 2019-07-07
x-ms-creation-time: Tue, 12 May 2020 19:35:52 GMT
x-ms-lease-status: unlocked
x-ms-lease-state: available
x-ms-blob-type: BlockBlob
x-ms-server-encrypted: true
Date: Tue, 12 May 2020 19:35:51 GMT
[content]

[DennisJantos]

Hi @seanmcc-msft,
thanks a lot for trying to reproduce the issue that fast. I also did a test with the code you provided and it also worked for me without any issues.

I could figure out why your code works:
When uploading a blob with the path/name "2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%20%20XXXXX.json" by using await blobClient.UploadAsync(stream);, the characters "%20" are automatically removed or converted to actual spaces so that the file is stored on the blob storage as "2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X XXXXX.json" without %20. Therefore it is possible to download the file wihtout issues with the BlobClient afterwards.

If you manage to store a file, which still contains the %20 characters in the name, the BlobClient won't be able to find and download it. Here is a screenshot of two additional examples in my Azure Storage Explorer:

The issue affects the second file. Everything works with the first one.

How did I manage to upload the files with the %20 characters?
Before I switched to the new nuget package Azure.Storage.Blobs a couple of days ago, I were using the old package Windows.Azure.Storage (v9.3.3).
So I have uploaded the files containing the %20 characters with the following code (simplified):
var path = $@"{someVariable1}-{Uri.EscapeDataString(dataStringToEscape)}-{someVariable3}.json"; var blob = Container().GetBlockBlobReference(path); await blob.UploadTextAsync(data).ConfigureAwait(false);

Workaround
As a workaround it is still possible to find and download the %20 files successfully with the old nuget package Windows.Azure.Storage (v9.3.3) by using the following code:

var legacyContainerClient = CloudStorageAccount.Parse(ConnectionString).CreateCloudBlobClient().GetContainerReference(ContainerName); var blob = legacyContainerClient.GetBlockBlobReference(blobItem.Name); return await blob.DownloadTextAsync().ConfigureAwait(false);

In my case, we are planning to remove the encoding characters %20 from the file names anyways to avoid those kind of issues and to be able to completely migrate our code to the new nuget package. But as long as we have json files with %20 characters in its file name that we need to find and download on the blob storage, we have to stick to the old Windows.Azure.Storage package.

Regards,
Dennis

[seanmcc-msft]

This is by design, and how the Storage Service handles escaped special characters.

If you would like to upload or download a blob literally named 2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%20%20XXXXX.json, you need to manually escape the %s yourself - 2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%2520%2520XXXXX.json

using Azure.Storage.Blobs;
using System;
using System.IO;
using System.Threading.Tasks;

namespace ConsoleApp2
{
    class Program
    {

        static async Task Main(string[] args)
        {
            string connectionString = "";

            BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
            BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("mycontainer45");
            try
            {
                await containerClient.CreateAsync();
                BlobClient blobClient = containerClient.GetBlobClient("2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%2520%2520XXXXX.json");
                byte[] buffer = GetRandomBuffer(1024);
                using MemoryStream stream = new MemoryStream(buffer);
                await blobClient.UploadAsync(stream);
                await blobClient.DownloadAsync();
            }
            finally
            {
                await containerClient.DeleteAsync();
            }
        }

        public static byte[] GetRandomBuffer(long size)
        {
            Random random = new Random(Environment.TickCount);
            var buffer = new byte[size];
            random.NextBytes(buffer);
            return buffer;
        }
    }
}

Fiddler:

PUT https://seanmcccanary.blob.core.windows.net/mycontainer45/2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%2520%2520XXXXX.json HTTP/1.1
Host: seanmcccanary.blob.core.windows.net
x-ms-blob-type: BlockBlob
x-ms-version: 2019-07-07
If-None-Match: *
x-ms-client-request-id: 464d783a-8895-4f55-a006-0e77e6bdb9e3
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.4.2 (.NET Core 3.1.3; Microsoft Windows 10.0.18362)
x-ms-date: Wed, 13 May 2020 19:33:05 GMT
Authorization: SharedKey ***
Request-Id: |72f83799-41c7cdd94c2aca60.
Content-Length: 1024
[content]

HTTP/1.1 201 Created
Content-Length: 0
Content-MD5: 1pIdqzuNLWMKXTe2jiktWw==
Last-Modified: Wed, 13 May 2020 19:33:04 GMT
ETag: "0x8D7F77475084D6D"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: bff641eb-501e-0056-2e5d-29bc4f000000
x-ms-client-request-id: 464d783a-8895-4f55-a006-0e77e6bdb9e3
x-ms-version: 2019-07-07
x-ms-content-crc64: aAiH5tq6uEs=
x-ms-request-server-encrypted: true
Date: Wed, 13 May 2020 19:33:04 GMT

GET https://seanmcccanary.blob.core.windows.net/mycontainer45/2020-05/12/9d35cbd9-5aae-6ee8-bc92-gga232c789e2-X%2520%2520XXXXX.json HTTP/1.1
Host: seanmcccanary.blob.core.windows.net
x-ms-version: 2019-07-07
x-ms-client-request-id: b30ec9d9-a485-4618-8062-a955da67b9fe
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.4.2 (.NET Core 3.1.3; Microsoft Windows 10.0.18362)
x-ms-date: Wed, 13 May 2020 19:33:05 GMT
Authorization: SharedKey ***
Request-Id: |72f8379a-41c7cdd94c2aca60.

HTTP/1.1 200 OK
Content-Length: 1024
Content-Type: application/octet-stream
Content-MD5: 1pIdqzuNLWMKXTe2jiktWw==
Last-Modified: Wed, 13 May 2020 19:33:04 GMT
Accept-Ranges: bytes
ETag: "0x8D7F77475084D6D"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: bff6422c-501e-0056-655d-29bc4f000000
x-ms-client-request-id: b30ec9d9-a485-4618-8062-a955da67b9fe
x-ms-version: 2019-07-07
x-ms-creation-time: Wed, 13 May 2020 19:33:04 GMT
x-ms-lease-status: unlocked
x-ms-lease-state: available
x-ms-blob-type: BlockBlob
x-ms-server-encrypted: true
Date: Wed, 13 May 2020 19:33:04 GMT
[content]

[DennisJantos]

Ok thanks a lot @seanmcc-msft for your fast response and your help!