Azure-sdk-for-go: GetBlob for blob names contain spaces fail

[avatar29A]

Hi.

I confused. I try to execute this code:

    basicClient, _ := storage.NewBasicClient(accountName, accountKey)
    storageClient := basicClient.GetBlobService()

    listOfBlobs, err := storageClient.ListBlobs("user1", storage.ListBlobsParameters{})

    if err != nil {  // PASS OK
        panic(err) 
    }

    for idx, blob := range listOfBlobs.Blobs {
        blobEntity, err := storageClient.GetBlob("user1", blob.Name);

        if err != nil { // GET ERROR
            panic(err) 
        } 
    }

Invokes two API methods the ListBlobs and GetBlob. ListBlobs's execution result is OK, but when I execute GetBlob, I get error:

storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:0a99ee7e-0001-0117-222e-d584a8000000

Could you help me, please? Thank's.

[ahmetb]

Did you do CreateContainer("user1")?
Container is not the same thing as Storage Account. It is like a folder/directory.

[avatar29A]

Container "user1" was created before.

I executed CreateContainerIfNotExists and got false

isExists, err := storageClient.CreateContainerIfNotExists("user1", storage.ContainerAccessTypeBlob)

For example, this code

listOfBlobs, err := storageClient.ListBlobs("user1", storage.ListBlobsParameters{})
for idx, blob := range listOfBlobs.Blobs {
        fmt.Printf("%v. %v\n", idx, blob.Name)
}

prints a name of files wich stored in container

0. alibra/Unit 2.1.mp3
1. rock/Ozzy - Dreamer.txt

[avatar29A]

May be anyone might to write right code, how download files from container?

[ahmetb]

@avatar29A ah sorry I missed the part that ListBlobs has succeeded for you. Hmm one thing that stands out from your output is, your blob names contain spaces. That might be causing a bug... Does GetBlob fail with the blob names without spaces as well?

@avatar29A there are 2 ways to download a blob:

1. with GetBlob, you will get a ReadCloser, which is essentially the response body of the authenticated call we make to Azure Storage with your storage key.
2. with GetBlobSASURI, you can get a URL which makes a blob readable for a specified period of time and multiple clients can download your blob (without knowing your storage key)

[avatar29A]

@ahmetalpbalkan thanks for answered. I uploaded with files from .net framework, he worked is fine. Tomorrow, I will check your the idea about spaces.

[avatar29A]

I checked it. Yes, if I try to invoke

blob, err := storageClient.GetBlob("user1", "alibra/demo.txt")

It works is fine, but if we'll change blob name on "alibra/Unit 2.1.mp3" for example, we'll get error

panic: storage: service returned error: StatusCode=403, ErrorCode=AuthenticationFailed, ErrorMessage=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

To my mind it's bug. I worked with Azure Storage through .net sdk and didn't get same errors.

[ahmetb]

@avatar29A I think I understood the problem, however it appears like the escaping parts the URL and generating the access signature has turned out to be more complicated than I anticipated when I tried to fix this. I'll leave this open and mark this as a bug. In the meanwhile, I can suggest using blob names not containing spaces as a mitigation.

[bcc]

I also came across this issue - it seems that SharedKeys and SAS Tokens rely on differing escapes depending on the protocol version. I've forked and fixed at https://github.com/bcc/azure-sdk-for-go/commit/026382a38077a643c99412f44cf04e038b465ec6 - and while this has resolved the issue for my use case and the tests still pass, I'd not assume it's a perfect fix. That said, if you want a PR for it, I can see about getting the CLA agreed by my work.

[matryer]

Also having this problem - really need a fix for it. I am passing in the name that Azure gives me:

func TestNamesWithSpaces(t *testing.T) {
    is := is.New(t)
    basicClient, err := storage.NewBasicClient(azureaccount, azurekey)
    is.NoErr(err)
    blobService := basicClient.GetBlobService()
    containersResp, err := blobService.ListContainers(storage.ListContainersParameters{
        MaxResults: 10,
        Prefix:     "",
    })
    is.NoErr(err)
    container := containersResp.Containers[1]
    blobs, err := blobService.ListBlobs(container.Name, storage.ListBlobsParameters{
        Prefix:     "",
        MaxResults: 10,
    })
    is.NoErr(err)
    blob := blobs.Blobs[0]
    log.Println("blob name:", blob.Name)
    props, err := blobService.GetBlobProperties(container.Name, blob.Name)
    is.NoErr(err)
    log.Println("props", props)
}

NOTE: the above code uses github.com/cheekybits/is for testing.

[matryer]

Can help with a fix but I'm not entirely sure what needs doing? I tried escaping the resource path when it builds the signature, but that didn't work, and you don't get any feedback other than "it's still wrong".

[ahmetb]

@matryer if you have a patch which seems to be working, we're happy to accept that.

Sorry for the problem here folks. The main issue is, Go net/http library does extra escaping of certain characters while making the HTTP request, and the path we pass down to Go libraries is what we use to sign the request. In the Azure Storage backend, even though escaped vs. non-escaped versions are equivalent, they are seen as not equals and the authorization fails.

[bcc]

The patch in my comment above certainly works for GetBlobProperties and CopyBlob with a SAS url for blobs with spaces. I've not tested with a new blob yet.

[matryer]

@bcc do you want to do a pull-request with https://github.com/bcc/azure-sdk-for-go/commit/026382a38077a643c99412f44cf04e038b465ec6 - I'd be happy to test it?

[ahmetb]

@bcc it's super easy to write a test (check blob_test.go). Again, if you send a PR we're happy to accept!

[bcc]

@ahmetalpbalkan Sure, I've already updated the tests in my fork - I'll go over them again later and make sure they make sense, then submit a PR. I've also fixed the missing ContentType in GetBlobProperties - do you want that as a separate PR?

[ahmetb]

I've also fixed the missing ContentType in GetBlobProperties - do you want that as a separate PR?

Preferably, thanks. If you could also add assertion for this to the relevant test, that'd be great.

[ahmetb]

This should be fixed with patch #361. Please pull from master and try it. Let us know if this fixes your issue.