Azurestorageexplorer: large file uploads fail with SAS URI connected container

[eschultink]

Storage Explorer Version: Both 1.1 and 1.2 (tried both)
Platform/OS Version: macOS and Windows (we've had people try on both platforms, although I'm personally on macOS)
Architecture:
Version: 1.2.0

Platform: macOS
Architecture: x64
Build Number: 20180604.2
Commit: f8373d5a
Support Id: 87e505ce-9d60-4576-5c5e-dfb4dccd49a9

Copyright © 2018 Microsoft. All rights reserved.
Regression From: Not sure it ever worked. We've tried on various machines and various versions.

Steps to Reproduce:

1. create a SAS URI for a bucket, with wdl perms
2. connect it by right clicking on "Storage Accounts" --> "Connect ..", choosing SAS URI method, pasting in the URI in there. Otherwise all defaults.
3. Upload with defaults. Small file works; Large file fails.

{
  "name": "StorageError",
  "message": "Forbidden",
  "stack": "StorageError: Forbidden\n    at Function.StorageServiceClient._normalizeError (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/azure-storage/lib/common/services/storageserviceclient.js:1191:23)\n    at Object.StorageServiceClient._processResponse (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/azure-storage/lib/common/services/storageserviceclient.js:738:50)\n    at Request.processResponseCallback [as _callback] (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/azure-storage/lib/common/services/storageserviceclient.js:311:37)\n    at Request.self.callback (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/request/request.js:185:22)\n    at emitTwo (events.js:126:13)\n    at Request.emit (events.js:214:7)\n    at Request.<anonymous> (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/request/request.js:1157:10)\n    at emitOne (events.js:116:13)\n    at Request.emit (events.js:211:7)\n    at IncomingMessage.<anonymous> (/Applications/Microsoft Azure Storage Explorer.app/Contents/Resources/app/node_modules/request/request.js:1079:12)",
  "code": "Forbidden",
  "statusCode": 403,
  "requestId": "f09047e2-501e-0013-070d-04d352000000"
}

Expected Experience:

It should work for both small and large files.

Actual Experience:

It works only for small files.

[jinglouMSFT]

What's the approx size of the larger files to repro this? Also there was a period of time yesterday Azure Storage was down.

[eschultink]

So with a SAS token that also has READ permissions, it works. Given this, I would guess that it uses a different upload mechanism for large files rather than small files; that upload mechanism tries to READ somehow, which fails?

[eschultink]

70MB+

[eschultink]

it is current, happens now.

[craxal]

Yeah, this is a known issue. Here's what happens.

For particularly large files (like your 70+MB), we want to know if a blob of that name already exists before trying to upload the file. Otherwise you could be waiting a few minutes just for the platform to tell you the blob already exists before dropping the uploaded file!

So, before uploading anything, we just ask the platform if the blob exists using the Get Blob Properties operation. Unfortunately, this means you need read permissions. In most cases, this isn't an issue. But in your case, your SAS needs read permissions. Otherwise larger uploads will fail.

See #140 for more info.

[eschultink]

OK, very confusing to have it work differently depending on the file-size - especially when it provides no useful feedback about the error.

The original token did have LIST permission as #140 indicates is required - just not WRITE.

Use-case is that I want customers to be able to securely upload data files. Hence we send them a URI that allows them to write only - so they can send us the files, but if that URI was ever compromised, it would not allow someone to read those files.

Also strange that the "Create SAS" wizard allows WRITE ONLY tokens to be created, if those have no utility.

[craxal]

In your case, the key is read permissions, not list.

We'll need to spend some time looking into this. We've gotten reports of this several times, so we'll want to look at our options. It looks like the platform supports just overwriting blobs if they already exist, but that may be undesirable in some cases. I'm sure, for example, you don't want users overwriting other users' blobs when they happen to upload something with the same name.

[eschultink]

well, we're not going to hand different customers SAS tokens for the same container. We create a container per customer. So that risk is minimal. And in any case, I think it's our responsibility to consider that rather than yours.

But regardless:

1. the behavior of this shouldn't vary based on file-size. If READ is needed to WRITE, it should ALWAYS be required to write. Otherwise I create a SAS token, test it, it works fine for me - so I send to customer - and then it fails for them. We look stupid.
2. if you allow a write-only token to be created, it should work for writing - the potential overwrite case is the user's problem. If you don't want to allow that, your UX shouldn't allow me to check the "WRITE" box without also having checked the "READ" box. Mysterious failures on the other end isn't correct.
3. there should be some permission short of READ that allows for reading of blob meta-data, but not the blob itself. I thought that's what the LIST permission did, but I guess not?

[craxal]

List permissions let you ask the container what blobs it has. Read permissions let you get specific data from the blob, including content and metadata.

Don't worry; we're with you on this one! We're aware of the problem, and we intend to fix it without breaking other user needs. We've been chatting, and we think we have a simple solution. Theoretically, we should be able to just query the container for blobs using the target blob as a prefix. If there are no matching blobs returned, then we don't have to worry about overwriting. This should release the requirement for read permissions.

[craxal]

I've just merged a potential fix for this issue. @eschultink, @grundencp, @chriseul, would you be interested in trying a preview build and verifying that this fix resolves your issues?

[eschultink]

Yeah, happy to give it a try.

[grundencp]

Sure. It's still not working for me.

[craxal]

Here are some links to preview builds you can use. Keep in mind they will expire in 7 days.

• Windows
• Mac
• Linux

[grundencp]

Thanks craxal! The preview build worked for me.

[MRayermannMSFT]

@eschultink , please let us know if the build did not work for you. Otherwise, we're closing this bug, and you will see the fix officially ship in 1.3.0. Thanks!

[seacen]

@craxal @MRayermannMSFT may I know the timing of this 1.3 release? We are having the same issue currently and have to change our SAS token permissions.

[seacen]

Can I also know if this issue applies to uploading via Blob APIs as well, or just GUI only?

[craxal]

If you're sending your own blob upload requests, you should only need write permissions, so this is likely just a bug in Storage Explorer. Other CLI tools, such as AzCopy, don't have the same issue.

You can expect 1.3.0 out sometime this week.

[unruledboy]

V1.6.2 with default access key 1 (not SAS) still has the same problem, any idea?

reading other issues, using exp. AzCopy looks working, but it did not, it says: 122%, still uploading.

If I use Azure Portal directly uploads, it will show failed message: "Upload block blob to blob store failed. Details: Make sure blob store SAS uri is valid and permission has not expired. Make sure CORS policy on blob store is set correctly. StatusCode = 0, StatusText = "

[A141864]

I'm experiencing the same issue via the Azure Portal
Upload block blob to blob store failed. Details: Make sure blob store SAS uri is valid and permission has not expired. Make sure CORS policy on blob store is set correctly. StatusCode = 0, StatusText = .

[craxal]

Closing due to inactivity. We've just release 1.11.0, so please upgrade and try again. If you continue to experience issues, please leave a comment, and we will revisit.

[skanalytic]

I am still experiencing an issue with 'WRITE ONLY' SAS tokens on Azure Storage Explorer. If I create an SAS token with 'read', 'list', 'write' (rlw) then I can connect to my Azure File Storage account and upload/download files no problem, but if I follow this exact same process without 'read' permissions - so just list and write (lw) - then I can still connect to the account but not longer upload (even though I should be able to with 'write').

I verified this issue does not exist using the same SAS token on Azcopy (e.g. with just 'wl' permissions i can upload but not download files) so I think it is a specific bug with Azure Storage Explorer.

I also beleive this is the exact same issue mentioned here: https://github.com/microsoft/AzureStorageExplorer/issues/305#issuecomment-397409292

Any support of this would be much appreciated.
Happy to provide more details if needed.

[skanalytic]

@eschultink did you ever figure out a solution to this https://github.com/microsoft/AzureStorageExplorer/issues/305#issuecomment-397409292

[craxal]

@eschultink What you're experiencing is really by design. Storage Explorer was never meant to work with write-only SAS. The explorer always tries to show blobs/files in a container, which requires list permissions. Read permission is also required, because we use it for certain checks. If you would still like Storage Explorer to work with write-only SAS, please open a new feature request issue.

[skanalytic]

OK cheers @craxal, I will do that and have found a work-round for now. Thanks for replying.

Just in general though - it does seem a bit counterintuitive to me that Azure will allow me to make an SAS token with just write permissions, will tell me I have just write permissions when I connect this token to Storage Explorer, but then will forbid actions that seem like just write tasks (e.g. uploading a file) without any warning or useful error message.

Anyway, thanks again - I will post a proper feature request.

[skanalytic]

FEATURE REQUEST OPENED HERE: https://github.com/microsoft/AzureStorageExplorer/issues/2793