Azure-storage-azcopy: The latest version does not work for local file - blob (using SAS)

[damosullivan]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.3.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

copy .\test.txt https://numberseus.blob.core.windows.net/builds/1.0.11399/ChromeTests.7z?api-version=2016-05-31&si=upload-builds&sig=-REDACTED-&sr=c&sv=2014-02-14 

What problem was encountered?

403 This request is not authorized to perform this operation using this permission.

How can we reproduce the problem in the simplest way?

Generate a container SAS, and try to upload a file. This does not work. This worked on version 10.2.1 (which I have locally and can confirm)

Have you found a mitigation/solution?

I will use 10.2.1 until this is resolved.

[zezha-msft]

Hi @damosullivan, thanks for reaching out!

Could you please double check your SAS? How did you generate it? I see the query parameter api-version=2016-05-31, which doesn't look right.

I tried with a SAS (based on access policy) generated from Storage Explorer, and it works fine.

./azcopy cp ./cmd/copy.go "https://zemaintest.blob.core.windows.net/foo?sv=2018-03-28&si=foo-16DC8E72595&sr=c&sig=REDACTED"
INFO: Scanning...

Job 8db69622-099a-7d4b-7321-52f21f190766 has started
Log file is located at: /Users/zed/.azcopy/8db69622-099a-7d4b-7321-52f21f190766.log

0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 


Job 8db69622-099a-7d4b-7321-52f21f190766 summary
Elapsed Time (Minutes): 0.0334
Total Number Of Transfers: 1
Number of Transfers Completed: 1
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 61092
Final Job Status: Completed

[damosullivan]

Hi, thanks for getting back to me.

Seeing that it worked for you, I said I'd try it again. I used a different storage account (still _Classic_), and created a new Access Policy, then generated a new SAS.

This worked with 10.3.0. Nice.

I then went back to the original container that was causing trouble. I used the same Access Policy (did not create a new one), and generated a new SAS. I then went to upload a file and this did not work. However, it did work for 10.2.1.

I then created a _new_ Access Policy to test a theory. I then generated a new SAS and used this to upload a file. This again worked on 10.2.1, but did not work on 10.3.0.

I then used a different container within the same storage account. Created a new Access Policy, Generated SAS, attempted to upload a file. Again, worked on 10.2.1, did not work with 10.3.0.

Here is the error:

Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)
   HEAD https://numberseus.blob.core.windows.net/jan/test.txt?si=damientest&sig=-REDACTED-&sr=c&sv=2018-03-28&timeout=901
   User-Agent: [AzCopy/10.3.0 Azure-Storage/0.7 (go1.12; Windows_NT)]
   X-Ms-Client-Request-Id: [30d9e538-9718-4214-50bc-97d12b343365]
   X-Ms-Version: [2018-03-28]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
   Date: [Mon, 14 Oct 2019 22:29:22 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   Vary: [Origin]
   X-Ms-Error-Code: [AuthorizationPermissionMismatch]
   X-Ms-Request-Id: [9a8b496d-401e-0115-46de-826028000000]
   X-Ms-Version: [2018-03-28]

I believe there is a bug introduced as I can still upload a file using the very same SAS in 10.2.1. I am happy to try anything else you can think of. This appears to be a problem in our classic storage account numberseus.

Damien

[damosullivan]

(I am also managing Access Policies and SAS via Azure Storage Explorer)

[JohnRusk]

@zezha-msft I'm going to mark this assigned to you, since you've replied above, but please bounce it back to me if you don't have time.

[zezha-msft]

Hi @damosullivan thanks a lot for doing the experiments!

Could you please share with us which permissions did you give to the access policy?

[damosullivan]

The Access Policy I used above to test had Add, Create, Write, an List.

[zezha-msft]

Thanks @damosullivan!

@adreed-msft this seems to be a behavior change from the refactoring, where the read access became needed for upload (probably to check whether it's a virtual dir), could you please take a look and confirm?

[adreed-msft]

Not at my desk right now so this is off memory

That's only the case on download or S2S, for the source.

The directory check on the destination, for Blob at least, is a syntax check.

I'll go check for real in a moment.

[damosullivan]

It appears read is now needed for an upload. Without the read it fails.

INFO: Scanning...

Job 404decbf-e9f5-9b4f-6a2d-a21fe16e09dc has started
Log file is located at: C:\Users\damien.NUMDOM2\.azcopy\404decbf-e9f5-9b4f-6a2d-a21fe16e09dc.log

0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,


Job 404decbf-e9f5-9b4f-6a2d-a21fe16e09dc summary
Elapsed Time (Minutes): 0.0333
Total Number Of Transfers: 1
Number of Transfers Completed: 1
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 19
Final Job Status: Completed

[damosullivan]

@adreed-msft I am uploading

[adreed-msft]

Ah! this isn't in the enumerators! This is as a result of destination length validation being on by default.

Could you try with the flag --check-length=false?

[damosullivan]

It works without read and with --check-length=false, thanks.

Do you think false should be the default?

[adreed-msft]

check-length is one of the ways that we attempt to verify that the destination data is in-tact. I don't think it should be false by default, as a result.

[adreed-msft]

It's not a _perfect_ check, but it does help in situations where we can't reliably get a MD5 to check.

[damosullivan]

Check the length of a file on the destination after the transfer. If there is a mismatch between source and destination, the transfer is marked as failed.

I understand a bit more now after reading the help. However it is not intuitive. Do you think you could maybe better handle the error?

Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)

Something like "Could not verify file was uploaded successfully. Needs read access. Use --check-length=false to skip this verification" <- I'm sure you can do much better than my attempt 🍰

[adreed-msft]

If you take a look in the log, we do actually log the error on a length check fail.

You're definitely right though, the stdout output is a little weak.

[damosullivan]

403 This request is not authorized to perform this operation using this permission.. When Upload Length check: Get destination length.

I see it now in the log. Thanks. I will update my tools to use version 10.3.0 again and just disable the check (scripts do not need read for my use case).

Thanks for your help,
Damien

[adreed-msft]

You're welcome! I'll keep this issue open and tag it as logging related.

[loganville]

@adreed-msft My team and I ran into this issue on linux as well. It seems to me that the default shouldn't be true since this can be a breaking change for users that don't require 'read' permissions. In the next major version bump it feels like it would be appropriate for this change to occur. I guess it depends on how closely you are following semantic versioning.