Azure-storage-azcopy: Azure Storage Explorer : 403 This request is not authorized to perform this operation using this permission. whith 1.11 version

Created on 12 Dec 2019  ·  25Comments  ·  Source: Azure/azure-storage-azcopy

Which version of the AzCopy was used?

Azure Storage Explorer 1.11
AzCopy 10.3.2

Which platform are you using? (ex: Windows, Mac, Linux)

Windows 10 1909

What command did you run?

Job-Command copy C:\Users\xxxxxxx\Downloads\S3100-9.11.3.0-Config.pdf https://xxxxxxxxxxxx.blob.core.windows.net/ingestiondata/S3100-9.11.3.0-Config.pdf?se=2020-01-11t11%3A53%3A09z&si=ingestionsasforazcopy201912061529080283&sig=-REDACTED-&sr=c&sv=2015-04-05 --output-type=json --overwrite=prompt --cancel-from-stdin --follow-symlinks --recursive --from-to=LocalBlob --blob-type=BlockBlob --put-md5

Launched by Azure Storage Explorer

What problem was encountered?

When I upload a file using a SAS key generated from O365 importation page with Azure Storage Explorer 1.11, I encounter this error :
403 This request is not authorized to perform this operation using this permission.
If I use the same SAS key in Azure Storage Explorer 1.9, it's okay.

How can we reproduce the problem in the simplest way?

Have you found a mitigation/solution?

Using The 1.9 version.

AzCopy log file :
2019/12/12 11:58:28 AzcopyVersion 10.3.2
2019/12/12 11:58:28 OS-Environment windows
2019/12/12 11:58:28 OS-Architecture amd64
2019/12/12 11:58:28 Log times are in UTC. Local time is 12 Dec 2019 12:58:28
2019/12/12 11:58:28 Job-Command copy C:\Users\xxxxxx\Downloads\S3100-9.11.3.0-Config.pdf https://xxxxxxx.blob.core.windows.net/ingestiondata/S3100-9.11.3.0-Config.pdf?se=2020-01-11t11%3A53%3A09z&si=ingestionsasforazcopy201912061529080283&sig=-REDACTED-&sr=c&sv=2015-04-05 --output-type=json --overwrite=prompt --cancel-from-stdin --follow-symlinks --recursive --from-to=LocalBlob --blob-type=BlockBlob --put-md5
2019/12/12 11:58:28 Number of CPUs: 4
2019/12/12 11:58:28 Max file buffer RAM 2.000 GB
2019/12/12 11:58:28 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2019/12/12 11:58:28 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2019/12/12 11:58:28 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2019/12/12 11:58:28 Max open files when downloading: 2147483311 (auto-computed)
2019/12/12 11:58:28 JobID=18681fc3-509b-a040-59ff-04875c176011, credential type: Anonymous
2019/12/12 11:58:28 PERF: primary performance constraint is Unknown. States: R: 0, D: 0, W: 0, F: 0, B: 0, E: 0, T: 0, GRs: 0
2019/12/12 11:58:28 scheduling JobID=18681fc3-509b-a040-59ff-04875c176011, Part#=0, Transfer#=0, priority=0
2019/12/12 11:58:28 INFO: [P#0-T#0] has worker 6 which is processing TRANSFER
2019/12/12 11:58:28 INFO: [P#0-T#0] Starting transfer: Source "%5C%5C?\C:\Users\xxxxx\Downloads\S3100-9.11.3.0-Config.pdf" Destination "https://xxxxxxxx.blob.core.windows.net/ingestiondata/S3100-9.11.3.0-Config.pdf?se=2020-01-11t11%3A53%3A09z&si=ingestionsasforazcopy201912061529080283&sig=-REDACTED-&sr=c&sv=2015-04-05". Specified chunk size 8388608
2019/12/12 11:58:28 ==> REQUEST/RESPONSE (Try=1/15.9593ms, OpTime=98.7374ms) -- RESPONSE STATUS CODE ERROR
HEAD https://xxxxxxxxx.blob.core.windows.net/ingestiondata/S3100-9.11.3.0-Config.pdf?se=2020-01-11t11%3A53%3A09z&si=ingestionsasforazcopy201912061529080283&sig=-REDACTED-&sr=c&sv=2015-04-05&timeout=901
User-Agent: [AzCopy/10.3.2 Azure-Storage/0.7 (go1.13; Windows_NT)]
X-Ms-Client-Request-Id: [86bf19be-d76b-4df2-5ba0-17200bbb32af]
X-Ms-Version: [2018-03-28]


RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
Date: [Thu, 12 Dec 2019 11:58:28 GMT]
Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
Vary: [Origin]
X-Ms-Error-Code: [AuthorizationPermissionMismatch]
X-Ms-Request-Id: [7ee9bdf0-201e-003c-6ae3-b0b91b000000]
X-Ms-Version: [2018-03-28]
Response Details:

2019/12/12 11:58:28 ERR: [P#0-T#0] UPLOADFAILED: %5C%5C?\C:\Users\xxxxxxx\Downloads\S3100-9.11.3.0-Config.pdf : 000 : Could not check file existence. -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)
HEAD https://xxxxxxxx.blob.core.windows.net/ingestiondata/S3100-9.11.3.0-Config.pdf?se=2020-01-11t11%3A53%3A09z&si=ingestionsasforazcopy201912061529080283&sig=-REDACTED-&sr=c&sv=2015-04-05&timeout=901
User-Agent: [AzCopy/10.3.2 Azure-Storage/0.7 (go1.13; Windows_NT)]
X-Ms-Client-Request-Id: [86bf19be-d76b-4df2-5ba0-17200bbb32af]
X-Ms-Version: [2018-03-28]


RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
Date: [Thu, 12 Dec 2019 11:58:28 GMT]
Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
Vary: [Origin]
X-Ms-Error-Code: [AuthorizationPermissionMismatch]
X-Ms-Request-Id: [7ee9bdf0-201e-003c-6ae3-b0b91b000000]
X-Ms-Version: [2018-03-28]

Most helpful comment

All 25 comments

To narrow down the cause, can you tell us please, when you used Storage Explorer 1.9, did you have AzCopy enabled or disabled? (In that version, I believe it only used AzCopy if you enabled the AzCopy integration).

Hello,
I used the 1.9 in july and august. I tried it few days because the 1.11 showed an error during upload.
I doesn't activate AzCopy in the 1.9, I use the default parameters.

@adreed-msft any chance you can reproduce this?

All my transfers are concerned by this error in the 1.11.

This is an error to check the file existence when the Overwrite setting is false or prompt.

It looks like your SAS token uses a signed identifier pattern-- My guess is, judging from a prior user's issue in a similar scenario, that your SAS token is Write-only. To fix this, overwrite should be true, and you should disable destination length checking. (--overwrite=true --check-length=false)

Please tell me if this helps your scenario @EricF2

@adreed-msft We have also noticed this issue with Azure Storage explorer 1.12 where write and list permissions for a SAS token are not enough to upload a file to blob.

Can you let me know if this is planned to be resolved in an upcoming release of storage explorer?

This issue is noticed in my case when the blob name is different from the file name that is being uploaded - Using Java SDK for Blob storage.

@datsabk if your issue relates to the Java SDK, I suggest you search for the right place to log it. (This isn't that place ;-) This is only for AzCopy v10)

@JohnRusk @adreed-msft This issue is still occuring even in v.1.13 where it was noted that https://github.com/microsoft/AzureStorageExplorer/issues/2336 had been resolved and a prompt was created. Can you investigate this further please.

@dttong13 In the current version of AzCopy, the solution is as per this comment above: https://github.com/Azure/azure-storage-azcopy/issues/790#issuecomment-569799061

Unfortunately, in Storage Explorer, I can't see any setting turn off the length check.

In the next version of AzCopy, it will automatically handle the situation, without the need to change the length check setting.

@JohnRusk Thanks for getting back so quickly.

Is there any update on when storage explorer will be able to leverage the next version of AzCopy? What would the version number be, so I can keep an eye out for it?

Current workaround for others: Use Storage Explorer v1.9

The AzCopy version number that you want will probably 10.5. (Although we might sneak it out in 10.4.something. If you subscribe to updates to the AzCopy releases pages, you'll be notified when all releases ship, and can check the release notes that are published there).

Sometime after that, Storage Explorer will release a version that includes the AzCopy update.

@JohnRusk Do you have an estimated ETA for such a release? This is just so we can keep our customers informed on when they can expect to update their storage explorer to v1.14 i imagine?

No, I don't sorry. We don't generally publish ETAs on the AzCopy team. IIRC the Storage Explorer team also generally don't publish ETAs.

This issue is now resolved in release v1.14. Please feel free to close this issue record

Hi duttong13
I've installed version 1.14 and still have this issue.

   Dst: https://link
2020/06/27 14:32:52 JobID=ID, Part#=0, TransfersDone=1 of 2
2020/06/27 14:32:52 ==> REQUEST/RESPONSE (Try=1/37.0021ms, OpTime=185.6622ms) -- RESPONSE STATUS CODE ERROR
   HEAD https://link
   User-Agent: [Microsoft Azure Storage Explorer, 1.14.0, win32,  AzCopy/10.4.2 Azure-Storage/0.7 (go1.13; Windows_NT)]
   X-Ms-Client-Request-Id: [ID]
   X-Ms-Version: [2019-02-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
   Date: [Sat, 27 Jun 2020 14:32:52 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   Vary: [Origin]
   X-Ms-Client-Request-Id: [ID]
   X-Ms-Error-Code: [AuthorizationPermissionMismatch]
   X-Ms-Request-Id: [ID]
   X-Ms-Version: [2019-02-02]
Response Details: 

2020/06/27 14:32:52 ERR: [P#0-T#0] UPLOADFAILED: some.pst : 000 : Could not check destination file existence. -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)
   HEAD https://link
   User-Agent: [Microsoft Azure Storage Explorer, 1.14.0, win32,  AzCopy/10.4.2 Azure-Storage/0.7 (go1.13; Windows_NT)]
   X-Ms-Client-Request-Id: [ID]
   X-Ms-Version: [2019-02-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
   Date: [Sat, 27 Jun 2020 14:32:52 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   Vary: [Origin]
   X-Ms-Client-Request-Id: [ID]
   X-Ms-Error-Code: [AuthorizationPermissionMismatch]
   X-Ms-Request-Id: [ID]
   X-Ms-Version: [2019-02-02]

I had a similar issue but it went away with 1.14 (before I was on 1.9).

Hi Triamus,
I've installed v1.14 on several servers and workstations: it doesn't work anywhere. I will install v1.14 on a server where it has never been installed

@TheAnahaym however my issue was that connecting to blob via sas token in azure storage explorer gui didn't work and now it does.

This is also happening in the 1.14.2 version of Azure storage explorer (AzCopy 10.5).

2020/08/03 20:59:16 AzcopyVersion  10.5.0
2020/08/03 20:59:16 OS-Environment  windows
2020/08/03 20:59:16 OS-Architecture  amd64
2020/08/03 20:59:16 Log times are in UTC. Local time is 3 Aug 2020 15:59:16
2020/08/03 20:59:16 Job-Command copy C:\Users\****\Desktop\TransformRulesMapping.csv https://*****.blob.core.windows.net/maindl/DROPBOX/DEVOPS/Link/TransformRulesMapping.csv --output-type=json --cancel-from-stdin --overwrite=prompt --from-to=LocalBlob --blob-type BlockBlob --follow-symlinks --put-md5 --follow-symlinks --recursive 
2020/08/03 20:59:16 Number of CPUs: 8
2020/08/03 20:59:16 Max file buffer RAM 4.000 GB
2020/08/03 20:59:16 Max concurrent network operations: 128 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2020/08/03 20:59:16 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2020/08/03 20:59:16 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2020/08/03 20:59:16 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2020/08/03 20:59:16 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2020/08/03 20:59:16 Max open files when downloading: 2147483119 (auto-computed)
2020/08/03 20:59:16 ISO 8601 START TIME: to copy files that changed after this job started, use the parameter --include-after=2020-08-03T20:59:11Z
2020/08/03 20:59:16 Authenticating to destination using Azure AD
2020/08/03 20:59:16 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2020/08/03 20:59:16 Final job part has been created
2020/08/03 20:59:16 PERF: primary performance constraint is Unknown. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  0, GRs: 20
2020/08/03 20:59:16 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:16 JobID=173a431d-a7dc-ef45-4ec9-14172f0835bf, Part#=0 2020-08-03 20:59:16.2975495 +0000 UTC token refreshed successfully
2020/08/03 20:59:16 JobID=173a431d-a7dc-ef45-4ec9-14172f0835bf, Part#=0 next token refresh's wait duration: 29m57.35122525s
2020/08/03 20:59:16 JobID=173a431d-a7dc-ef45-4ec9-14172f0835bf, credential type: OAuthToken
2020/08/03 20:59:16 scheduling JobID=173a431d-a7dc-ef45-4ec9-14172f0835bf, Part#=0, Transfer#=0, priority=0
2020/08/03 20:59:16 Final job part has been scheduled
2020/08/03 20:59:16 INFO: [P#0-T#0] has worker 40 which is processing TRANSFER
2020/08/03 20:59:16 INFO: [P#0-T#0] Starting transfer: Source "%5C%5C?\\C:\\Users\\****\\Desktop\\TransformRulesMapping.csv" Destination "https://****.blob.core.windows.net/maindl/DROPBOX/DEVOPS/Link/TransformRulesMapping.csv". Specified chunk size 8388608
2020/08/03 20:59:18 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:18 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:18 ==> REQUEST/RESPONSE (Try=1/61.8305ms, OpTime=1.999838s) -- RESPONSE SUCCESSFULLY RECEIVED
   HEAD https://****.blob.core.windows.net/maindl/DROPBOX/DEVOPS/Link/TransformRulesMapping.csv?timeout=901
   X-Ms-Request-Id: [0febce5f-301e-005e-22d8-69743e000000]

2020/08/03 20:59:20 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:20 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:22 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:22 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:24 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:24 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:26 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:26 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:28 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:28 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:30 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:30 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:32 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:32 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:34 PERF: primary performance constraint is Unknown. States: X:  1, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  1, GRs: 128
2020/08/03 20:59:34 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 
2020/08/03 20:59:36 ==> REQUEST/RESPONSE (Try=1/15.9598ms, OpTime=55.8528ms) -- RESPONSE STATUS CODE ERROR
   PUT https://****.blob.core.windows.net/maindl/DROPBOX/DEVOPS/Link/TransformRulesMapping.csv?timeout=901
   Authorization: REDACTED
   Content-Length: [68165]
   User-Agent: [Microsoft Azure Storage Explorer, 1.14.2, win32,  AzCopy/10.5.0 Azure-Storage/0.10 (go1.13; Windows_NT)]
   X-Ms-Blob-Cache-Control: []
   X-Ms-Blob-Content-Disposition: []
   X-Ms-Blob-Content-Encoding: []
   X-Ms-Blob-Content-Language: []
   X-Ms-Blob-Content-Md5: [OO/6haLaH+OywY6z/Qty+Q==]
   X-Ms-Blob-Content-Type: [application/vnd.ms-excel]
   X-Ms-Blob-Type: [BlockBlob]
   X-Ms-Client-Request-Id: [5c2621ae-38d5-439a-767f-697fa96d8132]
   X-Ms-Version: [2019-02-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform blob overwrites.
   Content-Length: [252]
   Content-Type: [application/xml]
   Date: [Mon, 03 Aug 2020 20:59:36 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [5c2621ae-38d5-439a-767f-697fa96d8132]
   X-Ms-Error-Code: [UnauthorizedBlobOverwrite]
   X-Ms-Request-Id: [0fec0a06-301e-005e-79d8-69743e000000]
   X-Ms-Version: [2019-02-02]
Response Details: <Code>UnauthorizedBlobOverwrite</Code><Message>This request is not authorized to perform blob overwrites. </Message>

2020/08/03 20:59:36 ERR: [P#0-T#0] UPLOADFAILED: %5C%5C?\C:\Users\*****\Desktop\TransformRulesMapping.csv : 403 : 403 This request is not authorized to perform blob overwrites.. When Uploading blob. X-Ms-Request-Id: 0fec0a06-301e-005e-79d8-69743e000000

   Dst: https://****.blob.core.windows.net/maindl/DROPBOX/DEVOPS/Link/TransformRulesMapping.csv
2020/08/03 20:59:36 JobID=173a431d-a7dc-ef45-4ec9-14172f0835bf canceled
2020/08/03 20:59:36 ==> REQUEST/RESPONSE (Try=1/92.7539ms, OpTime=92.7539ms) -- RESPONSE SUCCESSFULLY RECEIVED
   GET https://****.blob.core.windows.net/maindl/DROPBOX/DEVOPS/Link/TransformRulesMapping.csv?blocklisttype=all&comp=blocklist&timeout=31
   X-Ms-Request-Id: [0fec0a5b-301e-005e-42d8-69743e000000]

2020/08/03 20:59:36 JobID=173a431d-a7dc-ef45-4ec9-14172f0835bf, Part#=0, TransfersDone=1 of 1
2020/08/03 20:59:36 all parts of entire Job 173a431d-a7dc-ef45-4ec9-14172f0835bf successfully completed, cancelled or paused
2020/08/03 20:59:36 all parts of entire Job 173a431d-a7dc-ef45-4ec9-14172f0835bf successfully cancelled
2020/08/03 20:59:36 PERF: primary performance constraint is Unknown. States: X:  0, O:  0, M:  0, L:  0, R:  0, D:  0, W:  0, F:  0, B:  0, E:  0, T:  0, GRs: 128

Getting similar errors in Explorer 1.14.2. Unable to retrieve child resources.

Details:
["RestError:<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.\nRequestId:35d64ee9-b01e-0060-2d7f-6aecc1000000\nTime:2020-08-04T16:51:45.0261360Z</Message></Error>

i think there is problem with API endpoints. I am receiving the same error when try to programatically access the blob as well. The weird thing is that I can access everything with an older version of Storage Explorer, 1.12.0

This issue may be caused by not having “Storage Blob Delegator” at the account level.

Check Storage account firewall . Temporarily disable it and try .

Was this page helpful?
0 / 5 - 0 ratings