Azure-storage-azcopy: azcopy thinks file is directory when sas token has slashes

[astrahan87]

Which version of the AzCopy was used?

AzCopy 10.3.4

Which platform are you using? (ex: Windows, Mac, Linux)

Linux
(Running in a powershell script on a Jenkins node)

What command did you run?

$sourceUri = az storage blob generate-sas ... --full-uri
$targetUri = az storage blob generate-sas ... --full-uri
azcopy copy $sourceUri $targetUri --overwrite true

What problem was encountered?

Got the error:failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

The files being copied are .vhd files and definitely not directories.
The primary problem occurred when changing from using the Az powershell module to the az cli to generate the SAS URIs. e.g.:

New-AzStorageBlobSASToken ... -FullUri
# Changed to:
az storage blob generate-sas ... --full-uri

The issue seems to be that the az cli does not properly escape the / character. e.g.:

• az cli output: https://(account).blob.core.windows.net/(container)/(filename).vhd?...&sig=.../.../...
• Az powershell module output: https://(account).blob.core.windows.net/(container)/(filename).vhd?...&sig=...%2F...%2F...

This is probably also a bug for the az cli, but I think azcopy should update its parsing to better handle the SAS tokens. The current parsing should not mistake a file for a directory because of URI parameters.

How can we reproduce the problem in the simplest way?

Use the az cli to generate two file Uris+SAS and see if they have an unescaped / in the SAS token.

Have you found a mitigation/solution?

Switching back to the AZ powershell module works.
Manually escaping the slashes prior to the azcopy call should also work.

[JohnRusk]

@adreed-msft Can you take a look, and add to backlog as appropriate?

[adreed-msft]

Sure thing, though I have my suspicions this is actually _fixed_ after the text processing errors PR. I'll check both sides of that.

[Plasma]

Myself and others have been getting this error also, see https://github.com/MicrosoftDocs/azure-docs/issues/41969#issuecomment-586784889

[astrahan87]

I've created an issue on the az-cli side to fix the encoding: https://github.com/Azure/azure-cli/issues/12291

[Guilhermeslucas]

Im having the same issue when generating a sas token with the ip information as well, trying to copy from a blob to fileshare. Anyone has fixed that?

[JohnRusk]

Depending on the result of @adreed-msft 's check, above, it may be fixed in the upcoming release 10.4. (No date announced)

[skmkzyk]

As far as I tested, SAS token with '/' will work fine when I hit the URL in the browser.
So I think it's the issue of the azcopy itself, not az-cli.

The SignedVersion from az-cli is sv=2018-03-28, and same from Powershell is sv=2019-02-02.
PowerShell generated vaule sv=2019-02-02 is same as Azure Portal generated.
Would you please investigate how azcopy handles URL parameters?

[JohnRusk]

Given that there's a workaround (to escape the slashes) we won't look at this right now. But we'll leave this issue open to remind us to look at it later.

[inju]

Not solved in AzcopyVersion 10.5.0

Trying to fetch a ZIP file from Azure storage on MacOS (tried zsh and pwsh). File URL copied from portal and do get failure: URL is treated as directory

Note, the SAStoken has "/" escaped with %2F, does not help. GET request shows in URL still "restype=directory"

[patridge]

Is there a workaround for this? I saw mention of escaping the slashes, but that doesn't seem to work.

I just confirmed the issue happens in 10.5.1, causing an error on this Microsoft Learn module. When the tokens are retrieved in the portal there, the slashes are already escaped.

[astrahan87]

@patridge To workaround it, I've been using the powershell module instead of az cli:

$sourceUri = New-AzStorageBlobSASToken ... -FullUri
$targetUri = New-AzStorageBlobSASToken ... -FullUri
azcopy cp $sourceUri $targetUri --overwrite true

[connorcarolan]

Version 10.6.0
Hit this same error message when copying files that contained a plus symbol in their path (not related to the SAS token). My workaround was to encode the file path, converting + to %2B:

folder+one/foldertwo/file.txt
folder%2Bone/foldertwo/file.txt

[arroyc]

AzCopy 10.8.0 ... it was working well for us ... Since last week we are seeing this error

+ echo 'Blob '\''dotnet-5.0.102.tar.gz'\'' does not exist in Prod storage container '\''dotnet'\''. Copying it...'
+ /tmp/azcopy-tool/azcopy copy 'https://oryxsdksdev.blob.core.windows.net/dotnet/dotnet-5.0.102.tar.gz?***' 'https://oryxsdksprod.blob.core.windows.net/dotnet/dotnet-5.0.102.tar.gz?***'

Blob 'dotnet-5.0.102.tar.gz' does not exist in Prod storage container 'dotnet'. Copying it...
INFO: Scanning...

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

We are trying to copy a blob from one storage account to another with SAS token. Is there any workaround?

[arroyc]

any update on this?

[jackfoxy]

AzCopy 10.8.0 not working for me either

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

there are not slashes in my SAS token.

Trying to copy files from one storage account's blobs to another storage account's file share.

[nakulkar-msft]

@jackfoxy This error usually means azcopy could not list the properties of a certain path on the source account - invalid path or incorrect/insufficient permissions. Can you verify from azcopy log file if it indeed was the case?

[jackfoxy]

@nakulkar-msft thanks for reaching out.
azcopy is being inconsistent about writing logs to user/.azcopy.
The most recent log timestamps appears to be from earlier tests that failed for other reasons.

I ran the test just now, which wrote the following to stout:

INFO: Scanning...

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

but did not write a log file with today's timestamp. (And none of my last tests from yesterday have a matching timestamp. I know the TS is in UTC.)

Other odd things about the logs: it records delete statements for the dest file, even though there was no file to delete, all the messages are for the dest paths, no messages at all about the source path.

[jackfoxy]

and the log version is AzcopyVersion 10.7.0, even though I am running 10.8.0

[jackfoxy]

The only messages about properties look like this:

2021/02/12 23:17:59 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2021/02/12 23:17:59 INFO: [P#0-T#6] https://bdaasprodsouthcentralus.file.core.windows.net/locsim-data/tachcorp/fits/cfrfb202081998/final?se=2021-03-14t23%3A17%3A58z&sig=-REDACTED-&sp=rwdl&sr=s&sv=2018-03-28 (folder properties) Queuing folder, to be deleted after it's children are deleted Dst: fits/cfrfb202081998/final

[qianz2]

Please take this issue into priority

[arroyc]

I'm still getting this issue .. any update on this?

[arroyc]

any update on this? I'm still seeing this issue .. this is blocking our weekly release. What is the resolution? Seems like this issue is there since a month and moving from one version to another version. We regenerated the token multiple times with keeping the start date in past, but no way we could make it work.

+ devStorage='https://oryxsdksdev.blob.core.windows.net/dotnet/defaultVersion.txt?***'
+ prodStorage='https://oryxsdksprod.blob.core.windows.net/dotnet/defaultVersion.txt?***'
+ /tmp/azcopy-tool/azcopy cp 'https://oryxsdksdev.blob.core.windows.net/dotnet/defaultVersion.txt?***' 'https://oryxsdksprod.blob.core.windows.net/dotnet/defaultVersion.txt?***'

Blob 'dotnet-5.0.102.tar.gz' already exists in Prod storage container 'dotnet'. Skipping copying it...

Blob 'defaultVersion.txt' does not exist in Prod storage container 'dotnet'. Copying it...
INFO: Scanning...

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

As you can see we are trying to copy a file and not a directory, but somehow it always detect this as directory

@zezha-msft @JohnRusk

[patridge]

Not sure if others are experiencing the same problem I encountered when they got this message, but my error may not have been related to the token containing the slash as much as I was trying to operate on several objects in a container, for which my SAS token was not configured to allow. I was able to resolve this issue by adding Object to the Allowed resource types when I generated a SAS (shared access signature).