Azure-storage-azcopy: Slice bounds out of range when writing to drive root

[agentmorris]

Which version of the AzCopy was used?

10.1.2

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy cp "https://blob.url" ".destination_file"

What problem was encountered?

I am running azcopy from g:, so "." refers to "g:". I have write permissions to this drive. When running it this way, I get:

Job 03660234-d484-ef41-560b-b18a7c3fe358 has started
Log file is located at: C:Usersdmorris/.azcopy/03660234-d484-ef41-560b-b18a7c3fe358.log

0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, panic: runtime error: slice bounds out of range

goroutine 233 [running]:
github.com/Azure/azure-storage-azcopy/common.CreateParentDirectoryIfNotExist(0xc042966000, 0x11, 0xc0421b6710, 0xc0425a9ad0)
/go/src/github.com/Azure/azure-storage-azcopy/common/writeThoughFile.go:30 +0x128
github.com/Azure/azure-storage-azcopy/common.CreateFileOfSizeWithWriteThroughOption(0xc042966000, 0x11, 0x1a33f3c800, 0x0, 0xc35200, 0x0, 0x0)
/go/src/github.com/Azure/azure-storage-azcopy/common/writeThoughFile_windows.go:34 +0x40
github.com/Azure/azure-storage-azcopy/ste.remoteToLocal(0xcadf80, 0xc0429ab720, 0xc94c60, 0xc0421b66c0, 0xc04201a280, 0xc35208)
/go/src/github.com/Azure/azure-storage-azcopy/ste/xfer-remoteToLocal.go:107 +0xca9
github.com/Azure/azure-storage-azcopy/ste.parameterizeDownload.func1(0xcadf80, 0xc0429ab720, 0xc94c60, 0xc0421b66c0, 0xc04201a280)
/go/src/github.com/Azure/azure-storage-azcopy/ste/xfer.go:60 +0x6a
github.com/Azure/azure-storage-azcopy/ste.(jobPartMgr).StartJobXfer(0xc0431e8000, 0xcadf80, 0xc0429ab720)
/go/src/github.com/Azure/azure-storage-azcopy/ste/mgr-JobPartMgr.go:519 +0x6c
github.com/Azure/azure-storage-azcopy/ste.(jobPartTransferMgr).StartJobXfer(0xc0429ab720)
/go/src/github.com/Azure/azure-storage-azcopy/ste/mgr-JobPartTransferMgr.go:142 +0x4e
github.com/Azure/azure-storage-azcopy/ste.(jobsAdmin).transferProcessor.func1(0xcadf80, 0xc0429ab720)
/go/src/github.com/Azure/azure-storage-azcopy/ste/JobsAdmin.go:261 +0x1ae
github.com/Azure/azure-storage-azcopy/ste.(jobsAdmin).transferProcessor(0xc0420fc1e0, 0x19)
/go/src/github.com/Azure/azure-storage-azcopy/ste/JobsAdmin.go:269 +0x8a
created by github.com/Azure/azure-storage-azcopy/ste.initJobsAdmin
/go/src/github.com/Azure/azure-storage-azcopy/ste/JobsAdmin.go:188 +0x71e

Any way you spin it, this is a suboptimal error message, so ideally we should clarify this, but the good news is...

Have you found a mitigation/solution?

If instead I write to "g:some_folderoutput_file", everything works great, including the automatic creation of some_folder. It appears that it wants there to be a parent directory.

Can we either fix this or update the error message?

Thanks!

[JohnRusk]

Thanks for logging this. Its definitely a bug, not just an bad error message.

Does it still happen if you write azcopy cp "https://blob.url" "g:\destination_file" instead of azcopy cp "https://blob.url" ".\destination_file"?

[adreed-msft]

This actually makes sense, but yes, a proper error message should be given. . is effectively short for the cwd on Windows or source on UNIX-based OSes. Surrounding your path with "" treats it as a literal path, not a relative path. Does the crash persist without the quotation marks?

[agentmorris]

@JohnRusk ... the crash goes away if I replace ".blah" with "g:blah".

@adreed-msft ... the crash persists if I remove the quotes.

No one specifically asked this, but FYI the crash also persists if I replace ".blah" with just "blah", it needs to be "g:blah".

FWIW I am not blocked on this, as soon as I figured out this was a root-path issue, I was fine; it's perfectly fine in my scenario to just pick a different directory. But it would be great to either fix or clean up the error message. Thanks!

[adreed-msft]

I actually identified an issue with using relative characters in AzCopy as well while working on supporting long file paths. I've been able to fix the usage of the . character to specify the CWD (on windows) on another branch, but I'm redoing the work in a less janky way at the moment. This should be fixed along with long file paths, so I'm looping in #376 as well. Thanks for your patience.

[Eric-ITaaS]

Thank you. Looking forward to this being fixed.

[calebrob6]

I wanted to add that I run into this issue with the linux 10.2.1 build of azcopy (downloaded here https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10).

The only scenario in which azcopy cp "https://blob.url" destination_file does _not_ crash with the above error message is when destination_file is an absolute path. For examples:

• azcopy cp "https://blob.url.zip" destination_file.zip crashes
• azcopy cp "https://blob.url.zip" ./destination_file.zip crashes
• azcopy cp "https://blob.url.zip" "destination_file.zip" crashes
• azcopy cp "https://blob.url.zip" /home/caleb/destination_file.zip does not crash

[JohnRusk]

Thanks @calebrob6. When you say "the only situation where its doesn't crash" you're referring specifically to the case where its targeting a single file - i.e. the URL is the URL of a single file. Right? You're not seeing any crashes in the case where the source is a blob virtual folder or blob container, correct?

[calebrob6]

Yes - "https://blob.url" is the path to a zip file. I'm guessing this is the case for @agentmorris as well. I have not investigated the case where the source is anything else.

[JohnRusk]

Thanks. We currently have some significant refactoring under way of the relevant logic, so that should help (for the upcoming release). Sorry its inconveniencing you at the moment. I cant think of any better workaround than just using the full path as you are doing.

[adreed-msft]

Oh, I guess there's another good test case for the refactor.

[mffigueroa]

I also experienced this issue when using the cp command with the destination file path being relative to the current directory like so:

azcopy.exe cp https://server_name/folder_name/file_name file_name

However, when I went up a directory and did:

azcopy.exe cp https://server_name/folder_name/file_name folder_name/file_name

The exception was here:
https://github.com/Azure/azure-storage-azcopy/blob/2ad31646af432a74ec279042a285645abf80f10f/common/writeThoughFile.go#L30

This is likely because there is no parent directory and the forward slash string index isn't found with:
strings.LastIndex(destinationPath, AZCOPY_PATH_SEPARATOR_STRING)

Here is the full call stack/output:

0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, panic: runtime error: slice bounds out of range

goroutine 417 [running]:
github.com/Azure/azure-storage-azcopy/common.CreateParentDirectoryIfNotExist(0xc0003720b0, 0xe, 0xc000b08750, 0xc000b87af8)
/home/vsts/work/1/s/common/writeThoughFile.go:30 +0x12f
github.com/Azure/azure-storage-azcopy/common.CreateFileOfSizeWithWriteThroughOption(0xc0003720b0, 0xe, 0x103b65582d, 0x0, 0xb8ca00, 0x0, 0x0)
/home/vsts/work/1/s/common/writeThoughFile_windows.go:34 +0x40
github.com/Azure/azure-storage-azcopy/ste.remoteToLocal(0xcb13a0, 0xc000aebf40, 0xc8dc80, 0xc000b08700, 0x6a00328, 0xc0003de000, 0xb8c9a8)
/home/vsts/work/1/s/ste/xfer-remoteToLocal.go:107 +0xc74
github.com/Azure/azure-storage-azcopy/ste.parameterizeDownload.func1(0xcb13a0, 0xc000aebf40, 0xc8dc80, 0xc000b08700, 0x6a00328, 0xc0003de000)
/home/vsts/work/1/s/ste/xfer.go:62 +0x71
github.com/Azure/azure-storage-azcopy/ste.(jobPartMgr).StartJobXfer(0xc000abc600, 0xcb13a0, 0xc000aebf40)
/home/vsts/work/1/s/ste/mgr-JobPartMgr.go:527 +0x78
github.com/Azure/azure-storage-azcopy/ste.(jobPartTransferMgr).StartJobXfer(0xc000aebf40)
/home/vsts/work/1/s/ste/mgr-JobPartTransferMgr.go:143 +0x4e
github.com/Azure/azure-storage-azcopy/ste.(jobsAdmin).transferProcessor.func1(0xcb13a0, 0xc000aebf40)
/home/vsts/work/1/s/ste/JobsAdmin.go:272 +0x170
github.com/Azure/azure-storage-azcopy/ste.(jobsAdmin).transferProcessor(0xc000730000, 0x1)
/home/vsts/work/1/s/ste/JobsAdmin.go:280 +0x92
created by github.com/Azure/azure-storage-azcopy/ste.initJobsAdmin
/home/vsts/work/1/s/ste/JobsAdmin.go:199 +0x6b3

[adreed-msft]

Hi there, this should be fixed in the upcoming release-- It seems as though we were incorrectly trying to create the root path for the file. Thanks for opening up an issue!

[JohnRusk]

Fixed, and just released, in 10.3