Azure-cli: "Az Storage Blob Download .. -Max-Connections 1" file handling needs improvement.

[v-tisheg]

Currently the "AZ Storage Blob Download ... --Max-connections 1" arguments will only attempt to pull the larger files in one peice. When the file it is trying to download is larger than the memory on the workstation running the task the process consumes the workstation's entire memory and swap file then fails.

Looking to get the Azure CLI Storage blob download command structure to handle "--Max-Connections 1" like it does when the max connections is set to 2 or more. That is to handle files larger than the memory of the workstation by breaking the file into multiple peices and make multiple requests to get the full file to download.

The Customer needs to download the blob to a named pipe so they must be able to use a single download connection to prevent seek operations on the pipe.

[williexu]

The CLI relies on the storage sdk to download blobs.

Looping in storage team guys @seguler, @zezha-msft to answer the question regarding support via sdk.

[zezha-msft]

Hi @v-tisheg and @williexu, I took a look at the code, and it seems to me that this is a design flaw in the SDK and should be fixed.

To be more specific, the SDK currently does not address the scenario that @v-tisheg has described, where a large file needs to be downloaded serially to a non-seekable stream.

@williexu Please create an issue in the SDK's repo, so that we can track this work. Thanks!

[williexu]

Done: https://github.com/Azure/azure-storage-python/issues/464

[v-tisheg]

Has this been published?
Does the customer need to download the cli modules?
The customer tried again this morning and the issue persists

[zezha-msft]

No @v-tisheg, this work will be scheduled soon.

[v-tisheg]

Hello all,
I was out last week. has there been any update to this issue yet?

[williexu]

@zezha-msft

[zezha-msft]

Hi @v-tisheg, the code is in review. I'll update this thread once it's ready to be released.

[v-tisheg]

Thank you so much.

Timothy Shegrud

Support Engineer
Cloud & Infrastructure Solutions
Customer Services & Support | Microsoft Corporation
[phone]; * [email protected][e-mail] | Hours: Mon - Fri, 7:00am-4:00pm PST
Need help outside of my working hours for a technical issue?
*: [email protected][e-mail]

Meeting and exceeding your expectations are my top priorities. We are interested in any feedback you might have about the service you received on this incident. Please let my Manager, FNU Manjunatha know what you think of the level of service provided, by contacting him via email at [email protected][e-mail].

[cid:[email protected]] [cid:[email protected]]

From: Ze Qian Zhang [e-mail]
Sent: Monday, July 9, 2018 11:54 AM
To: Azure/azure-cli [e-mail]
Cc: Tim Shegrud (MINDTREE LIMITED) [e-mail]; Mention [e-mail]
Subject: Re: [Azure/azure-cli] "Az Storage Blob Download .. -Max-Connections 1" file handling needs improvement. (#6608)

Hi @v-tisheghttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fv-tisheg&data=02%7C01%7Cv-tisheg%40microsoft.com%7C01607baecc30436f0caa08d5e5cd661b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636667592697476575&sdata=fmKjMUXfW9K8MmKFPEm0jWMGijUZXfTxkNKgffwyELo%3D&reserved=0, the code is in review. I'll update this thread once it's ready to be released.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-cli%2Fissues%2F6608%23issuecomment-403583786&data=02%7C01%7Cv-tisheg%40microsoft.com%7C01607baecc30436f0caa08d5e5cd661b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636667592697476575&sdata=XQcepOTpacV%2FVIG661%2BcYhsR6gTn%2BfkmVLIOMqdythQ%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAjuECYW7_IewT7IXkcM2rxz_EmJQdYwnks5uE6bkgaJpZM4UuBEd&data=02%7C01%7Cv-tisheg%40microsoft.com%7C01607baecc30436f0caa08d5e5cd661b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636667592697476575&sdata=tmhlhB41dpqhynucZHdiXhLhNMIWSQcXPdY8TE6f8Wk%3D&reserved=0.

[zezha-msft]

Hi @v-tisheg, the issue has been fixed in version 1.3.1 of the blob package. It is up to the CLI to pick up the new version in their next release. Thanks.