Azurestorageexplorer: StorageError-"Server encountered an internal error. Please try again after some" while uploading blobs to local storage on Azure Storage Emulator

[mvmnrao]

Storage Explorer Version: 1.5.0
Platform/OS Version: Windows 10

Steps to Reproduce:

1. Downloaded blobs from cloud to local disc.
2. Uploading to blob container on Storage Emulator using Microsoft Azure Storage Explorer.

Error is also reproducible while uploading using AzCopy.

Exception Details:

{
  "name": "StorageError",
  "message": "Server encountered an internal error. Please try again after some time.\nRequestId:15694ebf-d08e-4daa-955f-029800ca11d0\nTime:2018-10-29T11:06:57.1328600Z",
  "stack": "StorageError: Server encountered an internal error. Please try again after some time.\nRequestId:15694ebf-d08e-4daa-955f-029800ca11d0\nTime:2018-10-29T11:06:57.1328600Z\n    at Function.StorageServiceClient._normalizeError (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\azure-storage\\lib\\common\\services\\storageserviceclient.js:1198:23)\n    at Object.StorageServiceClient._processResponse (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\azure-storage\\lib\\common\\services\\storageserviceclient.js:744:50)\n    at Request.processResponseCallback [as _callback] (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\azure-storage\\lib\\common\\services\\storageserviceclient.js:317:37)\n    at Request.self.callback (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\request\\request.js:185:22)\n    at emitTwo (events.js:126:13)\n    at Request.emit (events.js:214:7)\n    at Request.<anonymous> (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\request\\request.js:1157:10)\n    at emitOne (events.js:116:13)\n    at Request.emit (events.js:211:7)\n    at IncomingMessage.<anonymous> (C:\\Program Files (x86)\\Microsoft Azure Storage Explorer\\resources\\app\\node_modules\\request\\request.js:1079:12)",
  "code": "InternalError",
  "statusCode": 500,
  "requestId": "15694ebf-d08e-4daa-955f-029800ca11d0"
}

[jinglouMSFT]

Did you have your emulator running? Storage Explore doesn't automatically start the emulator.

[mvmnrao]

Hi,

Thank you for looking into it.

Yes, I am running emulator before trying the copy.

One more clue is, upload to local storage is failing only for few of the
blobs.

On Mon, 29 Oct 2018 at 23:53, jinglouMSFT [e-mail] wrote:

Did you have your emulator running? Storage Explore doesn't automatically
start the emulator.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/AzureStorageExplorer/issues/817#issuecomment-434022947,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AT6npDLLJRRj2M_Rb0iIEfQ6pgQihf4pks5up0eZgaJpZM4X-80Q
.

[jinglouMSFT]

What kind of blobs are those that are failing? Append blobs? Page blobs? Block blobs?

[MRayermannMSFT]

It sounds like this is an issue with the emulator. Can we ping someone on the emulator team to help investigate this?

[mvmnrao]

They all are block blobs.

On Tue, 30 Oct 2018 at 23:43, jinglouMSFT [e-mail] wrote:

What kind of blobs are those that are failing? Append blobs? Page blobs?
Block blobs?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/AzureStorageExplorer/issues/817#issuecomment-434408854,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AT6npADcjfUy4JY_kjfNM0FKHzfpv3Fwks5uqJbBgaJpZM4X-80Q
.

[mvmnrao]

Yes, we also suspect that, because we are getting this exception while
copying using AzCopy command also.

Please point me to right community.

On Wed, 31 Oct 2018 at 04:38, Matthew Rayermann [e-mail]
wrote:

It sounds like this is an issue with the emulator. Can we ping someone on
the emulator team to help investigate this?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/AzureStorageExplorer/issues/817#issuecomment-434502908,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AT6npFfLF9mv5QGcppPSmNafGPo0yNIMks5uqNvtgaJpZM4X-80Q
.

[jinglouMSFT]

I've already contacted the emulator team. Someone there will comment on the issue here.

[seguler]

It turns out the emulator is unable to keep up with the amount of requests AzCopy is sending all at once. It is a known issue. We'll have to tweak it so that AzCopy is sending requests to the emulator at a slower rate.

[mvmnrao]

It turns out the emulator is unable to keep up with the amount of requests AzCopy is sending all at once. It is a known issue. We'll have to tweak it so that AzCopy is sending requests to the emulator at a slower rate.

Could you let me know if there is any ETA for this?
Also one more clue is that the entire thing works fine when the copy command is executed from Ukraine but not working from Hyderabad/INDIA.

[zezha-msft]

@mvmnrao could you please clarify what is not working from Hyderabad? Are you uploading files to the public cloud?

[mvmnrao]

Using AzCopy, I am downloading block blobs from cloud storage to local
storage created in Azure Storage Emulator and this is not working.

On Sat, 3 Nov 2018 at 01:03, Ze Qian Zhang [e-mail] wrote:

@mvmnrao https://github.com/mvmnrao could you please clarify what is
not working from Hyderabad? Are you uploading files to the public cloud?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/AzureStorageExplorer/issues/817#issuecomment-435469215,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AT6npPuPDf6bceMZRt2gXZH8rbSvBP3Gks5urJ3wgaJpZM4X-80Q
.

[zezha-msft]

Hi @mvmnrao, thanks for getting back to us. Could you please clarify what "local
storage created in Azure Storage Emulator" means?

@MRayermannMSFT @jinglouMSFT Any insights?

[mvmnrao]

I created local storage for development and debugging purpose. Please refer below link for the details, those are the steps I followed to setup local storage.

https://docs.microsoft.com/en-us/azure/storage/common/storage-use-emulator

[zezha-msft]

Hi @mvmnrao, thank you so much for getting back to me!

I see, so you are transferring blobs from the public cloud to the emulator directly. @MRayermannMSFT, what is the AzCopy command that Stg Exp uses in this scenario? Does it do a download from public cloud to a local cache first, and then upload to emulator from local?

[mvmnrao]

Hi,
Below is the command we are using:
_/source:<> /dest:devstoreaccount1 /sourcekey:<> /destkey:Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== /pattern:<> /snapshot /y /s /synccopy_
And yes, I believe AzCopy downloads blobs to local cache first and then it pushes to local storage and it is behavior of the AzCopy. I don't think there is other way AzCopy works?

[zezha-msft]

I see. So you are using AzCopy V8. @EmmaZhu could you please help with this issue?

[EmmaZhu]

Hi @mvmnrao,

Just looked through discussion in this thread, there's same issue when uploading with Storage Explorer right?

From Sercan's answer, this should be a known issue in the emulator and slower request sending would be able to mitigate this kind of issue.

From what I understand, you'd want a way to slower AzCopy's request, right?
You can do this by lower AzCopy's parallel level with option /NC, with command line:
AzCopy /source:<> /dest:<> .... /NC:8
By default, AzCopy's parallel level is 8 * core numbers.

[mvmnrao]

Sure @EmmaZhu , I will try this option and get back, thank you for inputs.

[mvmnrao]

@EmmaZhu , I have tried with different parallel levels starting from 4 to 512, but I am still getting same error.

[craxal]

Doesn't appear to be a Storage Explorer issue, so I will close.

@mvmnrao, @EmmaZhu, @zezha-msft, since this appears to be an emulator issue, please continue the thread at an appropriate location.

[mvmnrao]

@craxal would you mind sharing locations for AzCopy and Storage Emulator? I will post it there because this is still an issue for us and needs to be fixed.

[EmmaZhu]

@mvmnrao, seems the error is reported from Storage Emulator, AzCopy just passes it out. I can only provide suggestions on AzCopy, you may still need Storage Emulator to fix it.

For AzCopy. it's designed to be able to resume from the last checkpoint. You can run AzCopy in a loop until its return code is 0 which means AzCopy has completed all transfers successfully.
This is only a workaround from AzCopy side. For AzCopy is a client side tool, if server (Storage Emulator) cannot process requests correctly, AzCopy may not be able to help with that.

Hope the workaround can work for you.

[craxal]

Unfortunately, I don't know where to go to post issues on Storage Emulator. There is a different emulator you can try: Azurite.

[fperreaultnv]

This error started happening to me when I was using a different storage account connection string for the runtime and for the blob trigger itself. AzureWebJobsStorage configuration value vs [BlobTriggerAttribute("some/path", Connection = "%ConfigSection:ConnectionString%").

[craxal]

@fperreaultnv Are you referring to the .NET client library? We use the _JavaScript_ client library.

[fperreaultnv]

@craxal Indeed I am.

[craxal]

@fperreaultnv Are you suggesting then the problem is with the emulator? Since we don't yse the .NET client library, I'm not sure how that fits in here.

[philipp-bananaware]

I tried to put (upload) multiple files on a container with SAS token, but get the same Response 500:
PUT http://127.0.0.1:10000/devstoreaccount1/inbox/9jsV5uKbAEM.jpg?sv=2018-03-28&sr=c&sig=ax9Flsc5F9wtjwqjAt76P9akEE3s3Geiy%2Bzmlpry0WY%3D&st=2020-09-18T18%3A53%3A10Z&se=2020-09-18T19%3A53%3A10Z&sp=w 500 (Server encountered an internal error. Please try again after some time.)
But it says "201 created" at the end...

[craxal]

@philipp-bananaware What emulator are you using? Connecting to an emulator via SAS is not supported scenario, as it depends on whether the emulator supports authenticating requests using a SAS. If you continue to encounter issues, and you believe this to be a bug in Storage Explorer, please open a new issue.