Azure-storage-azcopy: When azcopying blobs between 2 storage containers, blob metadata key values are capitalized.

[sharkymcdongles]

Which version of the AzCopy was used?

10.0.0.3

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

azcopy cp "https://CENSORED.blob.core.windows.net/CONTAINER1?sv=2017-11-09&ss=bfqt&srt=sco&sp=rwdlacup&se=2018-10-25T21:22:33Z&st=2018-10-22T13:22:33Z&spr=https&sig=CENSORED" "https://CENSORED.blob.core.windows.net/CONTAINER2?sv=2017-11-09&ss=bfqt&srt=sco&sp=rwdlacup&se=2018-10-25T21:22:33Z&st=2018-10-22T13:22:33Z&spr=https&sig=CENSORED" --recursive=true

What problem was encountered?

The blob metadata for the original container had 2 keys:

encryptedvalue
filesize

after the transfer the 2 keys were changed to:

Encryptedvalues
Filesize

How can we reproduce the problem in the simplest way?

Run the command to transfer the files same as above then compare the metadata fields. In the Azure portal the metadata fields look the same, but when you check in Azure storage explorer you can see they are different.

Have you found a mitigation/solution?

When you copy without the shared access signature and instead use the keys themselves it doesn't occur.

[zezha-msft]

Hi @sharkymcdongles, thanks for reaching out!

I reproduced this problem and have identified that the Blob SDK is causing this behavior.

In general though, we advise not to rely on the case of metadata keys, since they are implemented as HTTP headers, which are supposed to be case-insensitive. However, the Storage Service is case-preserving, for now.

I'll get back to you shortly. Thanks for bringing this issue to our attention!

[cy33hc]

I had the same issue. I had copied over 20 million blob from account to account only to realize all the metadata is corrupted. The Blob SDK for go should work like any other lang. We use the Blob java SDK and the case isn't converted.

[zezha-msft]

Hi @cy33hc, I apologize for the inconvenience. Unfortunately the HTTP client in Golang canonicalizes the headers when they are read off the wire, so we are having a really hard time fixing this issue.

I've enlisted the help of some service folks to see if there are any way to work around this, since HTTP2, which forces headers to be lowercase, will eventually create the same problem. I'll get back to you when I have more information.

[cy33hc]

Is there an eta of when this will be fixed? Also how to you copy from blob to blob with account key as the mitigation?

[JohnRusk]

@cy33hc I haven't quite understood what you mean about "copy with account key as mitigation". From your original comments when logging this issue it sounds like you have a way to preserve the headers in some situations, but I haven't quite understood what that situation is.

[erichexter]

@zezha-msft is there an update on this issue?

[zezha-msft]

Hi @erichexter, thanks for following up on this issue.

Unfortunately there's no easy solution for us right now. We'd have to make a change on the service end to pass the metadata differently.

[erichexter]

Copying the data is part of our disaster recovery plan, so this is actually a much bigger issue then I think it is getting attention for.

Since the .net storage client uses a dictionary to index the metadata it is by nature case sensitive. So this tool changing the casing of the keys, that essentially is breaking the supported apis. I will file a support ticket in azure to escalate this issue from the support side.

[zezha-msft]

Hi @erichexter, I assure you this issue is definitely on our radar. Solving this problem is not trivial though.

Unfortunately, the service (the source of truth, not the C# SDK) implemented metadata using HTTP headers. This means that by definition, the metadata keys are case-insensitive. However, the service has been case-preserving thus far, e.g. the same casing is preserved, and this resulted in users relying on the casing (some tools were not built correctly as well).

We are trying to figure out a way to solve this problem at its source. I'll update this thread as soon as there's more info.

[fpa-mls]

I still don't get why the service cannot simply be case preserving. But if it cannot do that implementing something simple and reliable like all lowercase would definitely be easier to cope with than this strange windows-like capitalisation. What more, there used to be a case preserving route in azcopy when using access key authentication but it has been removed leaving only the SAS route which suffers this bug. If it was possible with keys before why is it not possible with SAS? We currently need to use an older azcopy version to continue, so this is actually a regression.

[JohnRusk]

@fpa-mls What's the version number of the one where this worked for you?

BTW, it's not a service issue, unfortunately. In this service-to-service copying scenario, the data flows directly from source to destination but the service API requires that the headers flow through the client application (in this case, AzCopy). As you probably know, AzCopy v10 is written in Go. The designers of Go have taken an opinionated stance on this issue, in which they insist that its is correct for their SDK to change the case of the headers as they arrive. Technically, according to HTTP standards etc, they are correct, but it has undesirable side effects in cases like this. The reason it's so difficult for us to work around is that it's at a very low level in the Go HTTP stack, and the behavior is not configurable.

[fpa-mls]

So blob metadata is transferred as HTTP headers? We’re not talking about standard blob metadata like ContentType or ContentMD5 but the custom key value pairs which can be added to a blob

azcopy 7.3.0-netcore was handling this fine, but ONLY when using access key authentication which seem no longer possible in newer versions. SAS authentication had the same issue even back then.

[JohnRusk]

That is correct, this applies to custom metadata. I believe the only difference is that, for standard ones like ContentMD5, folks normally access those through an SDK that takes care of case insensitivity for them. i.e. No-one relies on case preservation at service level for those, because they access them through strongly-typed properties in their SDK.

I believe that AzCopy 8.1 also supports access key auth (at least the docs say it does).

[163satish]

Is there any plan on fixing this in upcoming release?

[JohnRusk]

We don't have a date for when it might be fixed, due to the difficulty of fixing it within the way the Go SDK works

[163satish]

Thank you for your response @JohnRusk . I am looking for a better tool for copying blob to blob.
I have encountered problem with the metadata where case is changing. I tried AzCopy V8 but problem with that is, it does not allow me to exclude folders/files. That is the reason I was hoping a fix to V10 as soon as possible. I understand that V10 is dependent on the Go SDK internally, but the metadata changing case is really a big problem for me.

[devax]

Is there an update on this? I just hit the same issue and it costed me hours to identify the source problem. I have not built a workaround, yet, just wanted to check what the state of this issue is.

[JohnRusk]

Sorry, no change as yet. (It's a hard problem, because its in the underlying framework, not in AzCopy itself).

[markl11]

Yes, we need a solution for this. It is quite a bad limitation.

[Johnny-Bee]

Hi there,

When we tried to copy Event Hub blob storage checkpoints from storage to another one.
We detect that azcopy does not keep the case of the custom HTTP header name which creates a bug when we try to use the copied version in C# using the Event Hub library version 3.0.0.0 on .NET CORE.

We had this error:

Microsoft.WindowsAzure.Storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. (Most recent call last)
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<AcquireLeaseCoreAsync>d__33.MoveNext() in Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<AcquireLeaseCoreAsync>d__33 line 1133
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in System.Runtime.ExceptionServices.ExceptionDispatchInfo line 17
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task) in System.Runtime.CompilerServices.TaskAwaiter line 39
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) in System.Runtime.CompilerServices.TaskAwaiter line 46
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult() in System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter line 11
at Microsoft.Azure.EventHubs.Processor.PartitionManager+<>c__DisplayClass12_2+<<RunLoopAsync>b__2>d.MoveNext() in Microsoft.Azure.EventHubs.Processor.PartitionManager+<>c__DisplayClass12_2+<<RunLoopAsync>b__2>d line 585

The cause is due to lost of the case of the HTTP header OWNINGHOST
Original:
Copied:

The HTTP header OWNINGHOST seems to be important on the server-side of the Event Hub and the case needs to be kept. https://github.com/Azure/azure-sdk-for-net/blob/809f48630e06b7672b4f3475f814cd46bfd97b33/sdk/eventhub/Microsoft.Azure.EventHubs.Processor/src/AzureStorageCheckpointLeaseManager.cs#L18

We lost a week to found that, so please keep the case during the copy to avoid any side effects.

Thanks,
Johnny

[JohnRusk]

We're aware of the issue, and looking for solutions Johnny. Unfortunately, the behavior is hard-coded into the runtime library of Go, the language that AzCopy v10 is written in, which is why its so difficult for us to find a solution. I don't have any other updates at this time, I'm sorry.

[Johnny-Bee]

Thanks for your response! Good luck guys!

[JohnRusk]

I've opened a fresh issue with the Go team here: https://github.com/golang/go/issues/37834

[cy33hc]

I had the same issue. I had copied over 20 million blob from account to account only to realize all the metadata is corrupted. The Blob SDK for go should work like any other lang. We use the Blob java SDK and the case isn't converted.

@JohnRusk @sharkymcdongles I was asking the question to the submitter of this issue. Under the section for mitigation, he/she mentions using account key as a mitigation. I don't know how that was achieved too.

[JohnRusk]

I don't know how using account key would help either. My only guess, about what that may have possibly meant, is that they went back to an older version of AzCopy (since versions 8 and earlier work differently. Unlike v10, they don't suffer from the header capitalization issue and they _do_ support account key authentication).

[JohnRusk]

Notes:

1. Looks like there might be some promising options recently raised in the GoLang thread (see link in comment above).
2. If those GoLang changes allow us to fix this issue, it's been suggested that we should retain AzCopy's current behavior as the default, and only preserve metadata case if a command line or environment variable flag is set. That approach makes sense in terms of a future-facing view because HTTP 2 won't allow case preservation, and so the internet is moving in the direction of not preserving case. But meantime, users who are aversely affected by this issue, can set the flag to preserve case and unblock their HTTP 1 data migrations.

[AdrianGin]

you can use az cli
az storage blob metadata update
as a workaround

This preserves case sensitivity

[philMarius]

Hi there, just ran into this issue myself and it took me all morning to find. What's most odd is that storage explorer was reporting the correct lowercase metadata but when I dived into the Python SDK I found that the first letter for all values were capitalised.

Has this been fixed yet?

[zezha-msft]

Unfortunately not, this is a limitation of the httpclient in Golang.

[philMarius]

So no fix will be issued for this at all? This is making using azcopy for a lot of our processes less viable

[zezha-msft]

@philMarius there's no short-term fix unfortunately. The long-term fix is on the service side, they will need to avoid putting metadata keys as HTTP header names, which are not supposed to be case sensitive.

[zezha-msft]

i understand the frustration, I'll follow up on this issue offline.

[bigbearzhu]

looks like https://github.com/golang/go/issues/37834 has just got the fix merged? Any good news for an azcopy fix and even Azure storage explorer? Many thanks!

[CODevOp]

Any update on this? It looks like we maybe made progress 3 months ago.

[CODevOp]

Does anyone know where to get a copy of azcopy 8.1.0 or azcopy 7.3.0-netcore. I cannot find a download or package that works.