Azure-storage-azcopy: Receive InvalidHeaderValue copying blob to azure gov

[jkattmann]

Which version of the AzCopy was used?

azcopy version 10.3.4

Note: The version is visible when running AzCopy without any argument

No

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy cp $src_url $dst_url

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

cp https://MYSTORAGEACCOUNT.blob.core.windows.net/SOMECONTAINER/SOMEFILE.vhd?se=2020-02-13t19%3A28%3A49z&sig=-REDACTED-&sp=rwd&sr=b&st=2020-02-12t17%3A28%3A49z&sv=2019-02-02 https://MYSTORAGEACCOUNT.blob.core.usgovcloudapi.net/SOMECONTAINER/SOMEFILE.vhd?se=2020-02-13t19%3A28%3A49z&sig=-REDACTED-&sp=racupwdl&srt=sco&ss=bfqt&st=2020-02-12t17%3A28%3A49z&sv=2019-02-02

What problem was encountered?

The PUT failed with The value for one of the HTTP headers is not in the correct format. specifying the Content-Length header as the malformed header.

Response Details:  InvalidHeaderValueThe value for one of the HTTP headers is not in the correct format. Content-Length0

How can we reproduce the problem in the simplest way?

Copy a blob from azure commercial to azure gov. I can reproduce the issue copying a blob from azure gov subscription to another azure gov subscription

Have you found a mitigation/solution?

No, I can see from fiddler the content length is 0 ... which seems to match the api docs ... https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob

User-Agent: AzCopy/10.3.4 Azure-Storage/0.7 (go1.13; Windows_NT) Content-Length: 0

Same error when attempting to use storage explorer to copy as well :(

[JohnRusk]

Can you clarify please, is this only when copying between public cloud and govt cloud, or is it between different accounts within the (same) govt cloud?

[jkattmann]

I can reproduce it using the same SPN copying between gov subscriptions under the same tenant.

Also worth noting I have tried various api versions by setting the AZCOPY_DEFAULT_SERVICE_API_VERSION variable but always get some form of the RESPONSE Status: 400 The value for one of the HTTP headers is not in the correct format.

[JohnRusk]

Could you send me a chunk of the log file please: it would be helpful if I could see the complete log section for one request. I.e. there'll be a line that contains REQUEST/RESPONSE, with a bunch of lines after it (they are request and response headers).

Log files don't contain any authentication secrets, but they do contain file names. You're welcome to redact or change the filenames in the log section, since I don't need the names.

My email address is in my profile.

[jkattmann]

Emailed, thanks for having a look!

[JohnRusk]

Was not an AzCopy bug. Root cause has been resolved.

[MatisseHack]

Are you able to share the resolution here? I'm running into the same issue trying to copy a VHD file from the public cloud into the government cloud. I am able to copy block blobs without issue, but can't get page blobs to copy over.

[JohnRusk]

@MatisseHack Public-to-government is possibly a different bug. We should check _which_ header field you are getting the error for. Can you check the AzCopy log, and see if its ContentLength? Look for something like this:

 <?xml version="1.0" encoding="utf-8"?> <Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format. </Message><HeaderName>Content-Length</HeaderName><HeaderValue>0</HeaderValue>

[MatisseHack]

When I run this command:

.\azcopy copy "https://source.blob.core.usgovcloudapi.net/resources/file.vhd?se=2020-03-19t22%3A06%3A20z&sig=-REDACTED-&sp=rl&sr=b&st=2020-03-18t22%3A06%3A20z&sv=2018-03-28" "https://destination.blob.core.usgovcloudapi.net/images"

I get this error in the logs:

2020/03/18 22:22:06 ==> REQUEST/RESPONSE (Try=1/1.9524ms, OpTime=1.9524ms) -- RESPONSE STATUS CODE ERROR
   PUT https://destination.blob.core.usgovcloudapi.net/images/file.vhd?comp=page&timeout=901
   Authorization: REDACTED
   Content-Length: [0]
   User-Agent: [AzCopy/10.3.4 Azure-Storage/0.7 (go1.13; Windows_NT)]
   X-Ms-Client-Request-Id: [0463c3cd-bc9f-4995-41be-61b13f729cf3]
   X-Ms-Copy-Source: [https://source.blob.core.usgovcloudapi.net/resources/file.vhd?se=2020-03-19t22%3a06%3a20z&sig=-REDACTED-&sp=rl&sr=b&st=2020-03-18t22%3a06%3a20z&sv=2018-03-28]
   X-Ms-Page-Write: [update]
   X-Ms-Range: [bytes=0-1048575]
   X-Ms-Source-Range: [bytes=0-1048575]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 400 The value for one of the HTTP headers is not in the correct format.
   Content-Length: [322]
   Content-Type: [application/xml]
   Date: [Wed, 18 Mar 2020 22:22:05 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [InvalidHeaderValue]
   X-Ms-Request-Id: [772d0848-601e-0068-4573-fd61c3000000]
   X-Ms-Version: [2018-11-09]
Response Details: <?xml version="1.0" encoding="utf-8"?> <Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format. </Message><HeaderName>Content-Length</HeaderName><HeaderValue>0</HeaderValue>

So does seem to be Content-Length. I also get an earlier error:

2020/03/18 22:22:06 ==> REQUEST/RESPONSE (Try=1/1.0395ms, OpTime=9.0414ms) -- RESPONSE STATUS CODE ERROR
   GET https://source.blob.core.usgovcloudapi.net/resources/file.vhd?comp=pagelist&se=2020-03-19t22%3A06%3A20z&sig=-REDACTED-&sp=rl&sr=b&st=2020-03-18t22%3A06%3A20z&sv=2018-03-28&timeout=901
   Authorization: REDACTED
   User-Agent: [AzCopy/10.3.4 Azure-Storage/0.7 (go1.13; Windows_NT)]
   X-Ms-Client-Request-Id: [7653cd34-957f-473f-5f54-1a748bb1e9fc]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 400 Authentication information is not given in the correct format. Check the value of Authorization header.
   Content-Length: [298]
   Content-Type: [application/xml]
   Date: [Wed, 18 Mar 2020 22:22:05 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Request-Id: [76da0e2a-601e-0017-5773-fdf20a000000]
Response Details: <?xml version="1.0" encoding="utf-8"?> <Error><Code>InvalidAuthenticationInfo</Code><Message>Authentication information is not given in the correct format. Check the value of Authorization header. </Message>

But I get this error even for transfers that succeed so it seems like a red herring.

Also, it doesn't seem to just be an issue with public-to-government. I have tried to move files between storage accounts in the same government subscription and gotten similar errors. Moving the same files between two storage accounts in the public cloud does work though.

[JohnRusk]

Thanks. Yes, this is the same issue, which is temporarily affecting some copy processing. Because it's not an AzCopy bug, it's not really my place to share the root cause. But I will forward the issue on to the relevant team. Can you please email me, so that I can reply to you by email once they have resolved it? My email address is in my profile page.

[MatisseHack]

Will do! Thank you for the help!

[srj91]

Hi Everyone,
This might sound stupid, but my operation with AzCopy is completed for 40 VHD's. Also I can see the VHD's in destination account.
I have checked by clicking on one VHD in storage account and copy status is simply nothing except a (-) sign.

I did some google search but didnt found any relevant solution or suggestions. Can u please clarify this thing

[sushilInfy]

@JohnRusk - Can you please assist here to provide the resolution for it ?

[JohnRusk]

@sushilInfy I moved to a different team a few months ago, so I'm not involved in AzCopy any more. @zezha-msft or @adreed-msft should be able to help.

[ChocPanda]

@sushilInfy, did you ever resolve this issue?