Powershell: [Invoke-WebRequest][PSVersion 6.0.2] Unable to set "expect" (100-continue) request-header field in multipart/form-data

[Fizcko]

Hi,

I'm trying to send en POST request with a file attatchment to a REST web server.
I need to wait for a 100 (Continue) response before sending the request body.

When i specify the "expect" request-header field it seems to be ignored/discard while the other fields do not (accept,user-agent or custom field).

Steps to reproduce

$head = @{
    auth_token="123456";
    accept="*/*";
    expect="100-continue";
}

$multipartContent = [System.Net.Http.MultipartFormDataContent]::new()

$myFilePath = "myFile.zip"
$FileStream = [System.IO.FileStream]::new($myFilePath, [System.IO.FileMode]::Open)
$fileHeader = [System.Net.Http.Headers.ContentDispositionHeaderValue]::new("form-data")
$fileHeader.Name = "uploadFile"
$fileHeader.FileName = "$myFilePath"
$fileContent = [System.Net.Http.StreamContent]::new($FileStream)
$fileContent.Headers.ContentDisposition = $fileHeader
$fileContent.Headers.ContentType = [System.Net.Http.Headers.MediaTypeHeaderValue]::Parse("application/octet-stream")
$multipartContent.Add($fileContent)

Invoke-WebRequest -Uri "http://MyRestServer/" -Method "POST" -Headers "$head" -Body "$MultipartContent"

Expected behavior

--------- HTTP Headers ---------
{"connection":"Keep-Alive","content-type":"multipart/form-data; boundary=\"49442f36-68c8-4e57-a5d0-feb7f1ce292a\"","accept":"/","user-agent":"Mozilla/5.0 (Windows NT 6.3; Microsoft Windows 6.3.9600; en-US) PowerShell/6.0.2","expect":"100-continue","auth_token":"123456","content-length":"79667491","host":"MyRestServer"}

Actual behavior

--------- HTTP Headers ---------
{"connection":"Keep-Alive","content-type":"multipart/form-data; boundary=\"49442f36-68c8-4e57-a5d0-feb7f1ce292a\"","accept":"/","user-agent":"Mozilla/5.0 (Windows NT 6.3; Microsoft Windows 6.3.9600; en-US) PowerShell/6.0.2","auth_token":"123456","content-length":"79667491","host":"MyRestServer"}

Environment data

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.0.2
PSEdition                      Core
GitCommitId                    v6.0.2
OS                             Microsoft Windows 6.3.9600
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[markekraus]

It looks like we have it hard coded to remove the Expect headers. I'm not exactly sure why

https://github.com/PowerShell/PowerShell/blob/1d549497cfba144e978f3c66c4f817926afd46ef/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs#L1105-L1108

The Expect header supplied to -Headers is applied first, but when Line 1108 is called, that removes any expect headers from the header collection. I have no clue why we would do that. By default, HttpClient doesn't send any Expect headers. All this line seems to serve is to remove any Expect headers supplied by the user which seems odd.

[markekraus]

Digging through the history, this seems to have been translated over from when the WebRequest API was used. In the oldest commits I can dig back to on this (60b3b30) where the PowerShell team copied the source code over, there is this:

https://github.com/PowerShell/PowerShell/blob/60b3b304f2e1042bcf773d7e2ae3530a1f5d52f0/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/WebRequestPSCmdlet.cs#L842-L845

I'm guessing this was to address some kind of issue at some point that doesn't seem relevant with HttpClient. I think it is safe to remove lines 1105-1108 in the current file now.

@SteveL-MSFT If you have someone who can dig up some history on this I would be interested to know why this was done. If not, I'd like to axe this code.

[SteveL-MSFT]

@markekraus the people who worked on this originally are no longer on the PS team. Based on your research, it seems that this code is no longer relevant and we should lave the Expect headers as-is. Perhaps if you can verify the cmdlets work with the change against Twitter, we can address the comment in the code.

[markekraus]

I did some more research on this and found this: https://github.com/dotnet/corefx/issues/8673

None of the concerns matter any more since HttpClient no longer uses WinHTTP nor curl. Looking at the source code, the new sockets handler has Expect 100 configured properly and linux no longer sends this by default (tested on a few flavors to be sure, but the shared code is pretty clear that it's not sent unless set in the HttpRequestMessage).

Per all my tests, and research this should safe to remove.