Powershell: Breaking change: Invoke-WebRequest -ContentType with empty string now fails

[jazzdelightsme]

Steps to reproduce

Unfortunately I don't have a simple repro at hand, but the change is that passing an empty string for the -ContentType parameter of Invoke-WebRequest used to work, through 6.2.3, but now it fails.

I'm using a popular internal module that makes calls to the Azure DevOps web service. It has an internal function that accepts a [string] $ContentType parameter, which then gets passed on to Invoke-WebRequest. When there is no body to send, and thus no content type, the calling code passes $null for $ContentType, but since the parameter type is [string], it gets converted to an empty string. The same thing happens in PS5.1. In PS5.1, passing in the empty string works (it treats it as no content type), and the same in PsCore 6. But on PSv7:

Actual behavior

The cmdlet cannot run because the -ContentType parameter is not a valid Content-Type header. Specify a valid Content-Type for -ContentType, then retry. To suppress header validation, supply the -SkipHeaderValidation parameter.

Expected behavior

(no error)

I don't know if this is worth breaking the glass on the RC, but at a minimum, because this is a heavily-used, workhorse cmdlet, I think it should be documented.

Environment data

Name                           Value
----                           -----
PSVersion                      7.0.0-rc.1
PSEdition                      Core
GitCommitId                    7.0.0-rc.1
OS                             Microsoft Windows 10.0.19041
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[spongemike2]

Here's a repro that works in PS5 and PS6, and throws a tantrum in PS7:

Invoke-WebRequest -Uri "http://microsoft.com" -ContentType ""

[vexx32]

/cc @markekraus do you know if there was a change made to this deliberately, or was there possibly a change in the .NET Core libraries we're using?

[iSazonov]

Strict check of Headers and ContentTypes is .Net Core feature.
You can skip this with SkipHeaderValidation switch.
If a behavior is not correct please report to dotnet/runtime repo.

[jazzdelightsme]

Adding -SkipHeaderValidation would break people running the script on lower versions (5.1, 6.x). Spongemike2 came up with a different workaround, which does not require checking a version number:

if (-not $ContentType)
{
    Remove-Variable   -Name ContentType
    $ContentType =   $null
}

I am of course not opposed to anybody reporting anything to the dotnet/runtime people. But without having done any research into it, it doesn't seem like it would be necessarily wrong for them to treat null and String.Empty as different. And even if they did decide to change something, I assume it would be quite some time before that change was available to PS.

However, this is still a breaking change for PS: it used to work, now it does not.

Another question to consider: should we want this breaking change in PS? Maybe the additional flexibility allowed by treating null and String.Empty differently are something that we would want to be able to pass on to PS users (albeit at the cost of being a breaking change). I don't do web-ish stuff for my day job, so I really don't have any sort of gut feel for it--I leave it to wiser and more experienced minds to consider. But if we keep the breaking change, it would be nice to have it documented. (I assume there will be a list of breaking changes when PSv7 is released.)

[spongemike2]

Since this ends up being an HTTP header, an empty string isn't really a valid "key", so passing in an empty string _should_ mean that there's no content type. As you mention, in my case, I've worked around it. Others who encounter it can as well. However, doing so may be painful, and in the cases where the code may not be modifiable, this could be really problematic. I think it would be worth working around in the PS7 code base in order to maintain compatibility. But that's just my opinion. :-)

[iSazonov]

so passing in an empty string should mean that there's no content type.

This assumes ignoring ContentType parameter.

For script language it’s not characteristic sometimes to distinguish between null and empty string.

What about other web cmdlets? Are they susceptible too?

[iSazonov]

@SteveL-MSFT I believe we could avoid the breaking change before 7.0 GA and ignore ContentType parameter for null and empty string argument.
(It works in 5.1 and 6.0.)

[SteveL-MSFT]

From what I found https://github.com/whatwg/fetch/issues/332 it seems that empty HTTP headers is legal although not defined. Created https://github.com/dotnet/runtime/issues/1095

[iSazonov]

Until we get Core fix we could make local fix.

[SteveL-MSFT]

@iSazonov a local fix with a comment to the dotnet issue seems fine

[spongemike2]

The bug is in PS Core, not in dotnet. The bug was introduced with commit 2285ece613. In PS6, if there's no request content (i.e., no body), the headers aren't added. In commit 2285ece613, that logic was changed. Now, PS Core only creates an empty request content if there's no body, which means it parses the content type.

Look at the change to line 1244 of file "src\Microsoft.PowerShell.Commands.Utility\commands\utility\WebCmdlet\Common\WebRequestPSCmdlet.Common.cs" from commit 2285ece613.

[iSazonov]

@spongemike2 Thanks for investigations! Do you want to pull a fix?

[spongemike2]

I would love to, and I have one tested and ready-to-go, committed into a local topic branch. However, it appears I don't have permission to push my topic branch. How do I change that?

remote: Permission to PowerShell/PowerShell.git denied to spongemike2.
fatal: unable to access 'https://github.com/PowerShell/PowerShell.git/': The requested URL returned error: 403

[iSazonov]

@spongemike2 You should fork the repo, create new working branch in your fork, add commits, pull new PR from the branch.
https://github.com/PowerShell/PowerShell/blob/master/.github/CONTRIBUTING.md#contributing-to-code

GitHub

PowerShell/PowerShell

PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

[spongemike2]

Thanks, iSazonov. That's the documentation I needed (and was looking for). I was used to working with ADO pull requests, which work differently. I'm on it now... stay tuned...

[spongemike2]

Hmm... okay, I forked the repo, made the change there in a topic branch, pushed it, made a PR to master in the forked repo, and merged it. The docs aren't clear about what to do now. How do I get this from the "master" branch in the forked repo to the "master" branch in the official repo?

[iSazonov]

@spongemike2 You shouldn't merge in your forked master. After you publish your working branch and add commit you could open https://github.com/PowerShell/PowerShell - GitHub will show new button to create new PR.
( Also you can find steps in GitHub help)

GitHub

PowerShell/PowerShell

PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

[msftbot[bot]]

:tada:This issue was addressed in #11421, which has now been successfully released as v7.0.0-rc.2.:tada:

Handy links:

• Release Notes