Powershell: Invoke-WebRequest / Invoke-RestMethod fails to follow HTTP redirects from HTTPS

[aburlak]

Steps to reproduce

1. Install Windows Server Nano (10.0.14393)
2. Connect to the server using PS Remoting, then use PowerShell Core (the built-in version) to download a file, e.g. try to download .NET Core 1.1 ZIP package:

Invoke-Webrequest -Uri 'https://go.microsoft.com/fwlink/?LinkID=835028' -OutFile C:netcore11.zip -MaximumRedirection 10

Expected behavior

File is downloaded, HTTP 302 Redirect returned by go.microsoft.com is handled properly.

Actual behavior

The following error occurs:
invoke-webrequest : Response status code does not indicate success: 302 (Moved Temporarily).
And there is no workaround, because the thrown exception is a HttpRequestException thrown by EnsureSuccessStatusCode(), which loses all header information (so, unable even to access the Location header).

Issue is still present in the master branch:
https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/CoreCLR/WebRequestPSCmdlet.CoreClr.cs#L352
EnsureSuccessStatusCode() should not be used, as it doesn't support redirects, and it also obfuscates the exception, making it impossible to access the returned headers/body even if the exception is caught.

Environment data

PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
BuildVersion                   10.0.14393.1000
SerializationVersion           1.1.0.1
PSVersion                      5.1.14393.1000
PSEdition                      Core
WSManStackVersion              3.0
CLRVersion

[aburlak]

After some debugging, it looks like the root cause is that .NET Core's HttpClientHandler silently refuses to follow HTTPS-to-HTTP redirects.

However, there should be some way to at least look at the underlying response headers - currently, only a generic string description of the error is provided in the caught Exception (in desktop PowerShell - it is possible to retrieve the headers).

[SteveL-MSFT]

Perhaps we should add a -AllowRedirectToHTTPfromHTTPS type switch or just output a warning message

[SteveL-MSFT]

This repros with alpha.17 on Ubuntu16 so not Nano specific

[SteveL-MSFT]

@PowerShell/powershell-committee suggests -allowInsecureRedirect

[SteveL-MSFT]

@markekraus interested in taking this one :)

[markekraus]

@SteveL-MSFT I'll add it to my list. Is this a priority for 6.1.0?

[SteveL-MSFT]

@markekraus no, just when you can get around to it. Priority can change if more community members upvote, though :)

[fcabralpacheco]

Possible workaround.
Works in PowerShell Core 6.0.1 (Linux) and PowerShell 5.

function Get-RedirectedUrl() {
  param(
    [Parameter(Mandatory = $true, Position = 0)]
    [uri] $url,
    [Parameter(Position = 1)]
    [Microsoft.PowerShell.Commands.WebRequestSession] $session = $null
  )

  $request_url = $url
  $retry = $false

  do {
    try {
      $response = Invoke-WebRequest -Method Head -WebSession $session -Uri $request_url

      if($response.BaseResponse.ResponseUri -ne $null)
      {
        # PowerShell 5
        $result = $response.BaseResponse.ResponseUri.AbsoluteUri
      } elseif ($response.BaseResponse.RequestMessage.RequestUri -ne $null) {
        # PowerShell Core
        $result = $response.BaseResponse.RequestMessage.RequestUri.AbsoluteUri
      }

      $retry = $false
    } catch {
      if(($_.Exception.GetType() -match "HttpResponseException") -and
        ($_.Exception -match "302"))
      {
        $request_url = $_.Exception.Response.Headers.Location.AbsoluteUri
        $retry = $true
      } else {
        throw $_
      }
    }  
  } while($retry)

  return $result
}

$url = Get-RedirectedUrl "https://go.microsoft.com/fwlink/?LinkID=835028"

Invoke-Webrequest -Uri $url -OutFile C:\netcore11.zip 

Note: Exception type checking works better in this way. Prevent errors in PowerShell 5 (non-existent type).

[thezim]

This is more than HTTP to HTTPS. This repros as well.

Get-Command -Name New-Item | Select-Object HelpUri                                                                  

HelpUri                                       
-------                                       
https://go.microsoft.com/fwlink/?LinkID=113353


Invoke-WebRequest -Uri (Get-Command New-Item).HelpUri 

[markekraus]

@thezim It's the same problem..

Invoke-WebRequest -Uri (Get-Command New-Item).HelpUri 
$error[0].Exception.Response.Headers.Location.AbsoluteUri

result: http://technet.microsoft.com/library/hh849795.aspx

the problem is when an HTTPS uri redirects to an HTTP (no-S) uri.

[xied75]

@fcabralpacheco The code snippet looks great but it doesn't seem to work in Windows Nano Container.

[anderssonjohan]

I stumbled across this issue when I discovered my script using System.Net.WebClient worked on pwsh but not on Windows. On Windows, or in Windows Powershell, you'll get a 401 Unauthorized because WebClient follows redirects without adding Authorization headers.

System.Management.Automation.MethodInvocationException: Exception calling "UploadString" with "3" argument(s): "The remote server returned an error: (401) Unauthorized

But this was not the case with pwsh 6.0.0. It did not fail at all.

To fix the issue, I switched to Invoke-RestMethod:

Invoke-RestMethod -Uri $url -MaximumRedirection 0 -Method $method -Body $json -Headers $headers

Now the script worked fine on Windows Powershell but pwsh 6.0.0 now failed with the very same 401 Unauthorized response.

Microsoft.PowerShell.Commands.HttpResponseException: Response status code does not indicate success: 401 (Unauthorized).
   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)

That means, it followed the redirect even though I used -MaximumRedirection 0.
Upgraded to pwsh 6.0.4, same issue in that version.

Installed pwsh-preview (6.1.0-preview.4) and now got the same exception but for the redirect response.

Microsoft.PowerShell.Commands.HttpResponseException: Response status code does not indicate success: 303 (See Other).
   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)

Conclusion:

| | Windows Powershell | pwsh 6.0.0-6.0.4 | pwsh 6.1.0 preview |
|----|----|----|----|
| WebClient.UploadString | Follows redirect without auth headers | Follows redirect with auth, no error | Follows redirect with auth, no error |
| Invoke-RestMethod -MaximumRedirection 0 | No follow, no error | Follows without auth, throws 401 | No follow, throws 303 |

Gah, these differences makes it so hard to write something that works/can be tested on all our platforms.

[markekraus]

@anderssonjohan that appears to be a different issue. The issue here is when an HTTPS site redirects to an HTTP site, regardless of the presence or lack of auth headers. Would you be so kind as to open a new issue for that, please?

[anderssonjohan]

@markekraus Thanks! Sure, I'll do that. It seemed related to me and I didn't want to add duplicate issues.

[CarlGustavAlbertDwarfsteinYung]

This issue is present in ps7 but not the ps bundled with windows.

[SteveL-MSFT]

@CarlGustavAlbertDwarfsteinYung WinPS uses a different HTTP client from .NET Framework that could be argued is doing the insecure thing. .NET Core has a new HTTP client that rejects this and we'd have to code around that.

[CarlGustavAlbertDwarfsteinYung]

@SteveL-MSFT Hi thank you for your reply. As far as i can undestand from what I read the issue is that PS expects a standardized HTTP implementation but in reality there is no standard. TSeems like PS gives a 302 Found for every redirect. The standard added 303 and 307 that is expected in response to 302. Mozilla recommends using 302 sparingly because every browser uses their own system for redirects. Of course I am not an expert in this project so I may be completely wrong.=