Powershell: Invoke-RestMethod does not strip Authorization Headers

[dave-tucker]

Steps to reproduce

Attempt to download something using Invoke-RestMethod that requires an Authorization header AND generates a 302 redirect to the final location of that resource.

One such example is downloading a build artifact from an AppVeyor project.
E.g from the example here

Invoke-RestMethod -Method Get -Uri "$apiUrl/buildjobs/$jobId/artifacts/$artifactFileName" `
-OutFile $localArtifactPath -Headers @{ "Authorization" = "Bearer $token" }

Expected behavior

The request should succeed like it does on Windows

Actual behavior

The request fails with the following error

Invoke-RestMethod : Response status code does not indicate success: 400 (Authentication information is not given in the correct
format. Check the value of Authorization header.).
At /Users/dave/Desktop/appveyor.ps1:31 char:1
+ Invoke-RestMethod -Method Get -Uri "$apiUrl/buildjobs/$jobId/artifact ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Method: GET, Re...rShell/6.0.0
}:HttpRequestMessage) [Invoke-RestMethod], HttpRequestException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Having debugged this with AppVeyor support it seems they send a 302 redirect to a location in Azure. On Windows, I've verified with Fiddler that the Authorization header is stripped which allows this command to succeed. On Mac it would appear the Authorization header is not stripped, causing Azure to generate the error.

Environment data

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.0.0-alpha
PSEdition                      Core
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   3.0.0.0
GitCommitId                    v6.0.0-alpha.9
CLRVersion
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

[ScriptAutomate]

This may be related, otherwise I can open a separate issue.

On PowerShell Core, on both Nano Server and Linux:

[Nano - GA Release - PowerShell Core]

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      14393  1000    

[Linux - PowerShell Core (alpha.11]

Major Minor Patch Label
----- ----- ----- -----
    6     0     0 alpha

Authorization header format should follow:
http://www.ietf.org/rfc/rfc2617.txt

If my Authorization header includes commas, in specific cases, Invoke-WebRequest / Invoke-RestMethod will give a syntax error but do not get this error when on Windows PowerShell.

# Fails
$Headers = @{}
$Headers.Add('Authorization', "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-date, Signature=fe5f80f77d5fa3beca038a248ff0
27d0445342fe2855ddc963176630326f1024")
Invoke-WebRequest -Uri www.google.com -Headers $Headers

# Success
$Headers = @{}
$Headers.Add('Authorization', "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request SignedHeaders=host;range;x-amz-date, Signature=fe5f80f77d5fa3beca038a248ff0
27d0445342fe2855ddc963176630326f1024")
Invoke-WebRequest -Uri www.google.com -Headers $Headers

All I do is remove a single comma after the Credential=VALUE portion, and it passes? Works either way on normal Windows PowerShell.

Note: Example pulled from http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

Workaround:
It worked when I removed the comma after the Credential=VALUE portion, on both Core and Windows PowerShell, without issue.

[SteveL-MSFT]

@ScriptAutomate you should open that as a separate issue

@dave-tucker I invested this and it seems to be an issue with AppVeyor's REST API (or potentially libcurl). In anycase, if you don't specify the Authorization header, it succeeds. If you specify the Authorization header with curl, it similarly fails. I created a PR to fix their sample https://github.com/appveyor/website/pull/255

[SteveL-MSFT]

Looks like the problem is with libcurl which .Net Core depends on. The Auth header is needed by AppVeyor and after auth, they redirect to the Azure Blob Storage which doesn't need it, but because it's there tries to validate and fails. I'll follow-up with libcurl.

[SteveL-MSFT]

It doesn't look like libcurl is going to change this behavior. We can handle the 302 redirect in the cmdlets and remove the Auth header by default (make it consistent with Windows) and add a switch to include it on redirect if desired

[SteveL-MSFT]

Perhaps by default, if the redirect is to same domain, we preserve the headers; if to different domain, we clear them. Also expose -ClearHeadersOnRedirect switch

[dantraMSFT]

I recommend a less intrusive fix. Add a -ClearAuthorizationOnRedirect and strip the authorization header on the first redirect. This means we only handle the first redirect and allow the lower level to handle any subsequent redirects.
As far as the domain change, I suggest we defer that until we have better use cases. The above will address the appveyor->azure issue which 'should' be the majority of cases.
@joeyaiello and @SteveL-MSFT what do you think?

[SteveL-MSFT]

@PowerShell/powershell-committee should review:

1. what is the default behavior? clear or keep? (Windows PowerShell appears to clear as this worked previously)
2. do we want a switch specific to the Auth header or perhaps consider -ClearHeaderOnRedirect which takes an array of headers?
3. should we have different default behavior depending on if the redirected URL is within the same domain or not?

[joeyaiello]

Per @PowerShell/powershell-committee conversation:

• By default, PS Core should do the same thing as Windows PowerShell for all headers
• We should introduce a switch to PS Core that adds the authorization headers back (or we should add a switch that adds all headers back if it turns out that Windows PS behavior is to strip all headers).
• We're not concerned with question 3 right now because it's enough of a edge that customers can handle it in their own scripts right now.

[dantraMSFT]

Based on the comments from the committee; I'm going to change the current implementation to default to the behavior and also provide a PreserveAuthorizationOnRedirect to handle edge cases.

The reason for the change is as follows:
On FullCLR, WebRequest is used and, under the hood, the Authorization header is automatically stripped when a redirect occurs. On CoreCLR, HTTPClient is used which doesn't mirror this behavior. To ensure compatibility, I'll update WebRequestPSCmdlet to disable redirects when it detects an authorization header, handle the first redirect by removing the authorization and resubmitting the request to the redirected location with redirection reenabled.

The PreserveAuthorizationOnRedirect will disable this logic for the request for edge cases where the redirected request needs to include the authorization header.

NOTE: There are various discussions floating around about the reverse problem; the header is stripped and callers must handle redirects manually in code. The switch is intended to address this use case when the cmdlet is used.

[joeyaiello]

I think this needs to be merged into the PowerShell master branch before we can close or resolve it.