Powershell: Invoke-WebRequest should directly support Basic authentication

[rkeithhill]

We are working with more and more "Linux" based servers where we get instructions to run curl to say get back content for a .npmrc file from a product called Artifactory. This product uses Basic auth to accept credentials.

The instructions they give are:

curl -u<USERNAME>:<PASSWORD> http://artifactory.it.keysight.com:8081/artifactory/api/npm/auth

To get this to work in PowerShell using Invoke-WebRequest, this is the best I could come up with:

$userpassB64 = [byte[]][char[]]"hillr:OpenSesame" | ConvertTo-Base64 -NoLineBreak
$res = Invoke-WebRequest http://artifactory.my.company.com:8081/artifactory/api/npm/auth `
       -Headers @{Authorization = "Basic $userpassB64"}

First, ConvertTo-Base64 is a PSCX command (we really need B64 encode/decode in PS Core). Second, Basic auth should be built-in like it is in CURL.

This what I'd like to see from Invoke-WebRequest:

$res = Invoke-WebRequest http://artifactory.my.company.com:8081/artifactory/api/npm/auth `
       -UseBasicAuth -Credential (Get-Credential)

This -UseBasicAuth switch should also support the -UseDefaultCredentials parameter. However the trick is whether or not to include the Domain\ in the username field. That option, when coupled with -UseBasicAuth "probably" should not include the domain. In our scenario, Artifactory will fail to authenticate if the domain is specified. Either way, the user can always fall back to explicitly providing the username with -Credential (Get-Credential).

FYI, the curl docs on -u:

-u, --user
Specify the user name and password to use for server authentication. Overrides -n, --netrc and --netrc-optional.
If you simply specify the user name, curl will prompt for a password.
The user name and passwords are split up on the first colon, which makes it impossible to use a colon in the user name with this option. The password can, still.
When using Kerberos V5 with a Windows based server you should include the Windows domain name in the user name, in order for the server to successfully obtain a Kerberos Ticket. If you don't then the initial authentication handshake may fail.
When using NTLM, the user name can be specified simply as the user name, without the domain, if there is a single domain and forest in your setup for example.
To specify the domain name use either Down-Level Logon Name or UPN (User Principal Name) formats. For example, EXAMPLE\user and [email protected] respectively.
If you use a Windows SSPI-enabled curl binary and perform Kerberos V5, Negotiate, NTLM or Digest authentication then you can tell curl to select the user name and password from your environment by specifying a single colon with this option: "-u :".
If this option is used several times, the last one will be used.

[dragonwolf83]

We are having the same situation with Device42, an inventory management software. The REST API is requiring the same methods as above to login.

In my case, I didn't use the PSCX extensions though and just did the Base64 conversions in .NET manually. Then, I wrapped it in it's own function, so that all my Device42 cmdlets can reuse the logic.

The -UseBasicAuth parameter is needed for both Invoke-WebRequest and Invoke-RestMethod.

@rkeithhill based on the doc you linked, I think -UseBasicAuth should be a string with a [ValidateSet] attribute that can allow user to set the different methods of Basic Authentication that exist.

[SteveL-MSFT]

In the future, we may want to support other forms of Auth, so perhaps have a -Authentication parameter. Note that Basic won't work with -UseDefaultCredentials as I believe there's no way to retrieve the password to send to the remote target which is required for Basic.

[rkeithhill]

@SteveL-MSFT That makes sense.

[RandomNoun7]

Add Jenkins and even Github to the list of popular tools that would be easier to work with if iwr and irm supported something like -Authentication Basic

Also, since neither of these tools are really Linux based (Jenkins is Java which I run on my Windows servers), it points out that this is becoming a widespread issue with a lot of the OSS and otherwise community based tools that Windows folks are increasingly trying to work with using PowerShell.

[DexterPOSH]

+1
Having the -Authentication parameter would make sense.

[treestryder]

Basic Auth would be nice, but my current struggle is with "OASIS Web Services Security" authentication for Workday's API.

https://www.oasis-open.org/standards#wssv1.1.1

[APraestegaard]

+1

[vMarkusK]

Maybe the Syntax "-BasicAuth (get-credential)" is more intuitive. But a great enhancement!

[mkellerman]

+1 for -Authentication

[bitzbyte]

+1 for -Authentication. I have 3 different services that I have to manually Base64 encode credentials for routinely (multiple times per week for each of them) and more that come up on a less frequent basis.

[swinster]

The crosslinked issue #2112 above gives an example that without adding Basic Authorisation headers to requests, it results in a dual request, the first being challenged then the second with the header added. If you are then attempting to either upload or download very large files (in my case anything from a 1 GB upload or 3-4 GB download is normal), this seems to result in the entire file being uploaded or downloaded twice - a huge waste of time and resource.

[SteveL-MSFT]

We can address this in the 6.1.0 timeframe

[markekraus]

This one has finally made it to the top of my to-do list.

Since this works primarily with the Authorization header, I think -Authorization is a more appropriate name for the parameter.

Thoughts, comments?

[SteveL-MSFT]

In the future, we may want to support other authn schemes like OAuth. So my preference is to have something extensible. For the Basic scenario, one option is to support:

invoke-restmethod -authentication basic -credential user
# this will prompt as it would today with a PSCredential

Of course, creating a PSCredential from cleartext user/pass takes a few steps:

$password = "mypassword" | ConvertTo-SecureString -asPlainText -Force
$cred = [pscredential]::new("myuser",$password)

Seems like a bad practice to make it easy to create a pscredential with cleartext as then cleartext passwords show up in scripts, but I'm open to discussion on that as a separate issue.

So then the question is how would this work for OAuth. We could still take a PSCredential and just ignore the username where the token is put into the password, but this isn't obvious:

$oauthtoken = "tokenincleartext" | ConvertTo-SecureString -asPlainText -Force
$cred = [pscredential]::new("$null is not accepted for username",$oauthtoken)
invoke-restmethod -authentication oauth -credential $cred

Perhaps an acceptable solution is to support PSCredential, but maybe also Token?

invoke-restmethod -auth basic -token user:passwordincleartext
invoke-restmethod -auth oauth -token oauthtoken

Although it uses the Authorization header, authorization is what happens on the server side...

cc @HemantMahawar @LeeHolmes @joeyaiello

[markekraus]

@SteveL-MSFT

My preference has always been to discourage secrets in plain text by not accepting them. If the user wants to go out of their way to create a PSCredentail using plain text, that's on them. If we accept plain text secrets, that's on us. Though, this kind of breaks down if they can't be imported and exported conveniently (last I checked, you couldn't import/export secure strings in Linux #1654 ).

In my modules that wrap the web cmdlets I accept a credential object and use the password for the OAuth token. Depending on the API, OAuth access tokens are closer to long lived passwords than they should be (lasting up to 90 days in some ☹️ ).

In my current code, I'm doing the same thing. and since Basic and Bearer often go hand-in-hand I was planning to implement both at once.

Invoke-RestMethod -Uri $uri -Authorization Bearer -Credential $credential

[SteveL-MSFT]

@markekraus agree that only adopting the PSCredential pattern is best, Basic and Bearer makes sense although I wonder if Token is a better name for discovery reasons as I'm sure most people not familiar with how the authz headers work won't equate Bearer to OAuth

[markekraus]

@SteveL-MSFT I was thinking maybe OAuth could be an alias for Bearer. It's not easily resolved because many APIs require a Basic auth with the a combination of user/pass or client-id/client-secret and that is just as much OAuth as Bearer in some users' eyes.

Token could work. But it would be highly specialized for just OAuth. For Basic, there are many APIs that use it outside the context of OAuth where token makes no sense and Credential does. It could be a sort of "alias" for -Credential in that it is a real separate parameter but under the hood implements the same object with some error detection if both are supplied. but I feel that somewhat over complicates things.

[RandomNoun7]

+1 for:

Invoke-RestMethod -Authorization Basic -credential $credential
Invoke-RestMethod -Athorization OAuth -token $OAuthToken

Using a PSCredential object with a username that is ignored isn't very intuitive.

[markekraus]

Using a PSCredential object with a username that is ignored isn't very intuitive.

-Token would need to be a SecureString then.

[RandomNoun7]

Insisting on a secure string for -token seems ok.

If you were to not support -token and insist on -credential for OAuth, my impression is that most scripters are going to take a trip through ConvertTo-secureString anyway to create the credential object. Whether they are getting the Token from a file stored in plain text on disk, or from a password manager API, that's on them, but using a PSCredential object for both purposes doesn't solve that issue, as I think many many script authors will use the same or similar process either way.

So for my uses, whether I have to use a SecureString with -token, or a Credential object, it doesn't practically change the way I retrieve and handle that secret in my scripts.

[devblackops]

My two cents:

I don't think accepting a PSCredential -Token parameter where we only really need the password property makes sense. [SecureString] makes more sense to me as that is the only part we care about. If the user already has the token in a credential object, they can just pass in the securestring password with:

Invoke-RestMethod -Authentication OAuth -Token $cred.Password

Here are a couple of scenarios that I think Invoke-RestMethod and Invoke-WebRequest should support:

# Credential is a [PSCredential]
Invoke-RestMethod -Authorization Basic -Credential $cred
# Behind the scenes do the base64 encoding stuff

# Token is a [SecureString]
Invoke-RestMethod -Authorization OAuth -Token $token
# Under the covers set header Authorization = "Bearer $token"

[markekraus]

@devblackops that looks like +3-ish for that. I'm persuaded. I will modify that.

[devblackops]

@markekraus Cool. It may make the implementation more complicated but I think the intent is clearer.

[rkeithhill]

I wonder if the implementation should warn that with -Authorization Basic that the unencrypted password will be passed across the wire in base64 encoding (which is easily unencoded)? Of course, if you did this you'd need a way to suppress that warning. ConvertTo-SecureString uses -Force for this:

38:335ms> ConvertTo-SecureString "open sesame" -AsPlainText
ConvertTo-SecureString : The system cannot protect plain text input. To suppress this warning and convert the plain
text to a SecureString, reissue the command specifying the Force parameter. For more information ,type: get-help
ConvertTo-SecureString.
At line:1 char:1
+ ConvertTo-SecureString "open sesame" -AsPlainText
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [ConvertTo-SecureString], ArgumentException
    + FullyQualifiedErrorId : ImportSecureString_ForceRequired,Microsoft.PowerShell.Commands.ConvertToSecureStringComm
   and

39:308ms> ConvertTo-SecureString "open sesame" -AsPlainText -Force
System.Security.SecureString

[SteveL-MSFT]

I think if the address is https, I wouldn't provide a warning. If it's http, I think we should require -AllowUnecryptedBasic (or something scary like that)

[markekraus]

@rkeithhill hmm I initially agreed but the more I wrote and rewrote my reply, the more I started to disagree.

The current -Credential doesn't warn (it actually does send Basic auth.. just.. only when challenged with Www-Authenticate: Basic realm=). It should probably warn for anytime -Credential is supplied. That would get annoying for console users accessing a site/api that has basic auth though. At worst It might lead them to disabling all warnings. 👎 Adding a -force seems a bit.. awkward... for the web cmdlets. hrm

Part me also feels that it should just be a given that these credentials are being sent insecurely. I realize that's not the case, but, most forms are not doing any kind of client side encryption or encoding either before passing on credentials over the pipe as form fields.

Maybe if it warned when not going over HTTPS?

I'd like more feedback from others on this one.

[rkeithhill]

@SteveL-MSFT Good point. Most of the time I'm hitting internal servers were we're too cheap to buy SSL certs. :-) I like the -AllowUnencryptedBasic suggestion.

[markekraus]

@SteveL-MSFT and @rkeithhill how about -AllowUnencrypedAuthorization? I would like this to apply to OAuth as well... sending OAuth tokens over non-HTTPS is just as dangerous.

[dragonwolf83]

I still prefer a -Token parameter on PSCredential for a few reasons.

1. Don't need to learn yet another method for credential handling.
2. Messing with SecureString is not as clean since you have to use -AsPlainText -Force for passing it in interactively.
3. Reusability in custom cmdlets for other scenarios where a token or passphrase is needed. Custom SSH cmdlets come to mind. Everyone gets confused about how to do passphrase securely with PSCredential.
4. Create a standard method like GetNetworkCredential() called GetToken(). You can then use this in the web cmdlets to keep it encrypted until the last moment needed. I can then reuse this for a custom cmdlet.

$cred = Get-Credential -Token -Authentication OAuth
$response = Invoke-RestMethod -Credential $cred

Another idea to is you could put the -Authentication parameter in PSCredential too. Not sure if there is a situation where you would want to reuse the same credential across authentication types.

$cred = Get-Credential -Token 
$response = Invoke-RestMethod -Credential $cred -Authentication OAuth

Compare that to SecureString madness

$token = ConvertTo-SecureString -AsPlainText -Force
$response = Invoke-RestMethod -Token $token -Authentication OAuth

[dragonwolf83]

@markekraus how about if HTTP warn by default and then have a parameter -IgnoreUnsecureWarnings
Unsecure or Insecure?

@rkeithhill I guess we need some Let’s Encrypt cmdlets!

[markekraus]

@dragonwolf83 For interactive secure strings:

$Creds = Read-Host -AsSecureString "Token"

I don't think changing the behavior of PSCredential for this makes sense. SecureString is exactly what should be used for this kind of thing. I don't think the usecase for manually entering access tokens is a significant one. Most tokens would be extremely painful to enter interactively.

[RandomNoun7]

What if you could use this as an excuse to start moving in the direction browsers are starting to go, where they are warning users about any and all pages that are http?

A credential going over an un-encrypted connection is just the most obvious danger, but if it's the cmdlet authors job to warn users about things that are dangerous, you have to admit that you don't know what they are putting in the -Body parameter either.

To me, that argues for something like the following behavior

1. Implement a new parameter called -AllowUnEncryptedConnections
2. Any request whatever, to an http site would result in output to the Warning Stream wagging a finger at the user. Why any and all requests? Because even a regular GET to a non-ssl site can reveal compromising data if the returned page content contains sensitive data.
3. Any request, including those with credentials, that use the -AllowUnEncryptedConnections switch would not have that warning output.

To me, that allows the cmdlet to take the threat from unencrypted connections seriously and treat it comprehensively. Writing a warning rather than an error seems a good compromise rather than blocking the cmdlet from executing and breaking lots of people's scripts.

That said, I realize this is a bit of an extreme position to take, so I'm fine with an -AllowUnencrypedAuthorization parameter too if that's as far you are willing to push it.

[SteveL-MSFT]

For interactive use, I think we should prompt for password or token so user doesn't have to create the securestring.

[markekraus]

@SteveL-MSFT I'm trying to imagine interactive entry of an OAuth Token, and I'm thinking that is an edge case. Most of the APIs I've been exposed to have massive 60+ character tokens that are not valid for long. I mean, I'm not against adding it, i just don't see it being useful.

Is there a good way to check for an interactive session in the Cmdlet?

[rkeithhill]

There is an existing pattern for this that applies to all the PowerShell commands that take a credential e.g. Command -Credential (Get-Credential).

[markekraus]

@RandomNoun7 RE: HTTP warnings. I think that is a good idea, but perhaps that should be tracked as a separate issue and not coupled with the addition basic and bearer authentication.

@rkeithhill then irm -Token (Read-Host -AsSecureString) ...?

[markekraus]

Ok.. the latest commit has everything except the HTTP warning/block or the "prompting if not provided" (Basic, OAuth, Bearer, -Token,-Credential, errors, tests.. etc.)

[markekraus]

And now I have a super scary error just in time for Halloween 🎃
https://github.com/PowerShell/PowerShell/compare/master...markekraus:BasicAuthentication

@SteveL-MSFT I don't think prompting for the token makes sense. It would complicate the cmdlet code to handle something that should be done outside of it. As @rkeithhill mentioned, there is already an existing pattern for this kind of behavior and I still believe manually entering a token would be an edge case.

I'll wait until tomorrow to cleanup and submit the PR to allow for more comments/suggestions.

[SteveL-MSFT]

I'm ok without the prompting. Thanks for working on this!

[devblackops]

Thank you @markekraus! This will be a great addition.

[rkeithhill]

Heck yeah! Thanks @markekraus. I sure hope this make it into 6.0.0.

[markekraus]

@rkeithhill me too. This has been on my PowerShell wishlist for a long time. I have been wrapping this functionality for awhile. I can't wait to rip that out of my own code once it's finally baked into the cmdlets.

[RandomNoun7]

@markekraus This is going to be fantastic. Thanks for working on this.

[SteveL-MSFT]

@markekraus I'm assigning this to myself since GitHub won't let me put your name there so that no one duplicates your work

[markekraus]

PR #5052