Currently users only have 2 options for dealing with Server SSL/TLS Certificates with Invoke-WebRequest and Invoke-RestMethod: the default validation and to skip validation. Some scenarios warrant tighter security on web requests where a certificate is not fully trusted by the host environment but is known to be trusted by the user. This could include internal web APIs that use a self signed certificate with a specific thumbprint or from a known CA that is not trusted by the host. Or if a user wishes to ensure a certain CA/Thumbprint/Subject is blocked (a known bad actor).
Also [System.Net.ServicePointManager]::ServerCertificateValidationCallback has no effect in Core and HttpClient only uses the settings provided by HttpClientHandler.
Add a parameter of type Func<HttpRequestMessage,X509Certificate2,X509Chain,SslPolicyErrors,Boolean> that accepts a ScriptBlock to both Web Cmdlets. This is to be set on HttpClientHandler.ServerCertificateCustomValidationCallback. The -SkipCertificateCheck would have priority, meaning if both were supplied either a parameter exception is thrown or -SkipCertificateCheck would be applied and the callback ignored.
The Parameter will be named CertificateValidationScript
-SkipCertificateCheck?For name of the parameter we could start discussion with ServerCertificateCustomValidationCallback.
As for ScriptBlock we need remember about binary cmdlets too.
I don't think Callback fits well into the PowerShell user's domain. Also, Custom is a bit redundant IMO.
How about ServerCertificateValidationAction? Or may since the other parameter is just -SkipCertificateCheck (not SkipServerCertificateCheck), maybe the parameter should be just CertificateValidationAction?
@iSazonov I believe there is a way to accept both binary and shell friendly callbacks. I need to find an example to borrow from, but I'm sure I have seen that done somewhere before. I just can't remember where. But before I go down that rabit hole, is this something that really needs considering? The Web Cmdlets inherit from PSCmdLet, not Cmdlet there is a bit of an understanding there that it will be running inside a PowerShell runspace, a ScriptBlock in that scenario is probably no more or less cumbersome than Func<HttpRequestMessage,X509Certificate2,X509Chain,SslPolicyErrors,Boolean>
@rkeithhill Action hmm. I don't think that is a PowerShell friendly name either. Script maybe. CertificateValidationScript, CertificateValidationOverride hmm.
Action is used in a number of commands including Set-PSBreakpoint and Register-ObjectEvent. Fortunately, this sort of thing is easy to analyze in PowerShell e.g.:
15:10ms> (gcm -CommandType Cmdlet).ParameterSets.Parameters.Where({$_.ParameterType -match 'ScriptBlock'}) | Group Name
| Sort Count -Descending
Count Name Group
----- ---- -----
15 ScriptBlock {System.Management.Automation.CommandParameterInfo, System.Management.Automation.Com...
7 Action {System.Management.Automation.CommandParameterInfo, System.Management.Automation.Com...
3 InitializationScript {System.Management.Automation.CommandParameterInfo, System.Management.Automation.Com...
3 Expression {System.Management.Automation.CommandParameterInfo, System.Management.Automation.Com...
1 TransactedScript {System.Management.Automation.CommandParameterInfo}
1 FilterScript {System.Management.Automation.CommandParameterInfo}
1 End {System.Management.Automation.CommandParameterInfo}
1 Process {System.Management.Automation.CommandParameterInfo}
1 Begin {System.Management.Automation.CommandParameterInfo}
1 RemainingScripts {System.Management.Automation.CommandParameterInfo}
However, both Action and ScriptBlock are used as the entire parameter name. OTOH Script is used as a common suffix. So your suggestion of CertificateValidationScript is probably the best parameter name.
As for ScriptBlock we need remember about binary cmdlets too.
You can always expose a public property that is not marked with [Parameter] that is a delegate type.
Since I don't deal with binaries that talk directly to PowerShell, what exactly is the problem?
From what I can tell, if I type the parameter as a Func<HttpRequestMessage,X509Certificate2,X509Chain,SslPolicyErrors,Boolean> which is the same type as HttpClientHandler.ServerCertificateCustomValidationCallback, PowerShell script users will be able to pass a ScriptBlock to it.
Will this some how cause problems for Binary users?
When you are writing a binary cmdlet you can invoke other binary cmdlets. While you can create ScriptBlocks in C#, I guess it might be a little clunky to use that instead of a delegate. OTOH I'm not sure how common this scenario is. I've done this before but pretty rarely and never with something that took a delegate/ScriptBlock. OTOH if your parameter type is already a delegate then problem solved. :-)
All good then, that was the plan all along. I just wasn't clear in my proposal. 馃槃
Side note: The Web Cmdlets derive from PSCmdlet and not Cmdlet and cannot be invoked that way. They can only be invoked using PowerShell.Create(), AddCommand(), and AddParameter(). I guess a ScriptBlock would still be problematic that way.
OK, I have a working example here.
1 Problem: since the callback is run async the runspace is not available in that thread. To work around this, I am wrapping the provided delegate with code that creates the runspace, calls the delegate, then cleans up the runspace. This works fine. However, this means that there is no access to the current session state (variables, modules, functions, etc).
I'm not sure it needs to. If this is properly documented that the code in the script runs in its own context, it should be fine. I don't want to make an overcomplicated feature for something that has a limited target audience.
Thoughts and Feedback welcome.
# emulate -SkipCertificateCheck
$script = { return $True }
Invoke-RestMethod https://expired.badssl.com/ -CertificateValidationScript $Script
# Block all Certs
$script = { return $False }
Invoke-RestMethod https://google.com/ -CertificateValidationScript $Script
# -SkipCertificateCheck overrides -CertificateValidationScript
$script = { return $False }
Invoke-RestMethod https://expired.badssl.com/ -CertificateValidationScript $Script -SkipCertificateCheck
# Accept a .NET Delegate:
Invoke-RestMethod https://expired.badssl.com/ -CertificateValidationScript ([System.Net.Http.HttpClientHandler]::DangerousAcceptAnyServerCertificateValidator)
@lzybkr Could you please comment - what is right way to implement the callback parameter?
I'm not sure how important it is to support delegate callbacks - I'd imagine these cmdlets are mostly called from PowerShell scripts, so I would just take a scriptblock.
This mostly solves the problem of invoking the scriptblock without a runspace because PowerShell marshals the scriptblock.invoke() call back to the runspace creating the scriptblock - though this can cause a hang if that runspace is blocked in a .Net method.
@lzybkr so if I set handler.ServerCertificateCustomValidationCallback with the supplied script block, and that callback is called async, it will work with the current scope and without needing to initialize a runspace?
It depends on what is happening on the cmdlet's thread.
First, you will need to convert the scriptblock to the appropriate delegate type with LanguagePrimitives.ConvertTo<DelegateType>(scriptblock).
But as I mentioned - if the cmdlet's thread is stuck in code outside of PowerShell, there is no way for that thread to invoke the scriptblock - it will instead hang waiting until PowerShell gets control of the thread again.
In an ideal case, you have:
Pipeline thread (cmdlet's thread):
Starts some async code.
Receive and write some output.
Background thread
scriptblock.Invoke() - sends event to Runspace thread, blocking until event is processed
The important thing is to reach this code on the pipeline thread. This happens naturally if you execute some PowerShell script or write to the pipeline.
If neither of those things are happening, you need to do something like this assuming you can regain control after calling into some non-PowerShell code. Async methods normally give you control, so hopefully you can make this work.
So this ended up working for me:
Func<HttpRequestMessage,X509Certificate2,X509Chain,SslPolicyErrors,bool> certificateValidationDelegate = LanguagePrimitives.ConvertTo<Func<HttpRequestMessage,X509Certificate2,X509Chain,SslPolicyErrors,bool>>(CertificateValidationScript);
// This wraps the supplied CertificateValidationScript and sets the PowerShell runspace in the async callback.
// This allows for script users to supply a ScriptBlock and have it properly execute in the async thread.
Runspace defaultRunspace = Runspace.DefaultRunspace;
Func<HttpRequestMessage,X509Certificate2,X509Chain,SslPolicyErrors,bool> validationCallBackWrapper =
delegate(HttpRequestMessage httpRequestMessage, X509Certificate2 x509Certificate2, X509Chain x509Chain, SslPolicyErrors sslPolicyErrors)
{
Runspace.DefaultRunspace = defaultRunspace;
Boolean result = certificateValidationDelegate.Invoke(httpRequestMessage, x509Certificate2, x509Chain, sslPolicyErrors);
return result;
};
handler.ServerCertificateCustomValidationCallback = validationCallBackWrapper;
I thought maybe there would be some kind of hang or crash, but it appears to work without issue and the ScriptBlock runs with access to the calling RunSpace.
@iSazonov and @lzybkr Do you see anything wrong with this implementation? If not, I will clean it up, add tests, and do a PR.
Does "using:variable" work?
I don't think so, but it's not needed.
$Script = { return $condition }
# will accept any cert
$condition = $true
Invoke-RestMethod https://expired.badssl.com/ -CertificateValidationScript $Script
# will block any cert
$condition = $false
Invoke-RestMethod https://google.com/ -CertificateValidationScript $Script
# Set a value ion calling scope
$resulthash = @{}
$Script = { $resulthash['thumbprint'] = $args[1].Thumbprint; return $true }
Invoke-RestMethod https://google.com/ -CertificateValidationScript $Script
$resulthash['thumbprint']
These all work as would be expected., The first one bypasses the bad cert. the second one blocks the good cert, and the third one populates $resulthash['thumbprint'] with the cert thumbprint.
As far as I can tell, the using: implementation only works in certain special ScriptBlock scenarios (such as those used in remote sessions). and is not a standard feature. It also doesn't look simple to implement.
@iSazonov Actually, this made me realize something.
$condition = $true
$Script = { return $using:condition }
Invoke-RestMethod https://expired.badssl.com/ -CertificateValidationScript $Script
This hangs the thread. I'm guessing it's because the ScriptBlock throws a UsingWithoutInvokeCommand exception and never returns true or false from the delegate. A the very least I need to do some exception handling in the wrapper delegate.
My concern is that users will expect that "using:" will work. Although if the script block is in the same context it don't make sense - silently ignore by exception handling?
@iSazonov I don't think users should have that expectation, and if they do, then we would need to add it as a standard feature to all ScriptBlocks. $using: only works in very limited use cases (Invoke-Command and DSC). The documentation about $using: is about_Remote_Variables and in this case, the variables are all local.
We could at least call it out in documentation that it is not supported in this instance. But, my opinion is that users should assume it is not available unless the documentation explicitly says that it is.
The plan is to treat all exceptions from the ScriptBlock as a failure. It would err on the side of caution and allow for throw be used. Since $using: creates an exception in most cases, it would result in an ssl fail:
$condition = $true
$Script = { return $using:condition }
Invoke-RestMethod https://expired.badssl.com/ -CertificateValidationScript $Script
That would fail. To understand why, a user would only need to run $Script.Invoke() to find out that it is causing an exception.
I agree that if we document the parameter properly the behavior is good.
The same about param().
param() works
$Script = {
param(
[System.Net.Http.HttpRequestMessage]
$HttpRequestMessage,
[System.Security.Cryptography.X509Certificates.X509Certificate2]
$X509Certificate2,
[System.Security.Cryptography.X509Certificates.X509Chain]
$X509Chain,
[System.Net.Security.SslPolicyErrors]
$SslPolicyErrors
)
Return (
$HttpRequestMessage.RequestUri.AbsoluteUri -eq 'https://www.google.com/' -and
$X509Certificate2.Subject -eq 'CN=www.google.com, O=Google Inc, L=Mountain View, S=California, C=US' -and
$X509Chain.ChainElements[2].Certificate.Thumbprint -eq 'DE28F4A4FFE5B92FA3C503D1A349A7F9962A8212' -and
$SslPolicyErrors -eq 'None'
)
}
Invoke-RestMethod https://www.google.com/ -CertificateValidationScript $Script
pr: #4970
@markekraus Is the intent that the usual certificate checks are omitted when the -CertificateValidateScript is provided? If so, is there some way to invoke those checks from the user scriptblock?
The reason I am asking is that Invoke-WebRequest is currently very permissive of bad certificates (you can run this test for details) compared with browsers.
I like the idea of using -CertificateValidateScript to perform _additional_ certificate verification to compensate for its current permissiveness but I wouldn't want to give up the checks that Invoke-WebRequest already performs.
It seems like being able to enable and disable the built-in certificate check independently of -CertificateValidateScript would be the most flexible, but #4970 includes a test called "Verifies Invoke-WebRequest -CertificateValidationScript is ignored when -SkipCertificateCheck is present" which seems to imply that's not how it's intended to work.
@alx9r The intent is that the ScriptBlock completely replaces the usual logic associated with certificates with that of the logic supplied in the block.
However, you have access to the SslPolicyErrors object. The default check is to ensure that it is None
This is the equivalent PowerShell ScriptBlock implementation through my proposed PR
$Script = { $SslPolicyErrors -eq 'None' }
Invoke-RestMethod -Uri https://contoso.com -CertificateValidationScript $Script
Using that, the behavior would not change from normal operation. That means you can use the $SslPolicyErrors as a fallback like this:
$Script = {
if ($X509Certificate2.Subject -match 'Contoso') {
return $true
}
else {
return ($SslPolicyErrors -eq 'None')
}
}
Invoke-RestMethod -CertificateValidationScript $Script -Uri https://contoso.com/
That would accept any certificate with Contoso in the subject and for any other certificate it would be processed as normal.
Using SslPolicyErrors object is not obvious. Does it make sense to put it into parameters?
@alx9r Regarding the -SkipCertificateCheck, that has precedence. meaning, that if it is supplied all validation is ignored and any certificate will be accepted.
PowerShell Core uses HttpClient and the current implementation of the Web Cmdlets creates a one-time-use HttpClientHandler for each call to the Web Cmdlets. The HttpClientHandler is the mechanism through which the certificate validation callback is implimented. That means that we have the option of implementing validation on a per-call basis.
In Windows PowerShell this was done through [System.NetServicePointManager]::ServerCertificateValidationCallback and that callback persisted through all calls to the Web Cmdlets. To emulate the behavior in core you would use the following:
$Script = { <# whatever validation code #> }
$PSDefaultParameterValues['*:ServerCertificateValidationCallback'] = $Script
To allow for a one off command override the default, -SkipCertificateCheck is used.
@iSazonov Parameters for what? the SslPolicyErrors object is passed to the user supplied callback by the HttpClientHandler during SSL key exchange. In my PR it is already being passed as $SslPolicyErrors
and as $args[3] here
and the user can accept it in param()
$Script = {
param(
[System.Net.Http.HttpRequestMessage]
$Message,
[System.Security.Cryptography.X509Certificates.X509Certificate2]
$Cert,
[System.Security.Cryptography.X509Certificates.X509Chain]
$Chain,
[System.Net.Security.SslPolicyErrors]
$PolicyErrors
)
<# Validation code #>
}
The user can even mix and match in the same script if they want.
$Script = {
param(
[System.Net.Http.HttpRequestMessage]
$Message,
[System.Security.Cryptography.X509Certificates.X509Certificate2]
$Cert,
[System.Security.Cryptography.X509Certificates.X509Chain]
$Chain,
[System.Net.Security.SslPolicyErrors]
$PolicyErrors
)
if ($args[3] -eq 'None' -and $SslPolicyErrors -eq 'None' -and $PolicyErrors -eq 'None') {
return $true
}
else {
return $false
}
}
There is nothing obvious about any of this. Certificate validation callbacks are an advanced concept. Users will be required to learn about the 4 objects and draw from examples in the C# world. We can fix some of this with documentation, but most users who will use this feature are already advanced in knowing the dangers of -SkipCertificateCheck or may be familiar with them from C# or with [System.NetServicePointManager]::ServerCertificateValidationCallback in Windows PowerShell.
PowerShell allow do complex things easily. We should thing how make the callback easy to use.
-AfterCertificateValidateScript - validate after standard checks.
-ReplaceCertificateValidateScript - validate instead of standard checks.
@iSazonov I don't think we should over-complicate these cmdlets with more parameters for a feature that has a limited user scope that is already advanced. If I'm new to the language and I see the 2 primary web cmdlets have some these parameters:
-CertificateThumbprint
-SkipCertificateCheck
-AfterCertificateValidateScript
-BeforeCertificateValidateScript
-ReplaceCertificateValidateScript
I'm going to be very overwhelmed.
I was going to suggest an about_ topic that would include how to implement around $SslPolicyErrors. I personally think a singe replacement callback is all that's needed so long as it's documented.
Can you just add examples to the help to show what scriptblock is needed for these particular cases? That would be more useful than an about topic unless it requires a lot of supporting text. Honestly, the issue with about topics is that folks don't find them that often compared to the help for the command.
It, unfortunately, requires a bunch of supporting text, more than is acceptable for a parameter. The idea I had was to add the about_ topic and have that listed in the parameter help which would also have some small examples and details. "For more examples and details see get-help about_CertificateValidationScript".
If the web cmdlets didn't already have a heroic epics worth of text in their help, maybe this could all be in the parameter help.
Maybe adding another auto-variable like an $isValidCertificate bool could help. a little? It at least eases the SslPolicyErrors burden.
edit: I meant to say that i planed to include the before and after scenarios as Examples in the help.
@markekraus perhaps you can submit PR to https://github.com/PowerShell/PowerShell-Docs/blob/staging/reference/6/Microsoft.PowerShell.Utility/Invoke-RestMethod.md and add some examples?
The best traditional approach to creating documentation is to have three guides - a description of the syntax (fast discover features, syntax and short examples), a user guide (full feature description), and a description of the scenarios (HowTo).
@SteveL-MSFT I was planning to do that after the PR here was merged. But I guess there is no harm in doing it beforehand.
@iSazonov This was my planned approach:
It seems in alignment with what you suggested.
Here is a draft of the Invoke-RestMethod documentation.
https://github.com/PowerShell/PowerShell-Docs/compare/staging...markekraus:CertificatevalidationScriptDraft
@markekraus, awesome! Looks good enough to submit as PR. One thing I noticed is the capitalization of the word Four
@SteveL-MSFT Should I wait for #4970 to be approved first before making the PR to PowerShell-Docs?
(I made some fixes including the Four capitalization and added the equivalent Invoke-WebRequest documentation`).
Better wait merge - we can get useful feedback.
FWIW, I have implemented ServerCertificateValidationCallback calling into a user-supplied PowerShell scriptblock for this other project. There were a few things that weren't obvious at first that I had to overcome, so I will note them here in case it helps someone.
Runspace, and PowerShell InstanceServerCertificateValidationCallback can be called on a different thread from the one that initiated the connection to the server. The remarks in the Runspace.DefaultRunspace documentation include the following statement:
The Runspace used to set this property should not be shared between different threads.
Clearly the callback needs its own runspace which suggests the solution might be as simple as calling
```C#
Runspace.DefaultRunspace = RunspaceFactory.CreateRunspace()
scriptBlock.Invoke()
in the callback. The solution is not quite that simple and doing this in the callback causes PowerShell to do strange things or crash in many scenarios. I _think_ this is the result of the same state being accessed by the two threads in a non-threadsafe manner. In any case, @proxb [seems to have multithreading working reliably in PowerShell](https://github.com/proxb/PoshRSJob) and his examples use one `PowerShell` instance from `PowerShell.Create()` for each thread. I used those principles to create [a C# scriptblock invoker class](https://github.com/alx9r/BootstraPS/blob/3bbb0769fbc0cc7f0c69fba1c1826b7ca27dd227/BootstraPS.psm1#L328-L483) whose [invoke method](https://github.com/alx9r/BootstraPS/blob/3bbb0769fbc0cc7f0c69fba1c1826b7ca27dd227/BootstraPS.psm1#L412-L461) does something like this:
```C#
public void Invoke()
{
// ...
var iss = InitialSessionState.CreateDefault()
// define variables and functions, import modules
using (var rs = RunspaceFactory.CreateRunspace(iss))
using (var ps = PowerShell.Create())
{
ps.Runspace = rs;
rs.Open();
ps.AddScript(ScriptBlock.ToString());
// add arguments and parameters to scriptblock
ReturnValue = ps.Invoke();
}
// ...
}
Another method on that object is the callback which looks something like this:
C#
public bool CertValidationCallback(
object sender,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors sslPolicyErrors)
{
// prepare the parameters for the scriptblock
Invoke()
// interpret the output from the scriptblock and return a bool
}
This all seems to work rather reliably under testing including a race condition test similar to the one suggested by @lzybkr.
The stakes are fairly high here so I tried to use conservative interpretations of the scriptblock's output to reduce the chance of falsely interpreting a bad certificate as good. I settled on interpreting any of the following as a bad certificate:
An exception thrown in the callback is interpreted as a bad certificate and eventually appears in some deeply-nested exception on the caller's thread. When that exception originates in the scriptblock, it provides valuable diagnostic information. Exceptions thrown by the scriptblock should not be caught in the callback because doing so hides valuable information from the user about what's happening in their scriptblock.
At first I found it extremely difficult to write a acceptable certificate validation scriptblocks. Even the rather straightforward sha1-intermediate check was difficult because I didn't have access to the objects passed to the callback in a debugger or interactive session. I ended up writing a function that copies those objects to the degree I was able to and brings them into the caller's context. This capability seems rather important to write good certificate validation scriptblocks.
I'm fairly certain that meaningful certificate validation will rely on calling some sort of helper functions inside the scriptblock. Because the callback is in a different context from the caller, such helper functions have to be deliberately made available in the scriptblock. Injecting functions into the scriptblock by way of adding SessionStateFunctionEntrys to InitialSessionState.Command in the callback seems to work fine for this purpose.
BTW, thank you @lzybkr for the advice you gave in the #4970 review -- it probably saved me a few days of research.
@alx9r Thanks! I will have to look at what you provided this weekend. Have you tested any of this with the HttpClientHandler.ServerCertificateCustomValidationCallback? It appears to be a slightly different beast in CoreFX than what was on the ServicePointManager in full CLR.
@markekraus I did do a bit of testing of HttpClientHandler.ServerCertificateCustomValidationCallback using one of the v6.0.0-beta.8. I didn't notice any obvious differences in behavior.
It appears to be a slightly different beast in CoreFX than what was on the ServicePointManager in full CLR.
I haven't worked with ServicePointManager so I can't really speak to that. The work I talked about in my last post is all with full 4.7 WebRequestHandler.ServerCertificateValidationCallback (see here) which looks rather similar to core 2.0 HttpClientHandler.ServerCertificateCustomValidationCallback. I would have used HttpClientHandler instead of WebRequestHandler, but it didn't expose ServerCertificateCustomValidationCallback until 4.7.1 which was only released a couple of days ago.
Just to let you know, the callback currently (as of .Net Core 2.0) does not work on all platforms (e.g. OSX). The two modes that you are referring to (default validation and skip validation) are the only consistent modes currently supported by .Net Core. There is hope for this to get fixed with the next version of .Net Core.
I am trying to get this documented: https://github.com/dotnet/corefx/issues/24774
It might be ok to handle the PlatformNotSupported exception and inform the user that the parameter(s) are not supported on your platform.
@atanasa Yup, I'm aware. Though in some instances it seems to work and in others it doesn't. It was passing our macOS tests for awhile and then stopped. I finally got a loaner mac to troubleshoot, but I just haven't had time (real job getting in the way 鈽癸笍 ). We have a similar issues with -Certificate and certificate based authentication with the web cmdlets.