Powershell: Invoke-WebRequest should have a curl style -Verbose switch to display full server response

[1RedOne]

Today in cURL, we can specify the v/--verbose flag to show the CONNECT request to the proxy, as well as the SSL/TLS handshake process (including certificate), like

* Rebuilt URL to: https://www.example.com/
*   Trying 192.168.2.1...
* Connected to my-proxy.local (192.168.2.1) port 8080 (#0)
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.47.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: www.example.org (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=California,L=Los Angeles,O=Internet Corporation for Assigned Names and Numbers,OU=Technology,CN=www.example.org
*        start date: Tue, 03 Nov 2015 00:00:00 GMT
*        expire date: Wed, 28 Nov 2018 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.47.0
> Accept: */*
>

Today in PowerShell, there's not a great way to get all of this information. We can get some of it though.

Actual Behavior

The Microsoft.PowerShell.Commands.HtmlWebResponseObject object maintains some of the information for us, within the first 25 lines of .RawContent and also within the .BaseResponse properties.

$response.BaseResponse 

IsMutuallyAuthenticated : False
Cookies                 : {__cfduid=d84018de2d621df9d53eb52d97cd33a651511881763}
Headers                 : {Transfer-Encoding, Connection, Vary, Access-Control-Allow-Credentials...}
SupportsHeaders         : True
ContentLength           : -1
ContentEncoding         : 
ContentType             : application/json; charset=utf-8
CharacterSet            : utf-8
Server                  : cloudflare-nginx
LastModified            : 11/28/2017 10:17:27 AM
StatusCode              : OK
StatusDescription       : OK
ProtocolVersion         : 1.1
ResponseUri             : https://jsonplaceholder.typicode.com/posts
Method                  : GET
IsFromCache             : False

We can also get some good info in the first 25 lines or so of the RawContent property, as seen here. RawContent is, well, raw, so I apply a split on new-lines then use array indexing as depicted by [0..20] to select the first 21 lines.

$response.RawContent.Split("`n")[0..20]
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Pragma: no-cache
X-Content-Type-Options: nosniff
CF-Cache-Status: HIT
CF-RAY: 3c4e3f804f9d82f7-ATL
Cache-Control: public, max-age=14400
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Nov 2017 15:09:23 GMT
Expires: Tue, 28 Nov 2017 19:09:23 GMT
ETag: W/"6b80-Ybsq/K6GwwqrYkAsFxqDXGC7DoM"
Set-Cookie: __cfduid=d84018de2d621df9d53eb52d97cd33a651511881763; expires=Wed, 28-Nov-18 15:09:23 GMT; path=/; domain=.typicode.com; HttpOnly
Server: cloudflare-nginx
Via: 1.1 vegur
X-Powered-By: Express

Desired Behavior

If would be super nice to get cURL style connection response info back from within PowerShell. For instance, the -Verbose switch could service this information, or instead provide another switch for Invoke-WebRequest.

Invoke-WebRequest -uri https://jsonplaceholder.typicode.com/posts -ShowConnectionInfo


VERBOSE: Rebuilt URL to: https://www.example.com/
VERBOSE:   Trying 192.168.2.1...
VERBOSE: Connected to my-proxy.local (192.168.2.1) port 8080 (#0)
VERBOSE: Establish HTTP proxy tunnel to www.example.com:443
VERBOSE: CONNECT www.example.com:443 HTTP/1.1
VERBOSE: Host: www.example.com:443
VERBOSE: User-Agent: curl/7.47.0
VERBOSE: Proxy-Connection: Keep-Alive
VERBOSE: 
VERBOSE: HTTP/1.1 200 Connection established
VERBOSE: 
VERBOSE: Proxy replied OK to CONNECT request
VERBOSE: found 148 certificates in /etc/ssl/certs/ca-certificates.crt
VERBOSE: found 597 certificates in /etc/ssl/certs
VERBOSE: offering http/1.1

[markekraus]

I don't believe this is possible with HttpClient (the underlying .NET API called by the web cmdlets). BasicHtmlWebResponseObject.RawContent (there is no HtmlWebResponseObject in PowerShell Core) is created after the response is already received. We currently do not have access to this in a "live" fashion. I think even if we switched to doing an async request, we still wouldn't have any kind of live update for this as the HttpResponseHeaders and HttpContentHeaders objects are not populated until he response has completed, and that is how we are currently building RawContent.

Baring major changes to the underlying .NET API this won't be feasible and anything we did here would be redundant to the data provided on BasicHtmlWebResponseObject.

[charlieschmidt]

I don't think it needs to be live data, but -Verbose on Invoke-WebRequest and RestMethod is nearly useless compared to what I think most people /want/ it to do - which is something like above. For example, the verbose message on POST is always wrong, it says its sending with a -1-byte payload regardless of -Body contents.

Verbose messages that look like http traces that included the dns lookup results, the headers being sent, maybe the body (or part of the body) would be super helpful in my opinion.

The response being part of -Verbose... ehh, not as important to me personally, given we'll have it on the pipeline there anyway. Could still throw the headers and response code out on Verbose though, or with some additional param like suggested above. I don't know how verbose is too verbose, but what the two cmdlets do now isn't sufficient in my opinion

[markekraus]

@charlieschmidt I agree that the verbose output could use some work. Some of what you suggest may not be possible. Also, I though we fixed the incorrect -1 payload in OS Core?

invoke-restmethod -Verbose -Uri https://httpbin.org/post -Method POST -Body 'test'

result

VERBOSE: POST https://httpbin.org/post with 4-byte payload
VERBOSE: received 432-byte response of content type application/json
VERBOSE: Content encoding: iso-8859-1


args    :
data    :
files   :
form    : @{test=}
headers : @{Connection=close; Content-Length=4; Content-Type=application/x-www-form-urlencoded; Host=httpbin.org;
          User-Agent=Mozilla/5.0 (Windows NT 10.0; Microsoft Windows 10.0.16299; en-US) PowerShell/6.0.0}
json    :
origin  : 65.36.55.130
url     : https://httpbin.org/post

[charlieschmidt]

Yeah my mistake, it was fixed in Core; i switch between Core at home and v5 at work still so it's confusing for me, apologies.

to me, the perfect result of -Verbose (or hell, -VerboseHeaders ?) would be something like:

VERBOSE: POST https://httpbin.org/post with 4-byte payload
VERBOSE: Connecting 52.243.164.125 on port 80
VERBOSE: Request Headers:
VERBOSE: Content-Length: 4
...
VERBOSE: Received response status code 200
VERBOSE: Received 432-byte response of content type application/json
VERBOSE: Response Headers:
VERBOSE: Content-Length: 453
...


args    :
data    :
files   :
form    : @{test=}
headers : @{Connection=close; Content-Length=4; Content-Type=application/x-www-form-urlencoded; Host=httpbin.org;
          User-Agent=Mozilla/5.0 (Windows NT 10.0; Microsoft Windows 10.0.16299; en-US) PowerShell/6.0.0}
json    :
origin  : 65.36.55.130
url     : https://httpbin.org/post

Dunno. I suppose that is a lot of info. But from experience it's rare to want a Verbose web request and not want to see at least some of the headers

[FranklinYu]

Some one should point me to this issue! 😄

Related Stack Overflow question: https://stackoverflow.com/questions/47533798/how-to-show-tls-handshake-information-and-connect-request-in-invoke-webrequest

I did some research on this repository; I found

https://github.com/PowerShell/PowerShell/blob/13cf8af66734b4119643748f36ed92260ceae7da/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/CoreCLR/InvokeWebRequestCommand.CoreClr.cs#L16

So I searched for WebRequestPSCmdlet and found that HTTP is processed in this method:

https://github.com/PowerShell/PowerShell/blob/13cf8af66734b4119643748f36ed92260ceae7da/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs#L1428

Which is using System.Net.Http.HttpClient. I'm still reading their documentation to see whether this class provides such detailed information; if they don't then I guess we can do nothing here.

[marioja]

Is there an update on this issue? This functionality is still very desirable.

[he852100]

Details are required. This information is required for debugging. When the error happened, I don't know what the problem is.

[Zorono]

any update for this ???