Powershell: Productivity suggestions for Web Cmdlets

[mtman]

• Add a parameter that does the following:

  1. Accepts Hashtable, converts it into JSON (with -Depth 99 if the cmdlet is used), sets it as Body
  2. Sets ContentType to application/json

  The intend is to make sending JSON as easy as sending a form, given it's the more often used format for APIs.
  Example name: JsonBody

Inspired by Curl and especially intended for REPL:

• Add aliases for all or at least the commonly used parameters. Aliases, unlike partial names, provide certainty and have a short display form at the same time. Some might match with Curl if applicable.
  Example: b and/or d for Body, ua for UserAgent

• If Uri lacks the protocol, attach http:// automatically.

[markekraus]

1 quick comment:

with -Depth 99 if the cmdlet is used

This would not be a good idea. There are a a few common types which have recursive parent-child properties that would result in some very unexpected behavior if this was the default.

[iSazonov]

We have ConvertTo-Json.

Aliases in script text is bad. Aliases in interactive session don't make sense due to the IntelliSense.

[SteveL-MSFT]

Sending JSON today should be easy and I think we should keep JSON to the JSON cmdlets and not have it married to the web cmdlets.

[BrucePay]

@iSazonov

Aliases in script text is bad. Aliases in interactive session don't make sense due to the IntelliSense

He does say that the aliases are for interactive use. Practically, on the command line aliases are simple, fast and let you get more on the line. That's why we have them.

Also, Having a parameter that serializes if it's a hashtable or validates then passes the value through as a string seems like a nice accelerator.

@markekraus I would interpret "depth 99" to be something like "for hash tables, serialize to every level of the hashtable with a check for loops". The hashtable values themselves wouldn't be serialized beyond the default 1 level. @mtman Is this along the lines of what you are thinking of?

[dragonwolf83]

A [json] type accelerator would be better than to do this directly in the web cmdlets just like [xml].

[BrucePay]

@dragonwolf83 The thing with this approach is that, as opposed to XML, there isn't really a json object. So doing [json] @{a=1; b=2} would return a formatted string rather than a json object. (I supposed we could create a Json class that wrapped the string or something.) Anyway, I do agree with you - it feels kind of right, especially in being symmetric with XML.

[iSazonov]

@BrucePay

He does say that the aliases are for interactive use. Practically, on the command line aliases are simple, fast and let you get more on the line. That's why we have them.

We have a number of discussions about alias in a few years. A few in the last year. Many of them started with the request that aliases are good for interactive use and ended up being a huge headache. The last idea was to remove them all and allow users to decide themselves what aliases they want, perhaps with a separate module or modules.
The peak of discussions is #5870.

[SteveL-MSFT]

@iSazonov I believe @BrucePay was referring to parameter aliases, not cmdlet aliases.

[iSazonov]

😄 This is the next stage of the headache.

[markekraus]

@BrucePay I can't speak to @mtman's intentions, but I can explain my thinking there.

IMO, the JSON serialization and deserialzation PowerShell should be the same across all uses. In this case, we would serialize objects the same way we do with ConvertTo-Json. If we used -JsonBody as the parameter for this, then these 2 should be functionally equivalent:

$Result1 = Invoke-WebRequest $uri -Method POST -Body ($object | ConvertTo-Json) -ContentType 'application/json'
$Result2 = Invoke-WebRequest $uri -Method POST -JsonBody $object

In other words, we shouldn't use a different implementation for JSON serialization as someone would reasonably expect those to be functionally equivalent. This gets tricky. Sometimes you want to preserve the looping nature of objects and other times you do not. I don't believe there is currently any way to disable looping in the JSON Cmdlets. The only mechanism currently available is to limit the depth. But even if we did have the mechanism to prevent looping, we would be making a judgement call on behalf of the user on this parameter, that's probably not a good idea.

I understood the request as making the behavior functionally equivalent to this:

$Result3 = Invoke-WebRequest $uri -Method POST -Body ($object | ConvertTo-Json -Depth 99) -ContentType 'application/json'

This is problematic for the following: $Object = Get-Acl c:\. We also don't know what depth to stop at if the user supplies a sufficiently deep object (even without looping).

Parallel to this problem is the assumption of object types for JSON submission. A hashtable translates to a JSON Object. However, single value JSON literals (bools, ints, floats, strings) and JSON Arrays are also valid JSON to submit to endpoints.

Ultimately, as much as I would like to make it easy to submit a PowerShell Object as JSON without using ConvertTo-Json, I think it's probably best to keep to "do one thing" and let the JSON Cmdlets handle the the conversion/serialization separately from the Web Cmdlets. I would hate to keep the web cmdlets coupled with changes in the JSON cmdlets and I would not think an opinionated implementation would result in a good UX.

On the possible use of a [json] type accelerator, I think this is backwards from [xml]. The XML type accelerator is working on a string and transforming it into an object. We could look at something similar for JSON, but it would would like xml in that it deserializes a JSON string to an object. What the cmdlet here needs is to serialize an object into JSON string. It's maybe something to consider, but the serialization and deserialization are not always straight forward so whatever we implemented would have an opinionated default behavior.

On parameter aliases, I'm good with making the common ones simpler and easier to use on interactive sessions... especially since we have so many and some of them are long.

On the automatic scheme, I'm pretty sure that already works as requested:

$Result4 = Invoke-WebRequest 'google.com'
$Result4.BaseResponse.RequestMessage.RequestUri.Scheme

should be http.

[markekraus]

I would hate to keep the web cmdlets coupled with changes in the JSON cmdlets

I'll make one caveat to my above statement. I am OK with adding functionality for deserializing web responses from JSON to objects in Invoke-RestMethod. It's already married there and where the cmdlet derives the majority of its value. For example, extending IRM to include the AsHashtable feature or working with object models (should that ever be added).

[mtman]

Aliases in script text is bad. Aliases in interactive session don't make sense due to the IntelliSense.

Aliases are very different than Intellisense because they are short and don't require iteration. In case of scripts, the common problem is positional arguments. If those script writers adopt aliases instead, the tooling can replace it with full names, which cannot be done with positional args (they are not unique across all parameter sets).

Pros:

• Writing and reading is short and confident
• CLI history is clear and easy to navigate
• Helps replace positional args with full parameter names in scripts
• Easy adoption when coming from Curl

Cons:
?

@BrucePay, @markekraus I didn't consider looping. I think it's a very rare use case for RESTful queries. However, there is a coherent way to improve JSON serialization in general. ConvertTo-Json can have a -CircularDetection accepting Depth (default), Enable, Disable. Web Cmdlets should always use Enable. Once it detects a circular reference, it throws an exception alongside a hint to use ConvertTo-Json for more options. I think JSON.NET works similarly.

If a breaking change for ConvertTo-Json is not an issue, the default can be Enable. And since it already fails silently once the default depth (2 or 3) is reached, I wouldn't mind to break the behavior at all.

Good point on auto scheme and primitive types. -JsonBody should accept object and act just like ConvertTo-Json (considering the above).

The key issue is that in case of REST APIs, JSON seems to be the default body format instead of forms. Take the default ASP.NET Core project as an example. It automatically converts a JSON body, but fails for forms.

[markekraus]

@mtman

I think it's a very rare use case for RESTful queries.

You would be wrong ;). The ACL example and DACLs require the child parent relationship in 2 of the APIs I work with.

Web Cmdlets should always use Enable

Again, that would be an opinionated implementation which would lead to a bad UX.

The key issue is that in case of REST APIs, JSON seems to be the default body format instead of forms.

There are 3 I use on a regular basis: JSON, XML, and multipart/form-data. XML is increasinly rare, but multipart/form-data and JSON are about equal. More often than not, most REST APIs don't even require a body. Only U operations in CRUD tend to need a body and even then a good portion of endpoints I work with still use query fields instead of body payloads.

Take the default ASP.NET Core project as an example. It automatically converts a JSON body, but fails for forms.

I'm not sure that is accurate. The WebListener in this project is testament to that.

Again, I understand that adding the ability to directly serialize an object as JSON in the web cmdlets seems useful. I have considered it myself. However, whatever implementation we use would be opinionated unless we add more parameters to accommodate various JSON serialization needs or risk endless support issues and questions about why our opinionated implementation doesn't serialize the way the user expects it to. At that point, it's best to stick to _do one thing_ and let the JSON cmdlets handle the serialization and the Web Cmdlets handle web requests.

On improvements to JSON serialization, there are quite a few I would like to make as well (I think i have open issues for several). @mtman Can you please open a separate issue for your suggestion of loop detection.

Honestly, once the Experimental Features feature is added, I want to possibly redo the entire JSON implementation. At this point in my own code, I'm making raw .NET Calls to Newtonsoft.Json because the current cmdlets are too simplistic and opinionated. When you are working with 40+ APIs, you start to see the entire spectrum of JSON and that requires greater flexibility.

[mtman]

@markekraus

You would be wrong ;). The ACL example and DACLs require the child parent relationship in 2 of the APIs I work with. ... When you are working with 40+ APIs, you start to see the entire spectrum of JSON and that requires greater flexibility.

If 2 out of 40+ APIs you work with require that, I think that counts as quite rare ;)

Again, that would be an opinionated implementation which would lead to a bad UX. ... However, whatever implementation we use would be opinionated unless we add more parameters to accommodate various JSON serialization needs or risk endless support issues and questions about why our opinionated implementation doesn't serialize the way the user expects it to

That's just the general way to achieve any simplification. As long as there is a simple and logical way to opt out, I don't see how that's bad UX. Even better, we can see a reception of the same decision for Newtonsoft.Json. You can find multiple Stackoverflow questions on how to opt out, but I haven't seen any rant against the choice there. Here, the solution would be contained directly within the error message.

Only U operations in CRUD tend to need a body and even then a good portion of endpoints I work with still use query fields instead of body payloads.

I think the general way is to accept body once query fields become overwhelming, e.g. Elasticsearch lets you make basic queries via query fields and advanced ones via the JSON body (both GET).

I'm not sure that is accurate. The WebListener in this project is testament to that.

1. Run dotnet new WebApi
2. Add public class FooBody { public string Foo { get; set; } }
3. Replace Post:

[HttpPost]
public void Post([FromBody] FooBody value) => Console.WriteLine(value.Foo);

1. dotnet run
2. irm http://localhost:5000/api/values -method Post -body @{Foo = 'bar'}
415: Unsupported Media Type

Auto scheme:

So the issue occurs only when used with a port:
irm duckduckgo.com:80:
Only 'http' and 'https' schemes are allowed.

On improvements to JSON serialization, there are quite a few I would like to make as well (I think i have open issues for several). @mtman Can you please open a separate issue for your suggestion of loop detection.

Sure: https://github.com/PowerShell/PowerShell/issues/6638

[markekraus]

If 2 out of 40+ APIs you work with require that, I think that counts as quite rare

Sure, if it wasn't for the fact that those 2 APIs have a higher usage than many others, with some getting less than 10 calls a month. *shrugs

That's just the general way to achieve any simplification. As long as there is a simple and logical way to opt out, I don't see how that's bad UX.

How deep do we set it? why? how do we make the user aware of that? Do we block looping or not? How do we communicate that to the user?

It's just that JSON is not as simple as many make it out to be. As I said, when you start to use a large number of APIs you come into issues with just how limiting the current implementation is. Adding a even more limited and opinionated implementation directly on a cmdlet that is focused on processing web content, not web submissions, and coupling the cmdlet further with another API is bad design with a bad UX.

If the Object to JSON serialization was clear and singular, I would not be opposed to adding it as another body type. But since we would need to add multiple flags to deal with the serialization to meet the 80% use cases, it makes less sense to included it.

So the issue occurs only when used with a port:

I believe that is by design. When you specify a port, you must specify a scheme because you are being explicit which indicates you do not with to use defaults. you specify "google.com:443" and the recommendation would be to prepend that with http. What if I need https over 80? Maybe ftp? Not sure it makes sense to change it... and if we did.. I would probably prefer HTTPS as the default.

[yecril71pl]

Please close this issue.

[msftbot[bot]]

This issue has been marked as answered and has not had any activity for 1 day. It has been closed for housekeeping purposes.