Powershell: Add Parameter to ConvertTo-Json to ignore unsupported properties

[mike-the-automator]

I'd like to propose an additional parameter for the ConvertTo-Json cmdlet that specifies the caller would like the cmdlet to ignore properties of the InputObject that cannot be converted to JSON successfully. Currently, the behavior of the cmdlet is to throw an exception with a message similar to the following

"ConvertTo-Json : The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or "deserialization of a dictionary. Keys must be strings.

My proposal is that if the -SkipIncompatibleProperties (parm name feedback appreciated :-) ) parameter is specified, then the cmdlet would silently skip over these properties and not throw any exceptions.

I've been frustrated by this error several times in my scripting in situations where I don't really care about the particular property that triggered the error; I would just like a quick and easy serialized representation of the object. In the most recent instance, I needed a way to record exception info in a VARCHAR column in SQL Server. The property of the System.Exception that triggered this error was the "Data" field which I didn't really care about, but it prevented me from easily using the cmdlet to serialize the exception object.

As an aside, it looks like there is a proposal by @KirkMunro to move the JSON cmdlets to their own module. I understand it may make sense to wait on this feature until the module split occurs.

[bergmeister]

@mike-the-automator Does -ErrorAction SilentlyContinue solve your specific case(s)? If not then it might be the better solution to rather tweak the error behaviour instead of adding a new parameter.

[mike-the-automator]

Thanks for your response. I tested -ErrorAction SilentlyContinue and that did not seem to solve my problem.

Transcript:
```
PowerShell v6.0.0-rc.2
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/pscore6-docs
Type 'help' to get help.

PS C:Program FilesPowerShell6RC2\6.0.0-rc.2> 1/0
Attempted to divide by zero.
At line:1 char:1

• 1/0
• ~~~
• CategoryInfo : NotSpecified: (:) [], RuntimeException
• FullyQualifiedErrorId : RuntimeException

PS C:Program FilesPowerShell6RC2\6.0.0-rc.2> $j = ConvertTo-Json $Error[0] -ErrorAction SilentlyContinue
ConvertTo-Json : The type 'System.Collections.ListDictionaryInternal' is not supported for serialization or deserialization of a dictionary. Keys must be strings.
At line:1 char:6

• $j = ConvertTo-Json $Error[0] -ErrorAction SilentlyContinue
• ~~~~~~~~~~~~~~
• CategoryInfo : InvalidOperation: (System.Collecti...tionaryInternal:ListDictionaryInternal) [ConvertTo-Json], InvalidOperationException
• FullyQualifiedErrorId : NonStringKeyInDictionary,Microsoft.PowerShell.Commands.ConvertToJsonCommand

PS C:Program FilesPowerShell6RC2\6.0.0-rc.2> $j -eq $null
True

``` In the scenario I envision, when the new parameter is supplied there would be no error thrown and the value of the variable$j` would be a JSON representation of the exception objects, ignoring any fields that would have triggered the InvalidOperation exception.

[mike-the-automator]

@bergmeister I have mulled over the suggestion to just change the behavior for -ErrorAction SilentlyContinue and my impression is that would be a counter intuitive way to achieve my goal. It would be a breaking change for users because a scenario that used to generate an Exception would no longer do so.

Also, silently discarding data without being explicitly told to do so feels _wrong_. It almost feel more like an occasion to use -Force as more of a "No, really this is what I want to do" kind of parameter.

Another means to get the result I want would be to just create a cmdlet to strip non-JSON compliant fields of an object that would result in this error and put it in front of ConvertTo-Json in the pipeline, e.g.

$j = $Error[0] | Remove-JsonIncompatibleProperties | ConvertTo-Json

I could always create a convenience wrapper-function to eliminate the need for a long pipeline (I suck at naming things). Given that perspective I can understand if the consensus is that the complexity/utility ratio doesn't pan out for this feature request, but I'd like to know what others think.

[FireInWinter]

I think it is a separate issue that ErrorAction is ignored and that it still raises an exception, but I think the correct functionality of ErrorAction with SilentlyContinue should be to return no data, not to strip out some data.

I think either Force or SkipIncompatibleProperties would be a better solution as you are explicitly telling it to get rid of data.

[mike-the-automator]

@FireInWinter Created a separate issue as requested. I think we are in agreement that the behavior requested in this issue should be implemented as a new parameter for the reasons already stated.

I'm interested in contributing and would be happy to work on a PR for either issue, but I'd rather not invest time in this feature request if it has a low probability of being merged. I'm not asking for a guarantee that my PR will be accepted, but it would be nice to know if I'm barking up the wrong tree here.

[bergmeister]

Can a maintainer please comment on the design/issue. @iSazonov @lzybkr ?

[SteveL-MSFT]

My preference would be to have a -SkipUnsupportedTypes switch. -Force is too overloaded to me. cc @joeyaiello @HemantMahawar for suggestions on the name of the switch.

[mklement0]

Generally, the use case you're describing is better handled using serialization based on CLIXML rather than JSON.

What makes this more cumbersome is the fact that you must export to a (temporary) _file_ first, using Export-CliXml, and also read from a file, using Import-CliXml; however, a new pair of ConvertTo-CliXml / ConvertFrom-CliXml cmdlets that directly output / read the XML as a strings are being worked on - see #3898

[SteveL-MSFT]

@mklement0 agree that if the use case is to serialize and deserialize later without care of the serialization format, CliXml is a better solution. However, I can see cases where you want specifically json (more human readable than xml...)

[mklement0]

@SteveL-MSFT:

I merely restricted my comment to pointing out the CLIXML alternative, because I don't know what the right answer to the specific problem at hand is, but here are a few thoughts:

A simpler way to reproduce the problem, using a hashtable with a non-string key:

@{ 1 = 'one' }  | ConvertTo-Json

The error message suggests that an explicit decision was made not to support non-string keys and to _fail_ in that event.

The alternatives - to be activated via new switch(es) - are:

• simply apply .ToString() to non-string keys and serialize nonetheless

• as suggested, _skip_ unserializable properties.

Given that JSON is not suitable for general-purpose serialization, however, I wonder if this is really necessary.
Generally, you will know what properties should be serialized ahead of time, and can filter out unsupported properties then, in the simplest case with Select-Object -Property ...

[iSazonov]

I wonder if this is really necessary.

I wonder too.
JSON is well standardized. I would have looked at what this standard says.

[markekraus]

I wonder if this is really necessary.

Let's not let convenience become irrelevant in this project.

I have use cases for this myself that I have had to work around. I have object A that has nested property B that cannot be serialized, but I need to upload this to a generalized logging endpoint. that expects a JSON object. I don't care about property B even in the context of PowerShell or .NET. I have to create Object C that has everything but property B and then I can serialize it. Or I have to use a half dozen other work arounds in this scenario including creatining my own JSON serialize or a proxy class.

[SteveL-MSFT]

This reminds me that in my HttpListener module, I had to work around JSON serialization limitations by manually stripping types that were not supported.

[B-Art]

I would suggest a function which converts all none-string values to string values before piping it to convertto-json.
You can put a select statement in between for your final json output.

@{ 1 = 'one' }.getenumerator() | select key,value | ConvertTo-Json
Works....

[mklement0]

@B-Art:

This issue is about _ignoring_ values that can't be converted to JSON.

By contrast, you're looking to for an opt-in method for _including_ such values by converting dictionary keys to strings, so I suggest you create a _new_ issue with your proposal.

As an aside, re your example: It isn't the select call that makes the command succeed, it is using .GetEnumerator(), because the latter sends a [System.Collections.DictionaryEntry] instance (rather than the whole hashtable) through the pipeline, which is itself not a dictionary, so the to-JSON conversion succeeds - however, that is _not_ the same as what @{ '1' = 'one' } | ConvertTo-Json would give you, i.e., the hashtable with stringified keys.