Powershell: ConvertFrom-Json does not handles same keys with different case (but JSON is case sensitive)

[NotCricri]

Steps to reproduce

$data = @"
{
"firstName": "Bill",
"Firstname": "Bob",
"lastName": "Gates"
}
"@

$json = $data | ConvertFrom-Json

Expected behavior

Convert JSON data, containing 3 pairs of key/value, with same key name "firstName" but with a different case:

• firstName: with the letter "f" in lowercase and the letter "N" in upper case
• Firstname: with the letter "F" in uppercase and the letter "n" in lowercase

Actual behavior

Exception thrown:

ConvertFrom-Json : Cannot convert the JSON string because a dictionary that was converted from the string contains the duplicated keys 'firstName' and 'Firstname'.
At line:8 char:17
$json = $data | ConvertFrom-Json

 CategoryInfo          : InvalidOperation: (:) [ConvertFrom-Json], InvalidOperationException
 FullyQualifiedErrorId : DuplicateKeysInJsonString,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

As PowerShell is case insensitive, it considers the second key "Firstname", with a different case, as a duplicate key.
JSON allows to have same key name but with different case

Environment data

Name Value
---- -----
PSVersion 5.1.14393.1066
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.14393.1066
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

[NotCricri]

https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/11088237-duplicate-keys-error-with-convertfrom-json

[NotCricri]

Workaround found using .Net method DeserializeObject (JavaScriptSerializer):

$json = (New-Object -TypeName System.Web.Script.Serialization.JavaScriptSerializer -Property @{MaxJsonLength=67108864}).DeserializeObject($data)

(source: http://wahlnetwork.com/2016/03/15/deserializing-large-json-payloads-powershell-objects/)

[SteveL-MSFT]

You must use -AsHashTable to support this usage. The error message has been updated to indicate this if you try this with latest PSCore6.

[StingyJack]

Sorry but this needs to be reopened as its not fixed. Using -AsHashTable does address the error, but it doesn't make the output usable either.

[vexx32]

@StingyJack can you expand on what you mean?

[StingyJack]

Json is case sensitive (as is .NET). The current problem I have is an ElasticSearch index mapping (GET /indexname/_mappings ), Its possible to have mappings with a "DocumentURL" and a "documentUrl" field existing at the same level and it be a legal json payload.

If I use Invoke-WebRequest to get the mappings, and call $response.Content | ConvertFrom-Json) I get the duplicate error mentioned in the OP. if I add the -AsHashTable parameter I get...

{
  "_doc": {
    "properties": {
      "documentUrl": "System.Collections.Hashtable",
      "ExceptionId": "System.Collections.Hashtable",
      "ExceptionType": "System.Collections.Hashtable",
      "FullText": "System.Collections.Hashtable",
      "DocumentURL": "System.Collections.Hashtable",
     etc.

... which is not very useful when I'm expecting to be able to navigate to the values easily and without a bunch of ["key"]-ing.

[vexx32]

You need to set the -Depth parameter to enable it to convert items beyond the default depth limit of 2. :)

[daveneeley]

These design choices seem to be add at odds with each other. Adding -AsHashTable seems to have been done to preserve backwards compatibility with PS5. BUT, -depth with a default limit of 2 was also added, which breaks backwards compatibility anyway.

[SteveL-MSFT]

ConvertFrom-Json has a default depth of 1024 which should be sufficient for most cases. ConvertTo-Json has a default depth of 2 due to some .NET types having recursive properties.

[vexx32]

Yeah, that and some of the CIM / WMI objects can get ridiculously huge even with a depth of 2, so making it any bigger by default would be veeery tricky.

Would be nice if there was a simple way to detect recursive props but it can be rather difficult at times

[jbpAtEy]

I'm sorry but @StingyJack is correct. The solution here is a work around at best. As the JSON RFC, https://tools.ietf.org/html/rfc8259, states, the characters are converted to numerics for evaluation, which implies that A and a are different, and thus case sensitivity is supported in JSON. Given this situation, there is no way we can say that a JSON Object Support without support for case sensitivity is supporting JSON in full. Given the additional information from @StingyJack , this issue is not resolved. a work around is stated. This should still be open and should at some point be fixed.

RFC 8259 - The JavaScript Object Notation (JSON) Data Interchange Format

[arkadios]

Dear Collaborators, bots, MS folks and @SteveL-MSFT,
Forgive me if I am missing something, but this issue is still relevant (today, 2020-08-23, Win10, Powershell v5.1.18362.752).
This has not been resolved. And part from whether this can be resolved or not, this issue should be open, not closed.

I have searched for other ConvertFrom-Json posts, but could not find any other posts describing this "anomaly" .

'{ "Id":"value1", "ID":"value2" }' | ConvertFrom-Json
results in error:
ConvertFrom-Json : Cannot convert the JSON string because a dictionary that was converted from the string contains the
duplicated keys 'Id' and 'ID'.
while expected behavior should be different.

a function that claims to be able to convert from json should be able to do that with a well designed json object.

please re-open

tnx @NotCricri for the workaround

[MaximoTrinidad]

Keep in mind! Window PowerShell 5.x is complete, and only security updates may be applied.
All development efforts are now on PowerShell 7 moving forward.

:)

[arkadios]

Tnx @MaximoTrinidad I will keep that in mind :)
This a nice moment (for me) to start using PS7... curious.