Powershell: Select-Object modifies input objects when using both -ExpandProperty and -Property

[bstrautin]

Select-Object should never modify its input objects, but it does, when using both the -ExpandProperty and -Property parameters.

This behavior is documented, but wrong.

Steps to reproduce

$NestedObjects =
  [pscustomobject]@{a=1; b=[pscustomobject]@{c=2}},
  [pscustomobject]@{a=3; b=[pscustomobject]@{c=4}}

$NestedObjects | Out-Host
$NestedObjects | Select-Object a -ExpandProperty b | Out-Host

$NestedObjects | Out-Host
$NestedObjects | Select-Object a -ExpandProperty b | Out-Host

Expected behavior

Re-running the same Select-Object statement should produce the same output, and should not modify its input objects, so the example code should produce the following output:

a b
- -
1 @{c=2}
3 @{c=4}



c a
- -
2 1
4 3


a b
- -
1 @{c=2}
3 @{c=4}



c a
- -
2 1
4 3

Actual behavior

Instead of creating new objects with the appropriate fields, Select-Object modifies the objects specified by -ExpandProperty, which causes an error the second time the same code runs:

a b
- -
1 @{c=2}
3 @{c=4}



c a
- -
2 1
4 3



a b
- -
1 @{c=2; a=1}
3 @{c=4; a=3}


Select-Object : The property cannot be processed because the property "a" already exists.
At line:9 char:18
+ $NestedObjects | Select-Object a -ExpandProperty b | Out-Host
+                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (@{a=1; b=}:PSObject) [Select-Object], PSArgumentException
+ FullyQualifiedErrorId : AlreadyExistingUserSpecifiedPropertyExpand,Microsoft.PowerShell.Commands.SelectObjectCommand


Select-Object : The property cannot be processed because the property "a" already exists.
At line:9 char:18
+ $NestedObjects | Select-Object a -ExpandProperty b | Out-Host
+                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (@{a=3; b=}:PSObject) [Select-Object], PSArgumentException
+ FullyQualifiedErrorId : AlreadyExistingUserSpecifiedPropertyExpand,Microsoft.PowerShell.Commands.SelectObjectCommand

c a
- -
2 1
4 3

Environment data

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.0.1
PSEdition                      Core
GitCommitId                    v6.0.1
OS                             Microsoft Windows 10.0.17134
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[mklement0]

Strictly speaking, it's not the _input objects_ that are modified, but the _value of the property_ that -ExpandProperty targets.

That property value can be an instance of a _value type_ or an instance of a _reference type_.

Either way, the properties specified in -Property are added as NoteProperty members to that value.

In the case of a _value-type_ instance, these NoteProperty members are by definition added to a _copy_ of that instance.

In the case of a _reference-type_ instance, these NoteProperty members are added to the _object pointed to by the reference_, and any pre-existing references to that object will see the changes - in your case, that is the $NestedObject array.

While that behavior may be surprising, it makes sense, given that there's no _guaranteed_ way to _clone_ reference-type instances - and implicit, automatic cloning seems to be what you're looking for (leaving aside the need to distinguish between shallow and deep cloning).

While _some_ reference types support cloning - and [pscustomobject] happens to be one via its .Copy() method - not all do, so the more consistent behavior is not to even attempt it.

[iSazonov]

It would be grandiose breaking change.
We can not even add a new switch parameter since @mklement0 pointed out that there is no generic way to clone an reference object.

[iSazonov]

@bstrautin

This behavior is documented, but wrong.

You could open a issue in PowerShell-Docs repo.

[bstrautin]

Cloning is not necessary; in the same way that Select-Object -Property * creates new pscustomobjects with the necessary properties, a non-surprising version of Select-Object -Property a -ExpandProperty b would create new pscustomobjects with property a and all the properties from b.

To introduce non-modifying behavior as a non-breaking change, it seems like there are basically two options:

1. Add a switch parameter to change the behavior of -ExpandProperty, named something like -NoModifyExpandedProperty or -ExpandPropertyAsCopy
2. Add another parameter, named something like -CopyAndExpandProperty, which would create new pscustomobjects rather than adding note properties. It should behave more like -Property, i.e. accept multiple property names, wildcards, scriptblocks, and hashtables.

[mklement0]

@iSazonov: I think what @bstrautin meant was that _the documentation is correct_ in that it correctly describes the current behavior, but that they consider _the current behavior_ to be wrong.

[mklement0]

OK, so we're talking about a _feature request_, not a problem with the existing behavior.

To be clear: the purpose of -ExpandProperty is to return the specified property's value _as-is_, _optionally decorated_ with NoteProperty members, if -Property is also specified.

So, yes, to get what you want, a new switch is required, but I'm unclear on the semantics:

How would you construct the custom object if the value of the property you're targeting is an instance of a CLR runtime type such as [bool], which has no properties, or [string], which has only a Length property?
Or do you suggest _ignoring_ the new parameter if the value happens to be a value-type instance or a string, given that you'll _implicitly_ get copies in that case anyway?

What if the property value is a collection?

[bstrautin]

It's only a feature request if you don't consider the existing behavior to be a bug, which I do. (A Select-... cmdlet should never modify its input, but Select-Object does, semantics about "decorating" vs "modifying" aside.)

My preference would be to change the current behavior, which would be a breaking change if anyone were actually relying on it (which seems unlikely, given how weird it is.) If that is not palatable, then it becomes a feature request.

[bstrautin]

How would you construct the custom object if the value of the property you're targeting is an instance of a CLR runtime type such as [bool], which has no properties, or [string], which has only a Length property?

-ExpandPropertyAsCopy would always return objects, never scalars.

Explanation by way of examples:

Expanding a scalar property would return an object with that property. (The same as -Property):

[pscustomobject] @{ one = 1; two = 2; three = 3 } | 
  Select-Object -ExpandPropertyAsCopy three

would output

three
-----
    3

Including -Property and expanding a scalar property would return an object with both properties. (Instead of making -Property and -ExpandProperty mutually exclusive per #6161)

[pscustomobject] @{ one = 1; two = 2; three = 3 } | 
  Select-Object -Property one -ExpandPropertyAsCopy three

would output

one three
--- -----
  1     3

What if the property value is a collection?

Basically the same as -ExpandProperty, but creating new objects instead of modifying existing ones.

Including -Property and expanding an object property would return an object with the indicated parent and child properties, duplicating the parent properties for each child.

[pscustomobject]@{
  a=1
  b=[pscustomobject]@{c=2; d=3}, [pscustomobject]@{c=4; d=5}
},
[pscustomobject]@{
  a=6
  b=[pscustomobject]@{c=7; d=8}
} |
  Select-Object -Property a -ExpandPropertyAsCopy b

would output

a c d
- - -
1 2 3
1 4 5
6 7 8

-ExpandPropertyAsCopy with multiple properties:

[pscustomobject]@{
  a=1
  b=[pscustomobject]@{d=2}
  c=[pscustomobject]@{e=3}
},
[pscustomobject]@{
  a=4
  b=[pscustomobject]@{d=5}
  c=[pscustomobject]@{e=6}
} |
  Select-Object -Property a -ExpandPropertyAsCopy b,c

would produce

a d e
- - -
1 2 3
4 5 6

-ExpandPropertyAsCopy with multiple list properties having would behave like LINQ's SelectMany or SQL's cross join:

[pscustomobject]@{
  a=1
  b=[pscustomobject]@{d=2},[pscustomobject]@{d=3}
  c=[pscustomobject]@{e=4},[pscustomobject]@{e=5}
},
[pscustomobject]@{
  a=6
  b=[pscustomobject]@{d=7}
  c=[pscustomobject]@{e=8},[pscustomobject]@{e=9}
} |
  Select-Object -Property a -ExpandPropertyAsCopy b,c

would output

a d e
- - -
1 2 4
1 3 4
1 2 5
1 3 5
6 7 8
6 7 9

IMO, this is how -ExpandProperty should have been designed in the first place, but that ship has sailed.

[mklement0]

It's only a feature request if you don't consider the existing behavior to be a bug, which I do.

Let's summarize:

• The behavior is documented.

• The behavior is useful and _opt-in_: by using -ExpandProperty _in combination with_ -Property, you're asking that NoteProperty members be added to whatever the value of the property passed to -ExpandProperty is, as documented (which is what I called _decorating_).

• Nowhere does it state that "A Select-... cmdlet should never modify its input" - that may be your personal expectation and it certainly makes sense as the _default_ behavior (which is the case), but that doesn't make the behavior in question a bug. Again: by combining -ExpandProperty with -Property you're _asking_ for the output objects to be decorated.
  It's not a commonly used feature, but it works predictably and can be useful.

But even if the community ends up agreeing with your assessment, backward-compatibility concerns alone would require implementing the alternative behavior envisioned by you to be _opt-in_.

[bstrautin]

Partially agree with the summary. Regardless:

Two possible enhancements, which I am willing to develop:

1. Change -ExpandProperty from accepting a single property name to accepting a list of property name, scriptblock, etc, like -Property. This would probably be a breaking change for consistency's sake, because expanding multiple IEnumerable properties would require creating new pscustomobjects rather than decorating the objects from a single property.
2. Add a new parameter, -FlattenProperty or -ExpandPropertyAsCopy , which would behave as described.

Another piece of code that demonstrates -ExpandProperty's buggy behavior, attempting to flatten two fields:

[pscustomobject]@{
  a=1
  b=[pscustomobject]@{b2=2},[pscustomobject]@{b2=3}
  c=[pscustomobject]@{c2=4},[pscustomobject]@{c2=5}
},
[pscustomobject]@{
  a=6
  b=[pscustomobject]@{b2=7}
  c=[pscustomobject]@{c2=8},[pscustomobject]@{c2=9}
} |
  Select-Object -Property a,c -ExpandProperty b |
  Select-Object -Property a,b2 -ExpandProperty c

Expected output:

c2 a b2
-- - --
 4 1  2
 5 1  2
 4 1  3
 5 1  3
 8 6  7
 9 6  7

Actual output, with erroneous values in b2 and errors:

c2 a b2
-- - --
 4 1  2
 5 1  2
Select-Object : The property cannot be processed because the property "a" already exists.
At line:12 char:3
+   Select-Object -Property a,b2 -ExpandProperty c
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (@{b2=3; a=1; c=System.Object[]}:PSObject) [Select-Object], PSArgumentException
+ FullyQualifiedErrorId : AlreadyExistingUserSpecifiedPropertyExpand,Microsoft.PowerShell.Commands.SelectObjectCommand

Select-Object : The property cannot be processed because the property "b2" already exists.
At line:12 char:3
+   Select-Object -Property a,b2 -ExpandProperty c
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (@{b2=3; a=1; c=System.Object[]}:PSObject) [Select-Object], PSArgumentException
+ FullyQualifiedErrorId : AlreadyExistingUserSpecifiedPropertyExpand,Microsoft.PowerShell.Commands.SelectObjectCommand

 4 1  2
Select-Object : The property cannot be processed because the property "a" already exists.
At line:12 char:3
+   Select-Object -Property a,b2 -ExpandProperty c
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (@{b2=3; a=1; c=System.Object[]}:PSObject) [Select-Object], PSArgumentException
+ FullyQualifiedErrorId : AlreadyExistingUserSpecifiedPropertyExpand,Microsoft.PowerShell.Commands.SelectObjectCommand

Select-Object : The property cannot be processed because the property "b2" already exists.
At line:12 char:3
+   Select-Object -Property a,b2 -ExpandProperty c
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (@{b2=3; a=1; c=System.Object[]}:PSObject) [Select-Object], PSArgumentException
+ FullyQualifiedErrorId : AlreadyExistingUserSpecifiedPropertyExpand,Microsoft.PowerShell.Commands.SelectObjectCommand

 5 1  2
 8 6  7
 9 6  7

[mklement0]

First, let's look at how what you attempted with your sample command can be implemented with the current features:

$NestedObjects | % { 
  $_.b.psobject.Copy() | Add-Member -PassThru -MemberType NoteProperty a -Value $_.a
}

• $_.b.psobject.Copy() clones the custom object that is stored in the input object's .b property
• Add-Member -PassThru -MemberType NoteProperty a -Value $_.a adds the input object's .a property to the clone and outputs the clone.

---

-ExpandPropertyAsCopy would always return objects, never scalars.

I assume you mean it would always return (newly created) _custom objects_ whose _properties_ contain the requested data.

However, that's exactly what -Property already does.

If I understand you correctly, you're saying, with -ExpandPropertyAsCopy foo,

• if the type of .foo is a _custom object_, clone _it_ (create a new custom object with the same properties)

• if it is a different (non-collection) type, _wrap it in a custom object_.

Varying the behavior based on the type of the property passed to -ExpandPropertyAsCopy sounds ill-advised to me.

Select-Object was not designed with special handling of custom objects as _input_ in mind.

---

-ExpandProperty by design returns a _single property value_ from each input object.

To me, the only logical extension of that concept to _multiple_ properties is to return their values as _individual output objects_.

---

Another piece of code that demonstrates -ExpandProperty's buggy behavior,

Again, while the behavior may be surprising, it works as designed:

You're attaching the _same objects_ as property values to _multiple objects_, so modifying the attached objects via one object is reflected in the others.

---

I'm personally not convinced by what you're proposing, but if you want to pursue this, I encourage you to open a new issue - or perhaps write an RFC - that explains the proposed new functionality in a focused manner.

[iSazonov]

I must note that this cmdlet is very sensitive to changes. There is very difficult to change something without a regression.

I believe this is ready to be concluded by PowerShell Committee
/cc @SteveL-MSFT

[dragonwolf83]

The root issue is not fully cloning the object or passing ByVal before making a destructive change to the original object. I've been frustrated by this too.

What I see here is an implementation detail in terms of modifying this. The end result, for someone like me that heavily uses this feature, is for the Object to have all it's properties included with the ExpandedProperty. How it does this doesn't matter so long as it produces the same output. Changing this behavior to default to a fully cloned PSObject would solve an issue that I've had when trying to test my scripts.

For example, I have a list of Servers that has multiple IP Addresses stored in a IP_Address property. I need to change the output to give me each Server Name, IP Address, and other Server details as a separate row (PSObject). I do Select * -ExpandProperty IP_address to achieve this result.
However, to rerun this, I have to repopulate the original PSObject with the server details because the properties already exist.

[mklement0]

@dragonwolf83:

The root issue is not fully cloning the object

-ExpandProperty makes no promise of _cloning_ anything - on the contrary, its promise is to pass the referenced property's values though _as-is_.

before making a destructive change to the original object.

The distinctions may be subtle, but they matter:

• The modification isn't _destructive_, it is an act of _decoration_ - which, however, can undoubtedly can have side effects.

• The modification isn't to the input object _per se_, it is to whatever objects the values of the one property referenced by -ExpandProperty contain.

  • Only if those values happen to be instance of _reference types_ is the input object affected, _indirectly_.

Changing this behavior to default to a fully cloned PSObject

-ExpandProperty is agnostic with respect to the type of input objects's properties.

As such, we cannot assume that they're [pscustomobject] (unfortunately aka [psobject]) instances, nor is it advisable to special-case their handling.

---

For example, I have a list of Servers that has multiple IP Addresses stored in a IP_Address property. I need to change the output to give me each Server Name, IP Address, and other Server details as a separate row (PSObject).

I understand the use case, but -ExpandProperty, given its established semantics, is _not_ the way to implement this feature.

What you and @bstrautin are asking for is an entirely new feature, and commingling it with
-ExpandProperty is inviting confusion.

[dragonwolf83]

This is why I said it is an implementation detail. How it is implemented is exactly as you described. However, what matters to the user is the final result, not how it gets there. The entire method of how it creates the result can be changed so long as the result is the same. The result is the contract, not the implementation.

From my perspective, the feature works great but has a nasty side effect. It modifies the original object. One way to solve it is to have some method to clone the original object to make the modifications. Making that modification is not going to break the contract of how the feature is used.

Another method is to create a new object with property from -ExpandProperty and add the master properties to it. This may be the easier method, less memory, but does subtly change the original behavior. This would be more experimental to verify it doesn't break any scripts that pipeline the results a few times.

Either way, I don't think we need a new property to Select-Object to make this work better. Adding a new property, btw, would be a breaking change because I would have to modify all my scripts to use it when the end result is the same. That would be confusing!

[mklement0]

The result is the contract, not the implementation.

The contract of -ExpandProperty is:

• return the property values of the single property specified, _as-is_.

• if -Property is - optionally - _also_ specified, decorate the output property values with NoteProperty members reflecting the value of the properties passed to -Property.

Another method is to create a new object with property from -ExpandProperty and add the master properties to it.

That is the gist of what you're asking for.
_It is unrelated to what -ExpandProperty currently does_, and I suggest opening a _new_ issue to ask for what you want.

[dragonwolf83]

I simply disagree about creating a new issue. It is related to what -ExpandProperty currently does. All that is being asked is to not do it to the original object.

return the property values of the single property specified, as-is.

It doesn't return as-is. It creates an array. No modification is done to the original object. It doesn't add a new property, or change a property to do it. It just creates an array.

if -Property is - optionally - also specified, decorate the output property values with NoteProperty
members reflecting the value of the properties passed to -Property

The entire code for how this is done is exactly what I want done, with one small change. Decorate the output property values on a new object or a cloned object.

There is no reason to copy this behavior into a new property or cmdlet just to achieve that one change. The expectation of using Select-Object is that it does not modify the original object, and yet it does for this one use case. So fix it with a small change.

[mklement0]

It doesn't return as-is.

You're right: if the property value is array-valued, the specified property value's are _enumerated_, _as-is_.

No modification is done to the original object.

Indeed - as long as you don't also specify -Property, in which case you're opting into the decoration of each output value.

The entire code for how this is done is exactly what I want done, with one small change. Decorate the output property values on a new object

That is not _one small change_ - it is fundamentally different behavior:
Instead of outputting a (possibly enumerated) _specific property value_ of the input objects, you're asking for _new custom objects_ that _contain_ the specified property (possibly one instance per property value), alongside the properties passed to -Property.

or a cloned object.

As stated, there is no type-agnostic way to clone arbitrary input objects.

Remember that the typical use case for -ExpandProperty is to "replace" input objects with a specific property _value_ of theirs, _without_ a [pscustomobject] wrapper:

[pscustomobject] @{ foo = 1 }, [pscustomobject] @{ foo = 2 } | Select-Object -ExpandProperty foo
1
2

That is, an array of [int] values was returned, _without wrappers_.

The semantics you're proposing would alter this behavior fundamentally.

And in case you're thinking of making the behavior dependent on whether -Property is _also_ specified:

• That would be a breaking change.

• If you don't also specify -Property, how would you distinguish between [pscustomobject] @{ one = 1,2,3 } | Select-Object -ExpandProperty one returning an [int[]] array vs. wanting custom [pscustomobject] wrappers around the integer values?

[dragonwolf83]

That is not one small change - it is fundamentally different behavior:
Instead of outputting a (possibly enumerated) specific property value of the input objects, you're asking for new custom objects that contain the specified property (possibly one instance per property value), alongside the properties passed to -Property.

Which in effect what it does today, only it modifies the original object. Try running the following example and look at the results.

$origObject = @(
    [PSCustomObject] @{ Name = "Bob"; 
        Phone=@(
            [PSCustomObject] @{PhoneNo="555-447-5285"} 
            [PSCustomObject] @{PhoneNo="555-447-5286"}
        )
    }
    [PSCustomObject] @{ Name = "Jones"; 
        Phone=@(
            [PSCustomObject] @{PhoneNo="555-449-7285"} 
            [PSCustomObject] @{PhoneNo="555-449-7786"}
        )
    }
)

$newObject = $origObject | Select * -ExpandProperty Phone

Now look at Get-Member for the results of both objects

$origObject.Phone | Get-Member
$newObject | Get-Member

They are both PSCustomObjects, $newObject by all indications looks like any other PSObject. They are identical. That behavior stays the same.

What happens underneath, however, is the issue. Run this command and then compare $newObject and $origObject

$newObject | % { $_.PhoneHo = "555-555-5555" }

The change impacts both, Unexpected result! The root cause is because the object is passed by reference so any modifications to other variables modifies the original object. Yet, this doesn't happen when you just do Select-Object -Property. When you do that, it creates a new PSCustomObject. You can test that by doing:

$realNewObject = $newObject | Select *
$realNewObject | % { $_.PhoneNo=0}

Notice only $realNewObject is modified. The original properties are not changed on $newObject or $origObject

It seems clear to me that the expected and correct result is to create a new object, then expand the properties when the property is an object. I can't think of any real breaking change here to a script.

[mklement0]

The change impacts both, Unexpected result!

It's only unexpected if you expect it to work differently than the documented behavior; to recap:
By _combining_ -ExpandProperty with -Property, you're _opting into_ having the _property values_ that are output decorated - _whatever they are,_ which, in the case of reference-type instances, means that the originating object sees the decorations too.

The purpose of -ExpandProperty is _not_ to create _new objects_; the purpose is to pass out _property values_.

The behavior when you combine -ExpandProperty with -Property may be surprising, but:

• it is documented (though the docs could stand improvement), and once the underlying logic is understood, not hard to conceptualize.
• there is _no_ non-breaking transition to the behavior you're looking for, because:
  
  
  
  • a property value that can be any type cannot be cloned generically
  
  
  • the alternative of wrapping the property value in a custom object is a fundamental deviation from -ExpandProperty's mandate.
  
  

What _could_ be done is to introduce a _new_ parameter that implements the behavior you're looking for:

• enumerate the values of a designated single property
• for each value, _create a custom object_ with a property of the same name
• attach any -Property-specified values (unenumerated) to each resulting custom object

A possible name is -EnumerateProperty, e.g., based on your example:

$newObject = $origObject | Select-Object -EnumerateProperty Phone -Property Name

This would produce the following output:

    [PSCustomObject] @{ Name = "Bob";   Phone=[PSCustomObject] @{PhoneNo="555-447-5285"} },
    [PSCustomObject] @{ Name = "Bob";   Phone=[PSCustomObject] @{PhoneNo="555-447-5286"} },
    [PSCustomObject] @{ Name = "Jones"; Phone=[PSCustomObject] @{PhoneNo="555-449-7285"} },
    [PSCustomObject] @{ Name = "Jones"; Phone=[PSCustomObject] @{PhoneNo="555-449-7286"} }

[PetSerAl]

You do not need to clone object to attach different properties to it. You just need to attach properties to PSObject wrapper instead of object itself. Actually Select-Object -ExpandProperty already doing exactly that thing unless property is PSObject already:

$a = [object]::new()
Add-Member -InputObject $a -MemberType NoteProperty -Name a -Value a
$b = [pscustomobject]@{ p = $a }           | Select-Object -ExpandProperty p -Property @{ Name = 'b'; Expression = { 'b' } }
$c = [pscustomobject]@{ p = $a }           | Select-Object -ExpandProperty p -Property @{ Name = 'c'; Expression = { 'c' } }
$d = [pscustomobject]@{ p = [psobject]$a } | Select-Object -ExpandProperty p -Property @{ Name = 'd'; Expression = { 'd' } }
$e = [pscustomobject]@{ p = [psobject]$a } | Select-Object -ExpandProperty p -Property @{ Name = 'e'; Expression = { 'e' } }
$f = [pscustomobject]@{ p = $b }           | Select-Object -ExpandProperty p -Property @{ Name = 'f'; Expression = { 'f' } }
$g = [pscustomobject]@{ p = $c }           | Select-Object -ExpandProperty p -Property @{ Name = 'g'; Expression = { 'g' } }

$a, $b, $c, $d, $e, $f, $g | Format-Table a, b, c, d, e, f, g

[object]::ReferenceEquals($a, $b)
[object]::ReferenceEquals($a, $c)
[object]::ReferenceEquals($a, $d)
[object]::ReferenceEquals($a, $e)
[object]::ReferenceEquals($a, $f)
[object]::ReferenceEquals($a, $g)

[mklement0]

Indeed, @PetSerAl, that is the current behavior, as documented (albeit in less technical detail).

It is this very behavior that @bstrautin and @dragonwolf83 aren't happy with, but, as I've tried to argue, it is well-defined and consistent, even if it may be surprising at first glance.

What they're really looking for, it seems to me, is _new_ functionality, where a single designated property's possibly enumerated values, if applicable, are each _wrapped in a custom object_ (rather than being passed through, as currently with -ExpandProperty), to which the (unenumerated) -Property values are then also attached.

As I've argued, a _new_ parameter would be needed to opt into this behavior, as described in the -EnumerateProperty proposal in my previous comment.

[PetSerAl]

@mklement0 I do not see current behavior as consistent:

class A {
    [int] $a
    A ([int] $a) { $this.a = $a }
}

$a1 = 1..3 | % { [A]::new($_) }
$a2 = 1..3 | % { [A]::new($_) }

$a1 | % { $_ | Add-Member b $_.a }
$a2 | % { $_ | Add-Member b $_.a }

$b1 = foreach($_ in $a1) { [pscustomobject]@{ p = $_ } }
$b2 = $a2 | % { [pscustomobject]@{ p = $_ } }

$b1 | Select-Object -ExpandProperty p -Property @{ Name = 'c'; Expression = { $_.p.a } } | Out-Host # have `a` and `c` properties
$b2 | Select-Object -ExpandProperty p -Property @{ Name = 'c'; Expression = { $_.p.a } } | Out-Host # have `a`, `b` and `c` properties

$a1 | Out-Host # have `a` and `b` properties
$a2 | Out-Host # have `a`, `b` and `c` properties

Also, can you point where this difference in behavior documented exactly?

[mklement0]

Also, can you point where this difference in behavior documented exactly?

I wasn't aware of the difference - and I don't understand it: please explain.

From my limited understanding so far, it comes down to invisible extra [psobject] wrappers that cause subtle behavior differences, as discussed in #5579.
Note: Calling them _extra [psobject] wrappers_ was my attempt at explaining what I've observed; given that I lack a deep understanding in this area, my conceptualization may be incorrect - do tell us, if so.

What causes the different outcomes in your example is the difference between:

• using a foreach loop
  
  
  
  • $b1 = foreach($_ in $a1) { [pscustomobject]@{ p = $_ } }
  
  
• and a pipeline with% (ForEach-Object)
  
  
  
  • $b2 = $a2 | % { [pscustomobject]@{ p = $_ } }
  
  

which are _seemingly_ equivalent (with the structurally equivalent $a1 and $a2, respectively).

That they're not equivalent can be verified as follows:

$b1[0].p -is [psobject]  # $False
$b2[0].p -is [psobject]  # $True - !! extra [psobject] wrapper

Doesn't the fact these statements are _not_ equivalent strike you as a more fundamental problem?
How, exactly, does the difference then surface in Select-Object -ExpandProperty?

---

Aside from this inconsistency, however, I hope we can agree that the primary intent of -ExpandProperty is to pass a given, single property's _value_ through - enumerated, if applicable - and not to construct a [pscustomobject] with that property attached.

[PetSerAl]

In v2 extra properties were linked to PSObject wrapper. So two PSObjects wrapping the same underlaying object can have different set of extra properties. In v3 properties now linked to underlaying object (with some exceptions), but not to PSObject wrapper. But in the process PowerShell devs decided to keep/grant Select-Object -ExpandProperty ability to create independent PSObject wrappers, which links properties to themselves rather than to underlaying objects.
https://github.com/PowerShell/PowerShell/blob/4831a9fd639e9386075f0622710675105c43cff9/src/System.Management.Automation/engine/MshObject.cs#L1001-L1014
As you can see here storeTypeNameAndInstanceMembersLocally is not used, when you already have PSObject. And that is causing difference in Select-Object -ExpandProperty behavior.

So, given PowerShell have this ability, you do not need to know how to clone arbitrary object. Instead you can have two independent PSObjects wrapping the same underlaying object. Although, I do not think Select-Object should really do this. I do not see reasons why it use/have this ability in the first place (compatibility? maybe).

[iSazonov]

@lzybkr Could you please clarify about Select-Object -ExpandProperty design (@PetSerAl's comments)?

[lzybkr]

I don't remember any discussions about Select-Object -ExpandProperty in V3 when we changed the underlying representation of instance members.

As I recall, the primary factors were:

1. Availability of ConditionalWeakTable which was designed for the scenario - it was unavailable for V2
2. Make Add-Member easier to use - no need for -PassThru
3. Preserve instance members when objects are passed through C# methods, e.g. in fluent apis.
4. Preserve the PSObject public api semantics

As for the suggestion by @PetSerAl to use multiple PSObject wrappers - I would think it is viable, but you do lose point 3 above.

[mklement0]

I appreciate the background information, @PetSerAl and @lzybkr.

However, @lzybkr, I don't think @PetSerAl is advocating use of separate PSObject wrappers - on the contrary:

Instead you can have two independent PSObjects wrapping the same underlying object. Although, I do not think Select-Object should really do this. I do not see reasons why it use/have this ability in the first place (compatibility? maybe).

That means that the inconsistency pointed out by @PetSerAl should be resolved toward the documented behavior:

Associate all instance members added via -Property _directly with the base (underlying) object, whatever its type_, which, based on the current implementation, means:

• Leave the -ExpandProperty behavior alone, if the property value happens to be a [psobject] (whether a bona fide [pscustomobject] or just an incidental wrapper), as it is already correct.

  • Of course, this won't give @bstrautin and @dragonwolf83 the behavior they want; more on that in a separate comment.

• The behavior must be fixed for all other types to _not_ use the extra [psobject] wrapper, because instance members can currently _be unexpectedly discarded_.

@PetSerAl's example above has already demonstrated the loss of instance members, but let me show it more succinctly:

$v = [datetime]::now; $v | Add-Member myProp myPropValue
"[$($v.myProp)]"
'---'
 $vToo = [pscustomobject] @{ prop = $v } | Select-Object -ExpandProperty prop
"[$($vToo.myProp)]"

The above yields:

[myPropValue]
---
[]

which demonstrates that the myProp instance member was lost.

Since this affects only types _other_ than [psobject] / [pscustomobject], however, this is a bug that is separate from the discussion here, so I've created #7937.

---

Generally, the - seemingly legacy - mechanism of associating instance member with a - usually invisible - [psobject] _wrapper_ rather than the wrapped object itself strikes me as highly obscure, and not something to rely on to _emulate_ cloning of [pscustomobject] instances.

[lzybkr]

I wasn't suggesting that it was a good idea, just that it was possible.

Personally I don't use Select-Object, and I don't find this use of Select-Object intuitive, I instead prefer creating a new object explicitly, something like (and I realize it's not exactly the same):

# If I want something like this:
Get-Process | Select-Object -Property Name -ExpandProperty Modules

# I use something like this:
Get-Process | ForEach-Object {
    $n = $_.Name
    $_.Modules | ForEach-Object { [pscustomobject]@{Name = $n; Module = $_ } }
}

[mklement0]

I'm glad to hear you don't necessarily think it's a good idea, @lzybkr.

I was responding to your statement that, "As for the suggestion by @PetSerAl to use multiple PSObject wrappers", pointing out that (a) from my understanding @PetSerAl suggested _not_ to dot that, and (b) that using multiple object wrappers:

• causes the bug I've documented in #7937
• generally represent a _step back_ from the improvements introduced in v3, in that they introduce far-from-obvious subtle variations in behavior that no end user should ever have to worry about.

In other words: it's not only _not a good_ idea, it's _a bad idea_.

---

As for your example:

Yes, the behavior resulting from combining -ExpandProperty with -Property is not exactly obvious, but it can be useful.

(And I realize that my -EnumerateProperty suggestion may have missed the mark, given that it is predicated on creating a _wrapper_ [pscustomobject] for the property targeted by -ExpandProperty rather than returning a _clone_ of the property value itself, with the latter probably being what @bstrautin and @dragonwolf83 were looking for.)

In short, if I read their intent correctly, what @bstrautin and @dragonwolf83 are looking for is for
-ExpandProperty to act as -ExpandPropertyOrACloneOfItIfItisACustomObject.

Clearly, that's not's what's happening now.

Possible solutions are:

• Using a ForEach-Object call in which the target property's value is explicitly cloned, as suggested in https://github.com/PowerShell/PowerShell/issues/7768#issuecomment-421171150

• Special-casing the behavior of -ExpandProperty to _implicitly_ pass out (and decorate) a _clone_, _if_ it happens to be a _custom object_ (a type-less property bag with no base object).

The latter is an option, but amounts to an inconsistency that would require careful documentation.

[SteveL-MSFT]

Can someone summarize what the proposed change is for @PowerShell/powershell-committee to review?

[mklement0]

First, a genuine bug was uncovered, but it is now tracked in a separate issue: #7937

Second, the question is whether Select-Object -ExpandProperty -Property works as designed and exhibits useful behavior, or whether it should be fixed / enhanced:

Select-Object -ExpandProperty, when combined with -Property, currently exhibits internally consistent behavior in line with the documentation, but the behavior can be surprising and isn't easy to understand:

• The _value_ of the -ExpandProperty property is output, whatever its type, and if -Property is also present, the specified properties are added as ETS NoteProperty members to the output object, which amounts to _modifying_ that object from an ETS perspective.

• If the ETS-modified output object is being referenced elsewhere, as part of a larger object, that larger object now sees the modified object, and sending that object to Select-Object -ExpandProperty -Property _again_ then causes failure, because the NoteProperty members already exist (see code in the original post).

@bstrautin and @dragonwolf83, if I understand them correctly, are looking for a _clone_ of the -ExpandProperty property value to be output, _if_ that value represents a _custom object_. They generally expect Select-Object _never_ to modify its input.

This would amount to special-casing the treatment of -ExpandProperty property values based on their type (cloning _all_ values is not an option, because not all reference-type instances can be cloned; in an ETS sense this means that non-custom objects would still be _modified_), with the following possible implementations:

• Quiet, automatic special-casing based on the property's type; this may be too obscure.

• @bstrautin suggested an opt-in mechanism with a new parameter named, e.g.,
  -ExpandPropertyAsCopy (see https://github.com/PowerShell/PowerShell/issues/7768#issuecomment-421002893 above); the behavior if an input object _isn't_ a custom object would have to be fleshed out.

The alternative is to not make any changes and work around the issue with a ForEach-Object-based solution with manual cloning (see https://github.com/PowerShell/PowerShell/issues/7768#issuecomment-421171150 above).

I hope this is a fair summary.
My personal sense is that we don't need any changes, so I'll leave it to others to flesh out a proposed change.

[mklement0]

P.S.: Another alternative to consider:

Perhaps _cloning_ isn't necessary, _if_ it is acceptable to repeatedly modify the NoteProperty members on the _same_ objects.

That is, we could let Select-Object -ExpandProperty -Property quietly accept _preexisting_ property members and simply _update_ them, instead of complaining about them.

Arguably, this is more in in line with Select-Objects general behavior, where you can select preexisting properties or create new ones on demand (calculated properties).

Example:

$obj = [pscustomobject] @{ propToDecorateWith=1; customObj = [pscustomobject] @{ prop = 'val' } }

# First act of decoration - currently OK
# -> [pscustomobject] @{ propToDecorateWith=1; prop = 'val' } 
$obj | Select-Object -ExpandProperty customObj -Property propToDecorateWith

# Update the input object
$obj.propToDecorateWith = 2

# Second act of decoration - currently BREAKS
#    "The property cannot be processed because the property "propToDecorateWith" already exists."
# If we quietly accepted preexisting properties, we'd instead get:
#   -> [pscustomobject] @{ propToDecorateWith=2; prop = 'val' } 
# But note that the very same custom object was modified in both cases.
$obj | Select-Object -ExpandProperty customObj -Property propToDecorateWith

[dragonwolf83]

I think a better word than _clone_ is _new_ given how this currently works.

When the property specified in -ExpandProperty is a PSObject it adds any properties in -Property to the new expanded object. The issue is that since the PSObject is a reference, the expanded object is modified in the root PSObject.

If you create a new PSObject first and add the properties to it, then it won't modify the original object and the output would be the same.

The only issue for existing scripts would be if someone wasn't capturing the results of Select-Object and going back to the original object to use it. I don't know how common a practice that would be. My expectation is that most people would capture the output: $results = Select-Object.

[mklement0]

If I understand correctly, @dragonwolf83, your vote is for the quiet, automatic special-casing.

However the special-casing is implemented, if at all, it should apply to true _custom_ objects only ("property bags" without a base object), not instances of mostly-invisible helper type [psobject].

If we pursued the other alternative - quiet repeated decoration of the _same_ object - existing scripts wouldn't be affected at all, but the question is: would that give you the behavior you want, i.e., is it OK to modify the same object repeatedly, as long as the NoteProperty values have the then-desired values?

[bstrautin]

Repeatedly modifying the same object would be trouble.

Specifically, in the situation where the same object appears in more than one parent object's property, data presented by Format-Table directly after Select-Object might look correct, but, when assigned to a variable, the actual data in the objects would be different from what was displayed.

e.g. This code would display different values for what should be the same data:

$foo1 = [pscustomobject]@{foo = 'foo1'}
$foo2 = [pscustomobject]@{foo = 'foo2'}

$bars = @(
  [pscustomobject]@{bar='bar1'; foos=$foo1,$foo2}
  [pscustomobject]@{bar='bar2'; foos=$foo1}
  ) * 10

$bars | select -ExpandProperty foos -Property bar -ov expandedBars | ft | out-host

$expandedBars | ft

[iSazonov]

Repeatedly modifying the same object would be trouble.

This behavior is already "by design".

I have to remind that this cmdlet is very sensitive and it is almost impossible to change something in it without breaking something else.

I should also mention that cloning is generally a custom operation due to property nesting and the threat of an infinite loop (like in ConvertTo-Json).

If we need a new behavior, we better consider something like new cmdlet ConvertTo-Object.

[dragonwolf83]

Forget about cloning. I think that is causing confusion in this discussion. We are not asking for a deep clone of the root object. That is not what this is about.

We are asking for the output that is produced to be exactly as it is today, same properties, but just applied to a new PSObject. That way, the original object is not referenced and modified. The output would be identical, minimizing the breaking change.

This is already special cased depending on whether the property to expand is an object or an array. Any changes would just be made to that section when it detects an object.

What I'm proposing is similar to how the Join-Object script that was written long ago keeps from modifying the original object. That script creates a new PSObject first and adds the properties of 2 objects into the new object. Which is very close to what -ExpandProperty does in this instance. It takes the expanded object and adds properties of the root object to the expanded object. But because it doesn't create a new PSObject, the expanded object embedded in the root is modified.

As for the breaking change, that is what needs to be investigated. Do people primarily capture the output of Select-Object -ExpandProperty or do they run Select-Object and use the modified root object to access that property? I think most would do the former and not expect the original object to be modified. Once that behavior is analyzed, then we can decide if a switch is needed to create a new object.

[mklement0]

This is already special cased depending on whether the property to expand is an object or an array.

I wouldn't consider this special-casing, given that it's a documented feature: collection-valued properties targeted by -ExpandProperty are enumerated.

In either case, it is the (potentially enumerated) property value _itself_ that is currently decorated, at least from a PowerShell user's perspective.

So, to selectively change that for [pscustomobject] instances (true "property bags") and _effectively_ create a shallow clone of them before decorating (even if you frame it as creating a _new_ object) amounts to special-casing.

It's important to understand that for _other_ types the -ExpandProperty value _itself_ is modified, and that _can't be changed_:

# Expand a [System.IO.DirectoryInfo] property value:
$obj = [pscustomobject] @{ foo = 'hi'; bar = Get-Item / }
($obj | Select-Object -ExpandProperty bar -Property foo).foo  # -> 'hi'
$obj.bar.foo # -> 'hi' - enclosing $obj sees the change too

Now, perhaps the predominant use case for the - itself not all-too common - combination of -ExpandProperty and -Property is with [pscustomobject] instances with the expectation of creating _new_ objects always, so, yes, analyzing current usage is needed.

If so, then the quiet special-casing of [pscustomobject] will indeed make things easier, but, as stated, the special-casing would have to be carefully documented so that users won't be surprised if non-custom objects are directly "modified" (decorated).

---

@bstrautin: Yes, if you want to retain _all_ iterations of a custom object, iterative modification of the same object is not an option, but I was thinking more of a "working object" scenario: a reusable object whose temporary state is captured by transformation to a different object / format.

My sense is that repeated Select-Object -ExpandProperty -Property calls on the same object should _not_ fail, and simply update existing properties, but that's a separate issue.

However, implementing the latter would - from the perspective of your new-object expectation - make the behavior potentially more obscure; if we opt for quiet special-casing, that point would be moot.

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this. We believe the original intent is to not modify the original object and this looks like a bug. This is a Breaking Change, but the benefit outweighs the potential impact. We agree that we should just fix the original behavior so that a new object is emitted.

[iSazonov]

I'd postpone the change till next PowerSHell Core LTS version.