Powershell: Helper method converting PSObject to string may fail

[SteveL-MSFT]

This was encountered by a MS Partner team. When calling a cmdlet that outputs lots of objects, the SmartToString() helper throws the exception "InvalidOperationException: Collection was modified after the enumerator was instantiated" as the PSObject is modified before the conversion completes. There is already code to catch an ExtendedTypeSystemException and we could extend that for any exception and just return string.empty. Or alternatively, we could make a copy of the object but that would impact perf.

In the repro, it was explicitly going through the Format-List code, although Format-Table may also use this helper.

[mklement0]

This may not be _technically_ related, but it reminded me of #6163

[iSazonov]

If the PSObject is changed while it is enumerated it looks like a race condition in user code (how do they change it? from another runspace?) - why should we hide the exception?

[SteveL-MSFT]

@iSazonov in this specific repro, they were enumerating a large collection of AD objects and it only repro'd for them when transcription was enabled although I'm not sure that is directly related. Transcription may have just slowed it down to make the race condition happen more easily.

[iSazonov]

It looks as fundamental problem. I caught the same with AccountManagement API. It depends on a context and every call to the object causes a "remote" call to ensure that the object is live. So if such check presents in enumerator we can not resolve this exception naturally.

[HerbertMauerer]

Hello,
Redacted by @joeyaiello

The problem happens in the Active Directory Admin Center (ADAC) when there are many objects in a container. I have a customer waiting for a resolution on this. And I have the worry that additional customers run into this as they switch from the older toolset to ADAC.

I'm aware you have other deliverables. I would appreciate if you could share an ETA when you think you can start working on this problem.

Thanks

Herbert Mauerer

[vexx32]

If what @iSazonov is saying is correct, I suspect the issue can only really be resolved in the ActiveDirectory module itself.

PowerShell can attempt to _mitigate_ the eventual effect of it, but it can't change the way the ActiveDirectory module is choosing to operate. The best we could hope for from the PowerShell end would probably be that some objects would end up quietly skipped and maybe a less intrusive error would be recorded after enumeration completes.

But even then, I'm not sure it's something PowerShell will be able to account for.

[iSazonov]

@HerbertMauerer Because ADAC is not OSS we need a simple repro like a script or GitHub shared C# project. We need to understand what API to investigate.
Perhaps you could share more information about the broken scenario. As I know ADAC limits to 20000 objects by default and max limit is 100000. Perhaps it is not right things to build so large PSObject and convert it to string.

[HerbertMauerer]

It is tough to rebuild what ADAC does. When I capture the command it sends and run the same get-adobject query, it succeeds.

Addtl. consideration:
We had a past issue where only ADAC was failing with PS transscripting enabled,:
3196545 ADDS: Admin: Active Directory Administrative Center does not work with PowerShell Transcripting
Redacted by @joeyaiello

This is the case here as well. When you turn off transscripting, it works.

[iSazonov]

I guess ADAC implements custom transcription and you could look the code - I wonder if it changes an object in the enumerator.

[HerbertMauerer]

Hello,

do you have Windows source access?
I could also hook you up with the DEV Owner of ADAC.

Herbert

[iSazonov]

No, I haven't the access.

[HerbertMauerer]

Redacted by @joeyaiello

[SteveL-MSFT]

@HerbertMauerer that information won't help anyone here in the community.

[SteveL-MSFT]

For @PowerShell/powershell-committee, a couple of options:

• do nothing
• add warning/non-terminating error and return empty string allowing pipeline to continue
• copy collection before enumerating

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this, we agreed for 7.1 we should have a warning message saying "Object skipped: " returning an empty string.

[rjmholt]

If the PSObject is changed while it is enumerated it looks like a race condition in user code (how do they change it? from another runspace?) - why should we hide the exception?

This happens in some contexts with PowerShell APIs when they are called off the pipeline thread. If your module does something like get a value from PowerShell, store it, allow the pipeline to continue and later tries to use or manipulate the variable, issues like this may occur. As someone who's hit issues like this in many places, it's painful, but it represents a race condition in your code.

It's also worth noting that formatters contain script, so they need a runspace/thread to work on.

So to resolve this properly in your code you must:

• Ensure you do not yield the pipeline before you have finished processing PowerShell objects. In a cmdlet this is simple; you just ensure that you have fully processed all PowerShell objects before returning from the cmdlet methods. If you are obtaining PowerShell objects without a pipeline (e.g. by API traversal or reflection) you should be very careful about how and when you do so. In all cases, the object's safe lifetime is tethered to the pipeline thread's state.
• Fall back to Out-String if needed. Out-String is the best mechanism for rendering PowerShell formatting to a string object. Similarly, Out-File is best for printing to a file. These can be run like this:

string strRepr;
using (var pwsh = PowerShell.Create(RunspaceMode.CurrentRunspace))
{
    strRepr = pwsh.AddCommand("Out-String").Invoke(inputObjectCollection);
}

I'll also note that if you're in a hosted context rather than in a PowerShell module, there are other ways to ensure that your code is always run on the pipeline thread. However, they require effort to implement, so you should make sure you have good reason to make the investment.

[iSazonov]

we should have a warning message saying "Object skipped: " returning an empty string.

It looks like hiding a race condition in customer code. :-( It can be dangerous.

[rjmholt]

From the original description, it sounds like the issue was occurring in the actual call of Format-List. If it's being called as we expect, it just sounds like there's some kind of race condition or over-aggressive optimisation in our code.

[iSazonov]

@rjmholt From the comment https://github.com/PowerShell/PowerShell/issues/11797#issuecomment-661056380 the issue is with transcripting or in transcripting.

[rjmholt]

Original public info from @HerbertMauerer:

AD Admin Center (DSAC.EXE) is switching to a container with 20000 or more objects.
This works fine without powershell transscripting. When it is enabled, you get an error dialog saying:
"Collection was modified after the enumerator was instantiated"

And you don’t get the list of objects in the container. The customer encounters this for the “Deleted Objects” container when he wants to restore AD objects, but I can also reproduce it with a OU with 20000 objects.
When you run the Get-ADObject query DSAC uses under the hood in Powershell, the query is successful. So you can’t really rebuild what DSAC is doing in plain PS.

I worked it to the point where we see an invalid result set size variable within .Net runtime.