Powershell: How do we customize serialization?

[Jaykul]

There was a PowerShell Team blog post years ago about how serialization works and how it should be possible to customize -- and of course, now I have access to all the source code, but so far I haven't been able to make extending the serialization work. If someone can help me figure this out, I'll make sure we get a HowTo document for future module authors out of it ...

I'm going to use a simple example, but remember that I'm looking for the way to make Deserialization work properly -- I don't want to hear about all the workarounds for this _particular_ case (there are several). The real problem I'm trying to solve is more complicated.

I need to call a remote function which takes a DateTimeOffset object, Unlike DateTime, DateTimeOffest isn't treated as a primitive type (although it should be #3172), so I want to fix that in my module.

Here's what I tried:

using namespace System.Management.Automation

class DateTimeOffsetDeserializer : System.Management.Automation.PSTypeConverter {

    [bool] CanConvertFrom([object]$sourceValue, [Type]$destinationType)
    {
        Write-Warning "CanConvertFrom $SourceValue to $($destinationType.FullName)"
        return ([PSObject]$sourceValue).PSTypeNames.Contains("Deserialized.System.DateTimeOffset")
    }

    [object] ConvertFrom([object]$sourceValue, [Type]$destinationType, [IFormatProvider]$formatProvider, [bool]$ignoreCase)
    {
        Write-Warning "ConvertFrom $SourceValue to $($destinationType.FullName)"
        [PSObject]$psSourceValue = $sourceValue
        return [DateTimeOffset]::new($psSourceValue.Ticks, $psSourceValue.Offset)
    }

    [bool] CanConvertTo([object]$sourceValue, [Type]$destinationType)
    {
        Write-Warning "CanConvertTo $SourceValue to $($destinationType.FullName)"
        return ([PSObject]$sourceValue).PSTypeNames.Contains("Deserialized.System.DateTimeOffset") -and $destinationType -eq "System.DateTimeOffset"
    }

    [object] ConvertTo([object]$sourceValue, [Type]$destinationType, [IFormatProvider]$formatProvider, [bool]$ignoreCase)
    {
        Write-Warning "ConvertTo $SourceValue to $($destinationType.FullName)"
        [PSObject]$psSourceValue = $sourceValue
        return [DateTimeOffset]::new($psSourceValue.Ticks, $psSourceValue.Offset)
    }
}

# This should be enough, right?
Update-TypeData -TypeName 'DateTimeOffset' -TargetTypeForDeserialization 'DateTimeOffsetDeserializer' -Force

# I tried adding all the rest of these
Update-TypeData -TypeName 'Deserialized.System.DateTimeOffset' -TargetTypeForDeserialization 'DateTimeOffsetDeserializer' -Force
Update-TypeData -TypeName 'DateTimeOffset' -MemberName 'TargetTypeForDeserialization' -MemberType 'NoteProperty' -Value 'DateTimeOffsetDeserializer'
Update-TypeData -TypeName 'Deserialized.System.DateTimeOffset' -MemberName 'TargetTypeForDeserialization' -MemberType 'NoteProperty' -Value 'DateTimeOffsetDeserializer'

But nothing makes the this come out as a DateTimeOffset

using namespace System.Management.Automation
[PSSerializer]::Deserialize( [PSSerializer]::Serialize([DateTimeOffset]::Now) ).PSTypeNames

Or make this not be "Deserialized"

Invoke-Command -ComputerName localhost { [DateTimeOffset]::Now } | % PSTypeNames

Or make this run without crashing:

Invoke-Command -ComputerName localhost { param([DateTimeOffset]$Time) $Time.LocalDateTime } -Args ([DateTimeOffset]::Now)

I also tried creating the DateTimeOffsetDeserializer class in C#, but I think I'm misunderstanding that original blog post or something about how the registration works...

[Jaykul]

Ok, after three days of trying ... I finally thought of searching through all of GitHub for use of TargetTypeForDeserialization and came across the solution. The TypeData needs to be like this:

Register the TargetType for the Deserialized type:

Update-TypeData -TypeName 'Deserialized.System.DateTimeOffset' -TargetTypeForDeserialization 'System.DateTimeOffset'

Then register the TypeConverter for that TargetType:

Update-TypeData -TypeName 'System.DateTimeOffset' -TypeConverter 'DateTimeOffsetDeserializer'

This is the part that was completely skipped over in the blog post -- if anyone can add a couple of lines to that post, that would be great! 😉

You can, of course, do all of this in a Types.ps1xml instead:

<?xml version="1.0" encoding="utf-8" ?>
<Types>
    <Type>
        <Name>System.DateTimeOffset</Name>
        <TypeConverter>
            <TypeName>DateTimeOffsetDeserializer</TypeName>
        </TypeConverter>
        <Members>
            <MemberSet>
                <Name>PSStandardMembers</Name>
                <Members>
                    <NoteProperty>
                        <Name>SerializationDepth</Name>
                        <Value>1</Value>
                    </NoteProperty>
                </Members>
            </MemberSet>
        </Members>
    </Type>
    <Type>
        <Name>Deserialized.System.DateTimeOffset</Name>
        <Members>
            <MemberSet>
                <Name>PSStandardMembers</Name>
                <Members>
                    <NoteProperty>
                        <Name>TargetTypeForDeserialization</Name>
                        <Value>System.DateTimeOffset</Value>
                    </NoteProperty>
                </Members>
            </MemberSet>
        </Members>
    </Type>
</Types>

I should note (because I haven't tested this on PowerShell 3 or 4): the explanation in the SDK makes this somewhat more convoluted -- rather than specifying the _actual target type_ as the TargetTypeForDeserialization, they state that you should specify the Deserializer as the TargetTypeForDeserialization (i.e. in place of System.DateTimeOffset above), which means that you need an extra call (or <Type> entry) to set the SerializationDepth ....

That makes _no sense_ to me, but perhaps it's actually necessary for some situations or older versions of PowerShell.

[iSazonov]

@Jaykul Great gem! Thanks!
You could put this in https://github.com/PowerShell/PowerShell-Docs

[Jaykul]

It turns out, there's a different problem: this does not work for PowerShell classes.

Take this simple case for example:

class Test {
    [string]$FirstName
    [string]$LastName
    [string]$FullName

    Test([string]$FirstName, [string]$LastName) {
        $this.FirstName = $FirstName
        $this.LastName = $LastName
        $this.FullName = $FirstName + " " + $LastName
    }
}

Because it has a custom constructor, you need a custom converter.

To hide some of the sillyness of this, I like to use an intermediate PSObjectConverter:

# A do-nothing converter, just to hide the "object" methods
class PSObjectConverter : System.Management.Automation.PSTypeConverter
{
    [bool] CanConvertFrom([PSObject]$psSourceValue, [Type]$destinationType)
    {
        return $false
    }

    [object] ConvertFrom([PSObject]$psSourceValue, [Type]$destinationType, [IFormatProvider]$formatProvider, [bool]$ignoreCase)
    {
        throw [NotImplementedException]::new()        
    }

    # These things down here are just never used. Why they must be here, I have no idea.
    [bool] CanConvertFrom([object]$sourceValue, [Type]$destinationType)
    {
        return $false;
    }

    [object] ConvertFrom([object]$sourceValue, [Type]$destinationType, [IFormatProvider]$formatProvider, [bool]$ignoreCase)
    {
        throw [NotImplementedException]::new();
    }

    [bool] CanConvertTo([object]$sourceValue, [Type]$destinationType)
    {
        throw [NotImplementedException]::new();
    }

    [object] ConvertTo([object]$sourceValue, [Type]$destinationType, [IFormatProvider]$formatProvider, [bool]$ignoreCase)
    {
        throw [NotImplementedException]::new();
    }
}


class TestConverter : PSObjectConverter
{
    [bool] CanConvertFrom([PSObject]$psSourceValue, [Type]$destinationType)
    {
        Write-Warning "CanConvertFrom $($psSourceValue.PSTypeNames)"
        return $psSourceValue.PSTypeNames.Contains("Deserialized.Test")
    }

    [object] ConvertFrom([PSObject]$psSourceValue, [Type]$destinationType, [IFormatProvider]$formatProvider, [bool]$ignoreCase)
    {
        Write-Warning "ConvertFrom $($psSourceValue.PSTypeNames)"
        Write-Warning "So there can be no doubt this does not work:"
        Write-Warning ("[Test]::new('" + $psSourceValue.FirstName + "', '" + $psSourceValue.LastName + "')")
        $Converted = [Test]::new($psSourceValue.FirstName, $psSourceValue.LastName)
        Write-Warning "REMOTE EXECUTION NEVER REACHES THIS LINE"
        return $Converted
    }
}

Finally, register the serialization:

Update-TypeData -TypeName 'Deserialized.Test' -TargetTypeForDeserialization 'Test'
Update-TypeData -TypeName 'Test' -TypeConverter 'TestConverter'

Now, test it. Locally, it works fine:

> using namespace System.Management.Automation
>  [PSSerializer]::Deserialize( [PSSerializer]::Serialize( [Test]::new("Joel","Bennett")  ) )

WARNING: CanConvertFrom Deserialized.Test Deserialized.System.Object
WARNING: ConvertFrom Deserialized.Test Deserialized.System.Object
WARNING: So there can be no doubt this does not work:
WARNING: [Test]::new('Joel', 'Bennett')
WARNING: REMOTE EXECUTION NEVER REACHES THIS LINE

FirstName LastName FullName
--------- -------- --------
Joel      Bennett  Joel Bennett

But when I try to run the command remotely, it just hangs?

Invoke-Command -ComputerName localhost {
    "Hello from ${Env:ComputerName}, I got $($args.ForEach{$_.PSTypeNames[0]}) and I'm going to send them back to you:"
    $args
    "Goodbye!"
} -Args ( [Test]::new("Joel","Bennett") )

Hello from USTWL-JOELB, I got Deserialized.Test and I'm going to send them back to you:
WARNING: CanConvertFrom Deserialized.Test Deserialized.System.Object
WARNING: ConvertFrom Deserialized.Test Deserialized.System.Object
WARNING: So there can be no doubt this does not work:
WARNING: [Test]::new('Joel', 'Bennett')

@lzybkr Is this just not possible?

[lzybkr]

Thanks for the excellent repro.

Getting this scenario to work might be tricky.

Scriptblocks have runspace affinity. Class methods are really just scriptblocks, they get runspace affinity too. I think this is an architectural flaw, but it's not easy to fix without requiring locks throughout the engine.

In this repro - there is only 1 local runspace. The deserialization is happening on a thread with no runspace and it's waiting for the Invoke-Command runspace (the only local runspace, the runspace where the module was imported) to become available to run commands.

One fix would be to not block in Invoke-Command - instead timeout and poll for events.

[Jaykul]

Thanks. I'll just ... write those classes in C# I guess. But I guess we need to document that you can write C# classes and PowerShell converters but not both in PowerShell, and not PowerShell classes with C# converters (I guess).

[Brain2000]

This is certainly not complete, but will at least let you copy over the class definition and properties to a PSSession. Maybe someone can add methods/constructors...

class test
{
  [string]$abc;
  [string]$def;
}

function getClassDef
{
    [OutputType([string])]
    param (
        [Parameter(Mandatory=$true)][Type]$classType
    )

    $create = "class $($classType.Name)`r`n{`r`n ";
    $create += $classType.GetProperties() | ForEach-Object { "   [$($_.PropertyType.ToString())]`$$($_.Name)`r`n"; }
    $create += "}";

    return $create;
}

$def = getClassDef([test]);
$s = New-PSSession someRemoteComputer
Invoke-Command $s { Invoke-Expression $Using:def } #this creates the class remotely

#now run a test
Invoke-Command $s { $t = [test]::new(); $t; }

[SSvilen]

Hi @PaulHigin - we were discussing the topic of object's rehydration after your session about remoting at PSConfEu. You suggested me to open an issue, so that you can discuss that internally and may be do a blog post about it. It seems that this is not a new topic - so I'll use that issue here as reminder. Thanks to @Jaykul I have something to work with, but may be new blog post about this topic would be a good start

[PaulHigin]

@SSvilen Thanks, I am back in office now. I am glad that this helps, but as we discussed this is really something that should be addressed in a blog. I'll try to do so in the not too distant future.

[kenny1983]

Hi guys, just wondering whether there's been any new developments on this issue in the past 4 months, as I'm trying to get the following script working:

class PSProvisioner {
    [Management.Automation.PSCredential] $Credential
    [Management.Automation.Runspaces.PSSession] $Session

    [TimeSpan] $TotalTime

    PSProvisioner() {
        $this.Credential = ([Management.Automation.PSCredential]::new(`
            "vagrant", $(ConvertTo-SecureString "vagrant" -AsPlainText -Force)))
        $this.Session = New-PSSession -ComputerName 192.168.15.11 `
            -Port 55985 -Credential $this.Credential
    }

    [void] PrintHeader([string] $header) {
        Write-Host $header
        ...Do some other stuff
    }

    [void] Test([PSProvisioner] $inst) {
        Invoke-Command -Session $this.Session -ScriptBlock {
            ($Using:inst).PrintHeader("This is a test!")
        }
    }
}

$provisioner = [PSProvisioner]::new()
$provisioner.Test($provisioner)

Which throws the following error:

Method invocation failed because [Deserialized.PSProvisioner] does not contain a method named 'PrintHeader'.
At D:\Dev\MutatioDev\provision-guest.ps1:393 char:9
+         Invoke-Command -Session $this.Session -ScriptBlock {
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (PrintHeader:String) [], RuntimeException
    + FullyQualifiedErrorId : MethodNotFound

Would absolutely adore some guidance on how to get this to work, if it's at all possible. In the meantime, I guess I'll have to go back to using scripts 🙁 MTIA!