Powershell: Session Reconnection Timeout

[emgerner-msft]

Hey folks,

I'm trying to setup a powershell connection to retry for longer than 4 minutes on network failure. I looked at pssessionoptions and tried IdleTimeout and MaxConnectionRetryCount, but I didn't think those would be right and they weren't... still the 4 minutes. Is there some other setting in here I've misunderstood the docs for, or is there another way of doing this?

For clarification, the message I get is below. I'd like rather than 4 minutes to try something like 10 minutes as I know the server is going to be back soon.

WARNING: The network connection to xx.xxx.xx.xx has been interrupted. Attempting to reconnect for up to 4 minutes...

[iSazonov]

You could set OpenTimeout property.

[emgerner-msft]

I don't think that has anything to do with reconnect -- just initial connect. I'm not an expert though. In any case, I did just try it anyways and I get the same old output about attempting to reconnect for 4 minutes.

[iSazonov]

I found that the 4 min MaxRetryConnectionTime come from WSMAN but I did not found that PowerShell sets the parameter.
Perhaps @PaulHigin could add more info.

[alexandair]

AFAIK, that value is hard-coded and cannot be set up.

[PaulHigin]

As mentioned, the connection retry timeout is performed at the WinRM layer. There is an undocumented registry key that let's you set the retry timeout. However, it was added to decrease the retry timeout for testing purposes. I don't know if you can use it to increase the retry timeout to a value greater than than the default max of 4 mins, but you can experiment with it and find out.

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN\Client
DWORD max_retry_timeout_ms

[PaulHigin]

You need to restart the WinRM service after setting the key value.

Restart-Service WinRM

[emgerner-msft]

@PaulHigin Thanks Paul, I'll give that a shot.

Is there a different way of dealing with temporary network outages that is more generally used? I tried the whole Receive-PsSession thing but found the docs (or my understanding of them) a little bit lacking and what I came up with has some limitations. If there's a sample laying around somewhere I can checkout that would be great.

[iSazonov]

You could do reconnect in explicit cycle.

[PaulHigin]

@emgerner-msft
I am not sure what docs you have read but this does a pretty good job describing PowerShell remoting disconnect/reconnect.

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_remote_disconnected_sessions?view=powershell-6

If a PowerShell remote connection is dropped due to network problems, it goes into a disconnect mode for a certain period of time (idle timeout) during which you can manually reconnect to the server either from the original client or another client. The above article talks about how the disconnect/reconnect works.

about_Remote_Disconnected_Sessions - PowerShell

[emgerner-msft]

@PaulHigin Thanks Paul, I had read through that. Here's the parts that threw me a bit:

1) How to get the results:

"You can use Receive-PSSession rather than using the Connect-PSSession cmdlet. If the session is already reconnected, Receive-PSSession gets the results of commands that ran when the session was disconnected. If the PSSession is still disconnected, Receive-PSSession connects to it and then gets the results of commands that ran while it was disconnected. Receive-PSSession can return the results in a job (asynchronously) or to the host program (synchronously). Use the OutTarget parameter to select Job or Host. The default value is Host. However, if the command that's being received was started in the current session as a Job, it's returned as a Job by default."

The return value of Receive-PSSession was unclear to me from this. Reading this, I thought that unless I specified out target as a job, it would return either the results of the command I invoked via Invoke-Command or attempt to reconnect and return nothing if the job hadn't finished. However, when I actually tested I saw I got a job even though I did not specify out target as a job. Is that expected?

1. State and Availability properties

In this section all the examples of busy sections were disconnected and busy was defined as:

A value of Busy indicates that you can't connect to the PSSession because it's connected to another session.

So, when I waited a bit on the job I got from Receive-PsSession I was pretty confused when I got a session that was opened/busy. I'm assuming that's because when the session reconnected the invoke-command I had run previously would continue meaning I couldn't send more to the session, but I'm still not certain I get the semantics here.

1. Based on 2, it seems like the job I get finishes when both the session reconnects and the previous command I invoked via invoke-command finishes. Assuming this is correct, it's mildly annoying now to do a reconnect timer because the job is waiting on both those things to finish -- not just the session reconnect. It's surmountable of course with a small window where I check if the session is now opened but the job is still busy, but it's much less simple than passing a simple timeout.

Here's roughly what I ended up with after a lot of fiddling. I've seen it work correctly a few times as far as I can tell from the print statements.

$results = Invoke-Command -Session $session -ScriptBlock {...}

while ($session.State -like "Disconnected")
{
        Write-Warning "Attempting to reconnect for up to 6 more minutes..."

        # Reconnect waiting 6 more minutes on top of the standard 4 (10 total)
        $job = Receive-PSSession $session
        for($i = 0; $i -lt 6 -and $session.State -like "Disconnected"; $i++)
        {
            Write-Warning "Continue attempting to reconnect to $($session.ComputerName) ..."
            Wait-Job $job -Timeout 60
        }

        # If the session has reconnected, continue to wait for the test to finished
        if ($session.State -like "Opened")
        {
            Write-Warning "The network connection to $($session.ComputerName) has been restored."
            $results = Receive-Job $job -Wait
        }
        else
        {
            break
        }
}

[PaulHigin]

Receive-PSSession will return a job by default if the original session was created as a job, or otherwise it just returns the output data. But you can always ensure what form you get by using the -OutTarget parameter.

Receive-PSSession -OutTarget <Host | Job>

Yes, unfortunately the PSSession state during a disconnect/reconnect is very confusing. We tried to squeeze disconnect semantics into existing State and Availability fields. Anyway, here is what each combination means:

State==Opened, Availability==Busy -> Client is connected to Target and Target is running
State==Opened, Availability==Available -> Client is connected to Target and Target is idle
State==Disconnected, Availability==None -> Client is disconnected from Target but can connect
State==Disconnected, Availability==Busy -> Target is connected to some other client. This client cannot connect.

When a session is disconnected (for whatever reason, network flakiness or manually), if script/command is running on the Target, it will continue to run although in a 'disconnected' state. This is complicated if the script returns data. The data can't be transferred to the client so it will be cached on the Target but only up to 1MB (I think is the limit). After that the script will block and cannot continue until a client connects to receive data. Unless you opted for a PSSession with -OutBufferMode set to 'Drop', in which case the Target script continues to run but output data is dropped after the cache is filled.

In any case, once you reconnect a client to a Target PSSession, if that session was/is running it needs to complete before you can run more script commands on it. 'Complete' may mean the script actually needs to finish running, or it may mean that cached output data needs to be transferred to the client. Once both those things occur then the script 'completes' and the session becomes available for more work:

State==Opened, Availability==Available

[emgerner-msft]

Okay, sounds like I should probably set the out target and then I can be certain this will run reliably. That does clarify things for the session state/availability. It looks like I lucked into the right explanation. I haven't set out buffer mode but we don't return all that much data -- basically just test output.

Wish there was an easier way to do this but thanks for the help. I'll close the issue.

[emgerner-msft]

Hey folks, I've been running this a while now with the code I pasted above and I'm still having issues. Only main change was specifying -OutTarget Job per the discussion and adding logging on session state when I started seeing problems. Sometimes this code seems to work, and other times I get something like the below. The session says it's disconnected, but when I try to run Receive-PsSession it indicates it's not and I don't get a job back per the log line below. The connection is now opened and busy probably running the command I sent it, but now i don't have a way to wait on the results of my command. What's the expected workflow and is there something else I missed in my above code? What am I supposed to do in this case?

WARNING: The network connection to 10.128.xx.xx has been interrupted. Attempting to reconnect for up to 4 minutes...
WARNING: Attempting to reconnect to 10.128.xx.xx ...
WARNING: Attempting to reconnect to 10.128.xx.xx ...
WARNING: Attempting to reconnect to 10.128.xx.xx ...
WARNING: Attempting to reconnect to 10.128.xx.xx ...
WARNING: Attempting to reconnect to 10.128.xx.xx ...
WARNING: The reconnection attempt to 10.128.xx.xx failed. Attempting to disconnect the session...
WARNING: Computer 10.128.27.28 has been successfully disconnected.
WARNING: 12/10/2019 2:35:31 PM: [ERR] Network connectivity to 10.128.xx.xx has been lost and the reconnection attempt failed. Please repair the network connection and reconnect using Connect-PSSession or Receive-PSSession.
WARNING: 12/10/2019 2:35:31 PM: [WRN] Attempting to reconnect for up to 6 more minutes...
WARNING: 12/10/2019 2:35:40 PM: [ERR] Reconnect attempt failed. Session state: Opened, session availability: Busy, exception: There is no disconnected command associated with this runspace.

[iSazonov]

At first look you need add OutTarget parameter:
$job = Receive-PSSession $session -OutTarget Job

[emgerner-msft]

Per the second sentence of my above message, I already did. ;)

[iSazonov]

I have only one idea - to add -AsJob to Invoke-Command -Session $session -ScriptBlock {...}

[PaulHigin]

I don't fully understand the scenario. You say you get a successful disconnect message above, but after you try to reconnect with Receive-PSSession -OutTarget Job, it fails? Can you provide more details in how it fails?

It should return a Job object that is running, or an error.

[emgerner-msft]

Yes, exactly. I'm stumped. To be clear, this happens <1/10 times -- most of the time it works just fine. I see the standard disconnect messages, and my code actually even checks that the session is disconnected before running the Receive, but then receive tells me I'm no longer disconnected and the session state agrees. The logs I get are above, word for word minus xx'ing out the IP. The error message I get when I try to run Receive-PsSession is incorporated in the last log line I pasted: "There is no disconnected command associated with this runspace." -- all I did is also include the session state when I logged that. I don't have multiple sessions and I'm not doing anything in parallel outside of this script or anything. Pasting the updated code below so you can map it to the log output, but the only change truly is logging and the -OutTarget Job change.

$results = Invoke-Command -Session $session -ScriptBlock {...}

while ($session.State -like "Disconnected")
{
        Write-Warning "Attempting to reconnect for up to 6 more minutes..."

        # Reconnect waiting 6 more minutes on top of the standard 4 (10 total)
        try
        {
            $job = Receive-PSSession $session -OutTarget Job
        }
        catch
        {
            LogError "Reconnect attempt failed. Session state: $($session.State), session availability: $($session.Availability), exception: $_"
            break
        }

        for($i = 0; $i -lt 6 -and $session.State -like "Disconnected"; $i++)
        {
            Write-Warning "Continue attempting to reconnect to $($session.ComputerName) ..."
            Wait-Job $job -Timeout 60
        }

        # If the session has reconnected, continue to wait for the test to finished
        if ($session.State -like "Opened")
        {
            Write-Warning "The network connection to $($session.ComputerName) has been restored."
            $results = Receive-Job $job -Wait
        }
        else
        {
            break
        }
}

[PaulHigin]

@emgerner-msft

Thanks for persisting with this. From your information I was able to find a simple repro on my machine, and this is definitely a regression bug. This works in Windows PowerShell but not in recent versions of PowerShell Core.

More Info:
When an Invoke-Command session is disconnected, a temporary job object is created to facilitate a session running command re-connection. The bug is that the temporary job object is not created correctly.

Possible workaround:
The reconnect should work if you run Invoke-Command as a job:

$results = Invoke-Command -session $session -Script {...} -AsJob | Receive-Job -Wait
...