Meshcentral: Powershell terminal is crashing on all tested nodes.

[ghplw]

I have tried connecting to a powershell terminal on several nodes at different sites but they all disconnect on their own within between 2 and 30 seconds after connecting. There are no connectivity issues to the standard terminal or any other services.

[ghplw]

On some systems it only starts crashing after the first command is issued. I'm just doing a directory listing for tests.

[theredkoala21]

I tested on a few machines that had no AV.

I have noticed that Avast seems to put Powershell in "CyberCapture" mode first run. This runs powershell, and monitors the execution. It then closes the process and reopens (if not malicious) under a different process. Which may confuse anything capturing its output (MeshAgent).

Do you have an antivirus running?

[Ylianst]

I have not seen this. What version of Windows are you guys running?

All we do in run "%windir%\System32\WindowsPowerShell\v1.0\powershell.exe" instead of the typical command shell.

If you can, try to see if there is a "meshagent.log" file in the "c:\program files\meshagent" folder. If so, send it over. That is the agent crash log - Thanks.

[theredkoala21]

Yup, can confirm. Fresh Windows 10 1903 install with nothing but Chrome, Firefox and 7zip installed.
No AV installed.
There is no meshagent.log generated.

It seems to open powershell, get the output. Displays initial output fine (PS header etc).
Send a command and get a response.
But as soon as the response has been completed the process is terminated (as seen in Taskmgr).

[ghplw]

Tested on these node windows versions with and without Windows Defender running.
10.0.17134.885, 10.0.17134.407, 10.0.17134.285, 10.0.17763.615

Tested from these setups.
Windows 10.0.18362.239 (64-bit) on VirtualBox (same virtual machine that is running the meshcentral sever)
Firefox 68.0.1 (64-bit) vanilla install with no addons.

Android 9 on Samsung Galaxy Tab S4
Chrome version 75.0.3770.143

There is no log generated on the node.

[ghplw]

Same thing happens if you go to the regular terminal and type powershell. Goes in to it then drops straight out.

[ghplw]

Fault bucket 1989735495742298711, type 5
Event Name: PowerShell
Response: Not available
Cab Id: 0

Problem signature:
P1: powershell.exe
P2: 10.0.17134.407
P3: System.ArgumentOutOfRangeException
P4: System.ArgumentOutOfRangeException
P5: unknown
P6: System.ConsoleKeyInfo..ctor
P7: unknown
P8:
P9:
P10:

Attached files:
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERE3E5.tmp.WERInternalMetadata.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERE3F6.tmp.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERE405.tmp.csv
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERE444.tmp.txt

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\Critical_powershell.exe_8bda8bdb304b6f8275de3637a425a3b97a659_00000000_03faeabb

Analysis symbol:
Rechecking for solution: 0
Report Id: bdb2b3f9-e3bd-47a1-9359-c0a913465a9d
Report Status: 268435456
Hashed bucket: 58a2fa90b421569fcb9cf5e42cb49a57
Cab Guid: 0

[ghplw]

Fault bucket , type 0
Event Name: PowerShell
Response: Not available
Cab Id: 0

Problem signature:
P1: powershell.exe
P2: 10.0.17134.407
P3: System.ArgumentOutOfRangeException
P4: System.ArgumentOutOfRangeException
P5: unknown
P6: System.ConsoleKeyInfo..ctor
P7: unknown
P8:
P9:
P10:

Attached files:

These files may be available here:

Analysis symbol:
Rechecking for solution: 0
Report Id: bdb2b3f9-e3bd-47a1-9359-c0a913465a9d
Report Status: 268697600
Hashed bucket:
Cab Guid: 0

[ghplw]

Those two events above are reported in the nodes Application event log for each crash.

[theredkoala21]

Video of Powershell crashing

https://streamable.com/y9hcj

[krayon007]

I think i almost figured it out... It seems to work fine, in 32bit mode, but the process exits upon completion in 64 bit mode... I'll do more testing later, but I think in 64bit mode, PowerShell might be spawning a new process and exiting the launched one. I saw a similar behavior with Windows TSClient. When I spawned that client from a 64 bit process, it immediately spawned a new process and exited the one I spawned. To fix that, I had to explicitely launch the 32bit Windows TSClient, (or something like that, it was a while ago), becuase I remember I had to specify the System32 folder differently, because Windows mapped the whole folder to Wow64 or something like that. I'm guessing since PowerShell is in a subfolder of System32, I'm running into the same situation.... I'll have to dig through some of my old code to see if this is what is going on....

[krayon007]

ok, I fixed it in GIT, but Ylian will need to pull the updates win-terminal.js and update the server. The issue was almost exactly the issue I saw with Windows TSClient... On 64 bit windows, if I spawn 64 bit PowerShell from the System32 folder, when I execute a command, PowerShell kills itself and respawns a 32 bit instance. This closes the pipe, and is why the terminal disconnects. I modified win-terminal.js so that it will detect if you are running 64 bit windows, and will spawn PowerShell from the SysWow64 folder instead...

[Ylianst]

Nice! It will be in the next published version.

[Ylianst]

Just published MeshCentral v0.3.8-o with Bryan's Powershell fix. Testing & feedback appreciated.

[ghplw]

Installed 0.3.8-o but still the same issue for me.

[theredkoala21]

Yeah. Same. Installed the latest -o version. Same issue. 😞 Sorry for the bad news @krayon007

[krayon007]

This might take a while to figure out, as I have the same versions of Windows on my test machines (10.0.17134.765 and Version 10.0.17134.885), but I'm not able to reproduce the issue anymore with the latest server. Let me make sure that the version running on our test server is the same as the one published in npm...

[ghplw]

Seems to be fixed in -p

[ghplw]

Spoke to soon, but I did find a node that doesn't have the issue. I'll try to figure out the differences between these two machines and let you know.

[theredkoala21]

Ver: 0.3.8-v
Clients: 2x Win 8.1

Using powershell terminal on two different machines to "ping 8.8.8.8 -t" for the last 40 minutes.
Seems the issue its fixed. Only thing is now that when i use the above command, i cant CTRL+C out of it. Both via Keyboard and the CTRL+C button.

Standard CMD terminal has same effect. Unable to quit looping program. Via CTRL+C

[krayon007]

The Ctrl-C thing is a known deficiency with the windows terminal, because we are using the windows console, which does not pass signals. It's not possible to fix in the current implementation, as I need to switch to using a virtual console, which Microsoft created for our exact use case (running a console app from within a console), but it requires the latest windows creators update, which is why I haven't gotten around to it yet...

[ghplw]

I'm still having problems. It's nothing to do with the build, I've got several examples of the same build and some work fine while others crash.

[PTR-inc]

can confirm the problem with the console terminating after the first command. On server2016 and win10 LTSC 1809 & 1607, all 64-bit
On a 64-bit server 2012R2 and win10 pro 1903 it works fine.

[ghplw]

I haven't been able to find any differences in ones that are and aren't working. Sames Windows and PowerShell builds, no AV running, same set of software installed. This is something that's going to take a much more detailed look to figure out what is causing it to crash.

[krayon007]

I finally got it to happen on Windows Server 2016, but it looks like it's the actual PowerShell process that is crashing. This may take some time to solve...

[krayon007]

I narrowed down the cause being PSReadLine. There are a bunch of bug reports on GIT on PowerShell, saying that PSReadLine crashes when stdin is redirected. I got PowerShell to crash on my Server 2016, and I got PowerShell to start with PSReadLine not loaded, and it no longer crashes...

I'm trying to see if it still crashes if I upgrade PSCore or something....

[krayon007]

Ok, I think I fixed it for real this time. There was a VK_TO_VSC mapping that takes place before it get's sent to PowerShell... With that being said, the browser sends a CRLF when you hit the enter key, and the result of the mapping for LF results in a value that this version of PowerShell crashes on.. So I modified win-terminal to ignore all mappings that are outside of the expected range that PowerShell expects (0-255).

This is fixed in Mesh Central v0.4.0-a

[krayon007]

False hope, nevermind :(
It looks like I can't just ignore scancodes outside of 0-255, because then characters like " : will stop working... But I can't just block 525, becuase then it'll crash powershell when you try to do "

So it looks like back to the drawing board. We might have to completely redo how we do keyboard input, which we'll have to do anyways to support non-english character input.

[ghplw]

It still doesn't explain why it doesn't crash on some devices. Maybe it's keyboard and language settings related?

[krayon007]

Well, there's a lot of variables at play here. Different versions of windows powershell, different versions of installed modules, and different versions of the .net runtime. Plus, when you look at the event log that was posted earlier, you can see that it's the same exception that I found. Plus when I look online, there are lots of issues around the specific module PSReadLine crashing when stdin is redirected, which is what we are doing when we do a terminal session...

To test, I modified the terminal, so the window stays visible. I verified that if I type input directly into the console window it worked, but if I specified it thru a redirected pipe to the exact same window instance, it immediately crashed PowerShell, if the character code was not between 0 and 255. The actual crash is happening inside the .net runtime when it tries to read a character from the input stream.

I don't think it's language/keyboard related, because I had everything set to US english.
I narrowed it all the way down to the call to the win32 call 'WriteConsoleInput'. If I write a value > 255, it crashes powershell on my Windows Server 2016 machine... When I was running an upgraded Powershell 6.2.2, I got a much better error log, that showed the Powershell stacktrace and exception message, which said that it was expecting a value between 0 and 255, and it happened in the .net runtime when reading from the input stream. My windows 10 machine was running the same version of PSReadline of 1.2, and it doesn't have any problems, so the only thing I can think of, is the .net runtime was different, as that's where the crash is happening. I verified that the same value is written to the win32 call. I tried both PSReadline 1.2 and 2.0 with Powershell 5 and Powershell 6 on my Windows Server 2016 machine. I didn't check the .net version, as I had to take off...

[krayon007]

Ok, I think I got it this time. The issue was when I called VkScanCode(), I used the result to populate the INPUT_RECORD struct... The issue is the VkScanCode returns the scan code in the low order byte, and the key state in the high order byte.... So I modified it, so I masked the low order byte, and now it works correctly, including all the other keys, like : and "

I just need to fix another issue before I have Ylian publish the changes... It looks like on the version of windows/terminal that crashes without this fix, has some different behavior (ie, scrolling is busted and non-powershell is exhibiting some odd behavior), so I'm going to try to fix those before we publish. (Ylian is on vacation anyways)

[krayon007]

If you can test this with the latest server (0.4.5-b), this should be resolved if you are using a newer build of Windows with the new ConPTY APIs. You'll know if you are using the new terminal, because the toolbar will show the executing command on the terminal. This version of the terminal is able to pass ctrl-c, etc to the running process, etc. It also seems to be much friendlier with powershell.