Posh-git: Powershell hangs on startup

[jpierson]

Just recently I installed Win32-OpenSSH using the new chocolatey package and I noticed that after rebooting I'm no longer able to start up PowerShell due to it hanging indefinitely. Not sure if this issue is an issue more with posh-git or Win32-OpenSSH but I thought I'd open some discussion.

Here is a starting place.
https://github.com/PowerShell/Win32-OpenSSH/issues/42

[jpierson]

Possibly related to https://github.com/dahlbyk/posh-git/issues/103

[dahlbyk]

I assume commenting out the posh-git import in your $PROFILE (or starting PowerShell with -NoProfile) fixes the issue. What do you get from Get-Command ssh-* | select Path? Are you able to ssh-add in isolation (from PowerShell or cmd.exe)?

[jpierson]

Actually I more surgically commented out the line Start-SshAgent -Quiet from profile.example.ps1 which then did allow me to get into PowerShell. Note that the fact that PowerShell ISE still worked was also a clue to where the problem was occurring.

Get-Command ssh-* | select Path

Path

C:\Program Files\OpenSSH-Win64ssh-add.exe
C:\Program Files\OpenSSH-Win64\ssh-agent.exe
C:\Program Files\OpenSSH-Win64\ssh-keygen.exe
C:\Users\myuser\ManuallyInstalledApps\OpenSSH-Win32\ssh-keygen.exe
C:\Program Files\OpenSSH-Win64\ssh-keyscan.exe
C:\Program Files\OpenSSH-Win64\ssh-keysign.exe
C:\Program Files\OpenSSH-Win64\ssh-pkcs11-helper.exe

I was able to run ssh-add without a problem as well once I commented out the line mentioned above.

[dahlbyk]

So we have a problem in Start-SshAgent. Do you see anything if you run Start-SshAgent (no -Quiet) PowerShell loads? Are you able to run ssh-agent by itself?

[jpierson]

Running Start-SshAgent seems to just hang after executing.

[dahlbyk]

Running Start-SshAgent seems to just hang after executing.

Can you tell where it's hanging? Are you able to run ssh-agent outside of Start-SshAgent?

[jpierson]

I'm unfamialr with ssh-agent but I am able to run ssh-agent in PowerShell. It gives me output like the following.

set SSH_AUTH_SOCK=C:\Users\myname/.ssh/agent-7138.19688
set SSH_AGENT_PID=18736
echo Agent pid 18736;

Keep in mind that I have Posh-Git still enabled but the Start-SshAgent -Quiet line commented out still as mentioned above in order to get in to PowerShell.

[bazzilic]

I have experienced the same issue when I tried to install win32-openssh using Chocolatey yesterday. I am not sure though, was it the win32-openssh package issue or posh-git. I ended up uninstalling the package.

[dahlbyk]

Well posh-git will use whatever ssh-agent it finds via PATH, and then attempts to guess where ssh-agent.exe might be located if it can't find it. Installing Win32-OpenSSH with Chocolatey will place it on PATH, so posh-git will use that.

My initial guess is that you're running into https://github.com/PowerShell/Win32-OpenSSH/issues/110 at https://github.com/dahlbyk/posh-git/blob/721089444e2ce9e2bfbcad0b5e0daa5ea8085d66/GitUtils.ps1#L319

https://github.com/PowerShell/Win32-OpenSSH/issues/35 is also related to ssh-agent, but does not seem related to this issue.

[dahlbyk]

Found it: https://github.com/PowerShell/Win32-OpenSSH/issues/110#issuecomment-190534110

Until that can be fixed, we might want to add a warning here if the ssh-agent agent looks like it's from Win32-OpenSSH. :frowning:

[theaquamarine]

Latest Win32-OpenSSH release no longer includes ssh-agent and ssh-add.
https://github.com/PowerShell/Win32-OpenSSH/releases/tag/3_19_2016

[bazzilic]

Based on the release notes it looks like this is temporary.

[sixlettervariables]

I get intermittent hangs when PowerShell is used in a post-build script with Visual Studio 2015 with PoSH installed. Appears related to the SSH_AUTH_SOCK.env file being locked. Is this related to this issue?

[dahlbyk]

I get intermittent hangs when PowerShell is used in a post-build script with Visual Studio 2015 with PoSH installed.

Try spawning PowerShell with -NoProfile.

Appears related to the SSH_AUTH_SOCK.env file being locked. Is this related to this issue?

_Probably_ not? posh-git writes the .env files here as the agent processes complete (ultimately using Set-TempEnv). As far as I can tell, this bug prevents ssh-agent.exe from yielding any output that might result in setenv being called.

That being said, does your build hang only occur on machines with Win32-OpenSSH installed?

[sixlettervariables]

I'm currently the only dev with PoSH installed and yes I have Win32-OpenSSH, but the -NoProfile flag appears to help.

[dahlbyk]

but the -NoProfile flag appears to help.

:+1:

[lzybkr]

There is a bug in V3 and V4 of PowerShell that you might be hitting. If you load psreadline and some other module that generates an error into $error (older versions of posh-git definitely did, I'm not sure about newer), then you would occasionally hit a hang.

If this is the issue, you can check if PowerShell starts without a hang, immediately look at $error, if you have PSReadline and $error is not empty and you have V3/V4 PowerShell, this is likely the issue. The best fix is to avoid writing anything to $error, maybe requiring an update to posh-git. Another workaround is to call 'Get-Module -ListAvailable' in your profile - this can be slow, hence not really recommended.

[dahlbyk]

https://github.com/PowerShell/Win32-OpenSSH/issues/110#issuecomment-219603837:

ssh-agent is reimplemented for Windows as a OS service. check out 5_15_2016 release details

[jkodroff]

@dahlbyk It appears to still be happening as of win32-openssh v2016.05.15. Maybe the ssh-agent as windows service thing is in the next release. Either way, it's not really a posh-git problem.

[dahlbyk]

If anyone were so inclined, we could suppress using ssh-agent if we can detect Win32-OpenSSH is installed. Or figure out a way to force using the one that ships with Git rather than the one on the path.

[dahlbyk]

From @DarwinJS on Chocolatey Gallery:

Installed the update to 0.6.1.20160330 to fix profile errors. Now when I launch powershell from anywhere, powershell opens, then a CMD opens. If I close the CMD, the powershell prompt never displays an actual input prompt - sits there like it's busy still booting up.

Do you happen to have Win32-OpenSSH installed? This seems like it could be related to [this issue].

In that package I do check if another sshd process / service is running and leave it alone if so.

What if you check if ssh-agent is already running before trying to start it?

Or is it possible to leave ssh activities up to the underlying git package? Let the user decide go on to properly configure git if they use keys?

There is a bigger issue here where SSH has been integrated into a lot of windows stuff - what to do if they then install win32-openssh?

I guess in theory win32-openssh would back down from being the primary on anything that is already installed in places it wants to go and it would issue warning messages?

Currently we default to whatever ssh-agent is found on PATH in Start-SshAgent. I wonder if we should skip that check and always use our Find-Ssh helper (which looks relative to the location of git.exe) instead?

[DarwinJS]

Currently we default to whatever ssh-agent is found on PATH in Start-SshAgent. I wonder if we should skip that check and always use our Find-Ssh helper (which looks relative to the location of git.exe) instead?

When Win32-OpenSSH is installed (and the user picked the option to install sshd), the ssh-agent is defined and started as a service. Unfortunately this means you have to get into system context to add keys - I do this with the scheduler and it can be seen in the code of the package.

What about looking for a running process called ssh-agent and just stand down if it is running?

Or better yet - pre-check if ssh-agent is running and which software on the system put it there? This would hopeful reveal the user's intent on which SSH they want to be primary on their system.

You can look in win32-openssh to see code patterns for figuring out the path of (and therefore which software package owns) a running sshd or ssh-agent. Win32-OpenSSH package only installs or upgrades or starts/stops ssh-agent or sshd IF the instance running is from it's own folder structure.

[dahlbyk]

What about looking for a running process called ssh-agent and just stand down if it is running?

Or better yet - pre-check if ssh-agent is running and which software on the system put it there?

We try to do this already in Start-SshAgent using a Get-SshAgent function that already works for MSYS ssh and plink. I would expect adapting these to also check for Win32-OpenSSH would be relatively simple.

[DarwinJS]

OK - here are the official install instructions: https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

I just realized my package is not compliant with them because they didn't previously specify the application folder - so I think I put the binaries in "C:\Program Files" under the same subfolder that is in the zip. Which is "OpenSSH-Win32" and "OpenSSH-Win64" respectively.

I will work on aligning the folder name on the next package revision.

[theaquamarine]

Does Win32-OpenSSH actually work with git? I've never been able to make operations perform correctly and https://github.com/PowerShell/Win32-OpenSSH/issues/247 suggests it's not just me. If it doesn't, I'd be inclined to say that we want to special case Win32-OpenSSH to avoid using it.

Equally, if/when git and Win32-OpenSSH do play well together, will git actually use the Win32-OpenSSH ssh.exe if it's first in the path, or will it use its own? If it prefers to use its own, I feel like we should also prefer to use the one provided with git, since the whole point of posh-git is to work with git.

[DarwinJS]

Win32-openssh probably has a ways to go to take up git responsibilities - but even then, who would want to do all that reconfig?

One challenge is that the ssh that comes with git is not actually used unless the individual sets the machine up to do so.

So maybe win32-openssh should look for a running instance of either ssh-agent or sshd that is not it's own - and if found, requires a special switch to forcibly install either of these on a system with it already configured?

It won't handle the reverse install order (win32-openssh installed first then git) - but it could be a good first step.

Maybe the same with the path if it finds a reference to ssh on it?

[dahlbyk]

Is anyone using Win32-OpenSSH able to take https://github.com/dahlbyk/posh-git/pull/295 for a spin to make sure it works as expected for you?

[rkeithhill]

@lzybkr

generates an error into $error

We are trying to avoid this but it is kind of a PITA. When we execute git.exe and it writes to stderr, we want to avoid users seeing that error text so we redirect - 2>$null. Unfortunately that causes the stderr output to be written to $Error.

We have code now that checks the $Error.Count and removes new errors. That kind of breaks down if $Error has hit its maximum count. Is there a better way to A) prevent users from seeing stderr and B) not writing stderr to $Error when executing a native command?

[dahlbyk]

This should be resolved by #586 in an upcoming v0.8 release.

[dahlbyk]

I believe this is adequately addressed by posh-sshell