Git: Git network operations fail when libeay32.dll and/or ssleay32.dll exist in system folders

[whoisj]

• [ ] I was not able to find an open or closed issue matching what I'm seeing.

I did find a similar issue #1038, but it was incomplete in scope.

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.12.0.windows.1
built from commit: d4baef1cdfef9de62d43f9682a98ba57adc842e4
sizeof-long: 4
machine: x86_64

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.14393]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Path Option: Cmd
SSH Option: OpenSSH
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: ConHost
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Enabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Only seen with Portable and MinGit packages. Doesn't seem to reproduce with installed instance of Git for Windows. Did not perform extensive testing honestly.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

None. Git is being launched directly from Visual Studio 2017.

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

<VS_INSTALL>\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\libexec\git-core\git.exe -c "gc.auto=0" -c "credential.helper=manager" -c "core.askpass=askpass" --no-pager clone --progress --verbose --recurse-submodules "https://<redacted_url>" "<redacted_local_path>"

• What did you expect to occur after running these commands?

Git to clone the repository from the remote URL to the local path.

• What actually happened instead?

Git exited with 128, because git-remote-https failed to correctly load the libeay32.dll and ssleay32.dll libraries. The load failure is caused by LoadLibraryEx. When an application asks Windows to load a library, Windows looks for a file with a matching name in a predictable pattern: local folder, system folder, folders on %PATH%.

When a badly behaved installer places a copy of libeay32.dll and/or ssleay32.dll in the system folder, git-remote-https is tricked into loading the wrong library. This is because git-remote-https lives in the mingw32/libexec/git-core folder where as the libraries it depends on live in wingw32/bin and Window's LoadLibraryEx searches system folders first.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

It's an "always on feature" unfortunately.

[whoisj]

My suggestion would be to place all of the libraries Git depends on in the same folder as the executable. I'm not sure if that is actually practical or not, but copying files around has resolved the issue for many users of Visual Studio 2017.

[fourpastmidnight]

Yeah, I've seen this before, too in other circumstances not related to Git for Windows when those DLLs are in the %System32% folder. I think what you proposed is probably the best option, as this sort of thing is what had resolved the issue for me in the past, too.

[dscho]

I have a experimental patch to move core Git's .exe files from libexec/git-core/ to bin/ (i.e. into the same directory where their .dll files live). Note: not all .exe files from there want to move, GCM, for example, wants to stay there as it has no dependencies in bin/.

I will push this patch pending verification that it does fix the issue.

[dscho]

Oh, and this patch is MinGit only, of course, as we already have code in the installer and in PortableGit's post-install script to hard-link/copy .dll files from bin/ into libexec/git-core/ as needed.

For the record: In the installer/portable Git, i do not want to move all those git-*.exe files from libexec/git-core/ into bin/, as it would clutter the tab-completion inside the Git Bash.

[dscho]

Whoops.

[whoisj]

I've started testing this but my environment isn't automated so it'll take a day or so.

Some day I'll get time to automate this stuff. :wink:

[dscho]

Thanks!

[shiftkey]

@whoisj @dscho glad to see this one resolved so quickly - there's many interesting installer in the wild that do this exact (naughty) behaviour!

[whoisj]

@whoisj @dscho glad to see this one resolved so quickly - there's many interesting installer in the wild that do this exact (naughty) behaviour!

More applications need to be aware of the dangers of hierarchical library placements and System32 dynamic library injection. Especially when talking about crypto and security related libraries.

[dscho]

glad to see this one resolved so quickly

@shiftkey that's unfortunately a MinGit-only solution.

My idea to address this in the Portable Git is to simply just duplicate all .dll files from mingw??\bin to mingw??\libexec\git-core, then letting the post-install.bat script replace them by hardlinks (if that is possible).

What do you think?

[whoisj]

@dscho seems like that would work - I'm still curious about the idea of just statically linking everything.

I get that git.exe would get massive, but I'm seeing a large-ish number of DLL loads for something as simple as git version with one taking 1-2 ms to load.

Here's a procmon log of the command being run. Libeay32.dll gets loaded for this as well, which means if I place a 0-byte dll in my System32 folder Git is completely dead in the water.
git-version.zip

[shiftkey]

@dscho yes, that workaround sounds acceptable. I've had to deal with similar issues to what @whoisj is reporting here, but I'm neutral on whether static linking is a better approach versus leveraging the existing post-install.bat script.

[dscho]

This took substantially longer than I hoped: 7-Zip was not smart enough to add only a couple of bytes, instead adding 4MB to the portable Git!

I solved this by upgrading p7zip to the current version, which was unfortunately more involved than simply replacing the version string in the PKGBUILD file...

While at it, I now also use p7zip to package MinGit's .zip files, shaving off ~3% of the file size.

Anyway, this ticket is now addressed.