Nerdbank.gitversioning: GetBuildVersion task fails for full Framework builds

[ap0llo]

Repro steps

• Create a new Console Application (.NET Framework)
• Install package Nerdbank.GitVersionong (either using packages.config or )
• Build the project

Error:
`` System.IO.FileNotFoundException: Could not load file or assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified. File name: 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' at Nerdbank.GitVersioning.VersionOracle.Create(String projectDirectory, String gitRepoDirectory, ICloudBuild cloudBuild, Nullable1 overrideBuildNumberOffset, String projectPathRelativeToGitRepoRoot)
at Nerdbank.GitVersioning.Tasks.GetBuildVersion.ExecuteInner()
at MSBuildExtensionTask.ContextAwareTask.Execute()
at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
at Microsoft.Build.BackEnd.TaskBuilder.d__26.MoveNext()
````

I tested this using both Visual Studio 2015 and 2017 (15.8.1) on both Windows 7 and Windows 10 machines.

I do not know if it is relevant, but the Console Application is targeting .NET 4.6.1

[ap0llo]

I played around a little more and installing the .NET 4.7.1 Targeting pack solves the issue as this seems to put netstandard.dll into the GAC

Also there is a open PR for LibGit2Sharp that adds a net461 target (https://github.com/libgit2/libgit2sharp/pull/1606), I guess updating the referenced version of LibGit2Sharp once this lands on NuGet might solve the issue too.

[AArnott]

Ah, that explains it. I was surprised to see the error since NB.GV doesn't target netstandard 2.0. So ya, libgit2sharp must have tripped us up here.

[devlead]

libgit2/libgit2sharp#1606 is now merged, so framework moniker should be in next release.

[AArnott]

❗️ Workaround: Use version 2.1.84 (which uses an older libgit2sharp dependency that didn't have the bug).

[AArnott]

libgit2sharp 0.26.0-preview-0054 is now published to nuget with the fix. I'm adopting it now.

[AArnott]

...and that version regresses non-Windows platforms... again. Man, this is so hard to keep stable.

I'll look into it and loop in our very helpful libgit2sharp expert if necessary.

[AArnott]

@bording: can you help decipher this log, which we got when updating to the latest libgit2sharp 0.26.0-preview-0054 release?
ld_debug.zip

Unhandled Exception: System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'git2-8e0b172' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libgit2-8e0b172: cannot open shared object file: No such file or directory
   at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()
   at LibGit2Sharp.Core.NativeMethods.InitializeNativeLibrary()
   at LibGit2Sharp.Core.NativeMethods..cctor()
   --- End of inner exception stack trace ---
   at LibGit2Sharp.Core.NativeMethods.git_libgit2_opts(Int32 option, UInt32 level, String path)
   at LibGit2Sharp.GlobalSettings.SetConfigSearchPaths(ConfigurationLevel level, String[] paths)
   at Nerdbank.GitVersioning.GitExtensions.OpenGitRepo(String pathUnderGitRepo)
   at Nerdbank.GitVersioning.Tool.Program.OnGetVersionCommand(String projectPath, String format, String versionOrRef) in D:\git\Nerdbank.GitVersioning\src\nbgv\Program.cs:line 235
   at Nerdbank.GitVersioning.Tool.Program.Main(String[] args) in D:\git\Nerdbank.GitVersioning\src\nbgv\Program.cs:line 104

[bording]

@AArnott I’ve been out of town, so I haven’t had a chance to take a look at it this yet. At this point it might be Tuesday before I’ll be able to. Wanted to let you know that it’s on my radar.

I’ll be so happy once we get rid of all the problematic native dependencies and fix this once and for all!

[bording]

@AArnott Here are the relevant lines:

521:    checking for version `CURL_OPENSSL_3' in file /usr/lib/x86_64-linux-gnu/libcurl.so.4 [0] required by file /home/andrew/.dotnet/tools/.store/nbgv/2.2.22-ga4374dff14/nbgv/2.2.22-ga4374dff14/tools/netcoreapp2.1/any/runtimes/linux-x64/native/libgit2-8e0b172.so [0]
521:    /usr/lib/x86_64-linux-gnu/libcurl.so.4: error: version lookup error: version `CURL_OPENSSL_3' not found (required by /home/andrew/.dotnet/tools/.store/nbgv/2.2.22-ga4374dff14/nbgv/2.2.22-ga4374dff14/tools/netcoreapp2.1/any/runtimes/linux-x64/native/libgit2-8e0b172.so) (fatal)

The problem you're running into is related to the version of curl that the library is being built against has changed between the releases. The 217 version of the library was built against libcurl-gnutls, which got picked up by default when that version was built on Travis CI.

I've since moved to building the libraries in docker, where I had to be more explicit about what dependencies were available, and I picked the curl-openssl package instead becuase it made more sense to me since we also rely on OpenSSL: https://github.com/libgit2/libgit2sharp.nativebinaries/blob/master/Dockerfile.linux-x64#L5

From my testing, the only distro I was aware of this being a problem on was Ubuntu 18.04, which came out after I had made that change. What Linux distro were you using when you got that error?

It's possible I could update the dockerfile to use libcurl4-gnutls-dev instead, but at this point I'd be afraid I'd just end up breaking more things in the process. The real fix for all of this going to be removing the OpenSSL and curl dependencies entirely.

[AArnott]

From my testing, the only distro I was aware of this being a problem on was Ubuntu 18.04, which came out after I had made that change. What Linux distro were you using when you got that error?

Yes, I was seeing it fail on Ubuntu 18.04. Given that is the newest LTS for Ubuntu, that's a significant problem for it to be broken by default there.

The real fix for all of this going to be removing the OpenSSL and curl dependencies entirely.

When will that come out?

I guess the best thing I can do in the meantime is document the requirement and update my own docker image to compensate.

[bording]

When will that come out?

https://github.com/libgit2/libgit2sharp/pull/1618 and https://github.com/libgit2/libgit2sharp.nativebinaries/pull/77 is where the work is going on for that.

However, in the mean time, I'm going to get https://github.com/libgit2/libgit2sharp.nativebinaries/pull/74 merged in, and put out an updated preview that has an Ubuntu 18.04 library, which should unblock all of this.

[AArnott]

Thanks!

So does that mean libgit2sharp will contain a new set of binaries for when it runs on ubuntu 18.04? Will I need to make any changes to help libgit2sharp find the right native binaries on the right OS and version?

[AArnott]

I tried running sudo apt install libcurl4-openssl-dev libssl-dev on my Ubuntu 18.04 system and it still fails. Is there anything else I should need?

[bording]

So does that mean libgit2sharp will contain a new set of binaries for when it runs on ubuntu 18.04? Will I need to make any changes to help libgit2sharp find the right native binaries on the right OS and version?

Yes, you would have to add code to choose the right library, similar to what was talked about for #219. Until the removal of the OpenSSL and curl dependencies are completed, there's no way to have a single linux binary what works for all distros, and MSBuild doesn't help you at all in getting it right.

I tried running sudo apt install libcurl4-openssl-dev libssl-dev on my Ubuntu 18.04 system and it still fails. Is there anything else I should need?

Unfortunately, there is nothing you can do here, because the current linux-x64 binary is built on Ubuntu 14.04, which has OpenSSL 1.0, which was also the version used all the way up to 17.x, but 18.04 switched to OpenSSL 1.1. That's why a new binary is needed.

It does look there might be an unofficial libssl1.0.0 package for 18.04, but I haven't tried it. You'd have to add the Launchpad PPA to APT to be able to try it.

[bording]

It does look there might be an unofficial libssl1.0.0 package for 18.04, but I haven't tried it. You'd have to add the Launchpad PPA to APT to be able to try it.

Actually, it looks like it might be available after all. Let me test a couple things and I'll let you know.

[bording]

@AArnott You can try installing both libcurl3 and libssl1.0.0 and see if that works. I think having those installed would get the existing linux-x64 binary working, until there's a specific 18.04 lib that is built against libcurl4 and libssl1.1.

[AArnott]

Thanks. I'll try that. In the meantime, I've written a collection of regression tests in terms of Azure Pipelines in #241. Now every change to NB.GV builds on Windows, then deploys to OSX, Ubuntu 16 and Ubuntu 18 to verify that it can actually run. :) No more manual testing misses leading to regressions. Yay.

[AArnott]

Your workaround worked. I'll document it and take the fix as-is for now.

[AArnott]

Yes, you would have to add code to choose the right library

I was hoping that libgit2sharp would include all the necessary special OS checks and library loads, since it's closest to its native dependency and knows exactly which native version it's using and available OSs are included in it, while libgit2sharp users would probably want to stay away from that as much as possible.

Installing libcurl3 and libssl1.0.0 worked on my Linux on Windows subsystem, but it isn't working in a docker container. Maybe there are more dependencies that are missing, I don't know. I'm installing packages pseudo-randomly to see if I can figure out what I'm missing.

[bording]

was hoping that libgit2sharp would include all the necessary special OS checks and library loads, since it's closest to its native dependency and knows exactly which native version it's using and available OSs are included in it, while libgit2sharp users would probably want to stay away from that as much as possible.

Unfortunately, I'm not aware of a way to do that, or it would definitely be something I'd be trying to do.

Installing libcurl3 and libssl1.0.0 worked on my Linux on Windows subsystem, but it isn't working in a docker container.

Getting an LD_DEBUG log from the container would help figure that out. Another thing worth trying would be to run ldd on the linux binary from the container and see what it reports.

[AArnott]

I checked the LD_DEBUG log myself and it has the same error I reported originally. But maybe there are other clues I'm missing? Checking again (after installing other packages), I noticed this:

/root/.nuget/packages/nerdbank.gitversioning/2.2.31-g995e912831/build/lib/linux/x86_64/libgit2-8e0b172.so: error: symbol lookup error: undefined symbol: DllMain (fatal)

Here is the full log:
ld_debug.zip

Unfortunately, I'm not aware of a way to do that, or it would definitely be something I'd be trying to do.

What is it that nerdbank.gitversioning is in a position to do that libgit2sharp can't do?

[AArnott]

I got it working. Now I'm just whittling down the script to find out what the minimal set of packages is.

[bording]

I got it working. Now I'm just whittling down the script to find out what the minimal set of packages is.

The ldd command is helpful that since it will show every dependency of the library and whiter or not it could be found on the system.

What is it that nerdbank.gitversioning is in a position to do that libgit2sharp can't do?

Primarily it's around using a custom AssemblyLoadContext. I was looking at the one being used in dotnet/sourcelink, and when I asked about it, I was told that it would need to be something the consumer provided, and wouldn't be something LibGit2Sharp could provide.

[AArnott]

Curious. Well those load contexts are black magic at best, so you're probably right.

That magic incantation for it to work is simply:

apt install -y libcurl3 libssl1.0.0 # libcurl4-openssl-dev libssl-dev

If I include the commented out packages, it fails.

[AArnott]

In particular, running dotnet-install.sh to install various versions of the SDK and runtime brings in something that breaks it. So I have to use a very minimal dockerfile layer on top of microsoft/dotnet:2.1-sdk-bionic that just installs those two packages (I also add sudo), and that works -- until I run dotnet-install.sh.

[AArnott]

I could only get the PR to build by writing a custom docker image. That's not an acceptable story for users of nb.gv. I'd like to investigate using a libgit2sharp fork that goes back to before this bionic regression but cherry picks the netstandard2 fix. That would seem to get the best of both worlds.

[AArnott]

I notice in looking at libgit2sharp history, that the native binaries were updated twice between the point where this worked on Bionic and where it stopped:

1.0.210->1.0.233
1.0.233->1.0.235

The first of these was in the same commit that added a net461 target. @bording, can you tell me which of these upgrades introduced the issue with bionic?

[bording]

It might make sense to take a look at the AssemblyLoadContext implementation in dotnet/sourcelink that does have a way to be RID-aware, and can load the appropriate assembly. That would let you take advantage of the new ubuntu.18.04-x64 library that will be in the next LibGit2Sharp preview, and should work directly with the microsoft/dotnet:2.1-sdk-bionic image. It would also gain you compatibility with all of the other Linux distros.

[bording]

The first of these was in the same commit that added a net461 target. @bording, can you tell me which of these upgrades introduced the issue with bionic?

Off the top of my head I'm not sure, but I should be able to check them and figure it out. One thing to keep in mind however is that the newer versions of the native library were updated because of security vulnerabilities in libgit2, so using an older version may not be a great idea.

[AArnott]

Off the top of my head I'm not sure, but I should be able to check them and figure it out.

I'd appreciate it, if you can come up with the answer without too much trouble. It will take me about 30 minutes of work to just try the first commit and hope it works. But if we know the first commit included the problem, there's no point in trying it.

the newer versions of the native library were updated because of security vulnerabilities in libgit2, so using an older version may not be a great idea.

I'm not sure which vulnerability this was that you're referring to, but I doubt it's of concern. By the time NB.GV invokes libgit2sharp, the contributors to the repo already have had ample chance to run arbitrary code on the machine, so any attack from the repo can already be mounted. Further, since NB.GV doesn't use libgit2sharp for any remote fetch/pull, any code not already on the box and in a position to compromise the machine won't be brought in. Is there anything I'm missing?

That would let you take advantage of the new ubuntu.18.04-x64 library that will be in the next LibGit2Sharp preview

That sounds good, but when will it come out?

[bording]

I'd appreciate it, if you can come up with the answer without too much trouble. It will take me about 30 minutes of work to just try the first commit and hope it works. But if we know the first commit included the problem, there's no point in trying it.

It looks like the last package you'd be able to use is 1.0.217. When the package expanded to include a lot more Linux RIDs, that's when the binary switched from the GnuTLS-based curl to the OpenSSL curl.

BTW, this weekend I spent some time investigating if I could switch back to using the GnuTLS version, but that would actually end up breaking OpenSUSE/SLES support, so I decided not to pursue it further, given the current goal of removing curl and OpenSSL altogether.

That sounds good, but when will it come out?

It should be really soon, just need to get https://github.com/libgit2/libgit2sharp/pull/1636 merged and then get the new package up on nuget.org.

[AArnott]

It should be really soon

Awesome. I'll just sit tight for it then.

[bording]

FYI LibGit2Sharp 0.26.0-preview-0062 is up on nuget.org and it includes a separate ubuntu.18.04-x64 binary.

[ekwus]

Hey

Possibly worth noting here that LibGir2Sharp 0.26.0-preview-0062 is still referencing libssl.so.1.0.0 which has been deprecated and considered unsafe. I'm still having to add it to the microsoft/dotnet:2.1 docker container to get GitVersioning to work.

Cheers

Dave

[bording]

@ekwus I think you might be misunderstanding how linux libraries work. While that might be named libssl.so.1.0.0, that doesn't actually tell you what version of the library it is. For example, on my Ubuntu 16.04 WSL instance, there is a /lib/x86_64-linux-gnu/libssl.so.1.0.0 file, but when I check the installed package version, it is:

libssl1.0.0/xenial-updates,xenial-security,now 1.0.2g-1ubuntu4.13 amd64 [installed]

So that's actually OpenSSL 1.0.2g, and the latest version offered on Ubuntu 16.04.

In other words, just looking at the name of the file isn't going to be enough tell you what version it is, especially for OpenSSL.

[ekwus]

Hey

I could easily be misunderstanding how it works as most of my experience is in the Windows world for good or bad :-)

I guess the issue is its not working and I haven't check recently but at the point if checking the previous time libssl 1.0.2 was included in the microsoft/dotnet container. However no matter how I tried to put a symlink in to the required version it would always complain one way or the other.

The only solution I've found that works is to install the actual libssl.so.1.0.0 file on to the system. I guess it is just an issue with what the microsoft/dotnet container is providing and we'll just need to keep working around it.

Cheers

Dave

[bording]

@ekwus You're hitting a problem similar to #219. LibGit2Sharp ships different binaries for all/most of the distros that .NET Core supports, but this project currently doesn't leverage them.

The dotnet image you're trying to use is likely the one based on Debian 9 (stretch), and LibGit2Sharp ships a separate binary for use with that. However, since Nerdbank.GitVersioning can't currently use it, that's why you find yourself needing to add another dependency to the package.

[AArnott]

@bording I'm adopting 0.26.0-preview-0070 but notice that as for supported RIDs, you have ubuntu.18.04-x64 but nothing for ubuntu 16. Which RID works on that one? Or I suppose since you now target .NET Core 2.0 you had to drop support for Ubuntu 16?

[AArnott]

It might make sense to take a look at the AssemblyLoadContext implementation in dotnet/sourcelink that does have a way to be RID-aware, and can load the appropriate assembly.

What is it that nerdbank.gitversioning is in a position to do that libgit2sharp can't do?

Primarily it's around using a custom AssemblyLoadContext. I was looking at the one being used in dotnet/sourcelink, and when I asked about it, I was told that it would need to be something the consumer provided, and wouldn't be something LibGit2Sharp could provide.

I'm guessing the RuntimeIdMap.cs file is part of the solution here that you're suggesting I copy into nb.gv. Is that right? This is a whopping amount of code. And just glancing at it, I don't understand why it can't be in libgit2sharp itself. Anyway, I'm looking into it, but I do hope @bording that you'll be available to code review #244 when it's ready.

[AArnott]

OK, I found why it's not great for a library to contain: the AssemblyLoadContext boundary must be treated such that there are only a few controlled entry points. ☹️

[bording]

@bording I'm adopting 0.26.0-preview-0070 but notice that as for supported RIDs, you have ubuntu.18.04-x64 but nothing for ubuntu 16. Which RID works on that one? Or I suppose since you now target .NET Core 2.0 you had to drop support for Ubuntu 16?

Based on the RID graph, Ubuntu 16 would end up using the linux-x64 RID, as it was previously doing.

Take a look at https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/runtime.compatibility.json

For a given RID, that will show you the fallback options if the exact RID doesn't exist in the package.

For example, here is ubuntu.16.04-x64:

 "ubuntu.16.04-x64": [
    "ubuntu.16.04-x64",
    "ubuntu.16.04",
    "ubuntu-x64",
    "ubuntu",
    "debian-x64",
    "debian",
    "linux-x64",
    "linux",
    "unix-x64",
    "unix",
    "any",
    "base"
  ],

[bording]

OK, I found why it's not great for a library to contain: the AssemblyLoadContext boundary must be treated such that there are only a few controlled entry points. ☹️

Yeah, you have to access all your types via reflection to get them out of the AssemblyLoadContext. So we could add the class with the RID logic to LibGit2Sharp (or more likely, and extra package you could reference), but the consumer of the package would still have to do some special things to use it.

That would consolidate the RID knowledge in one place though, vs. each consumer needing to add that separately and update things if/when we change the RIDs we ship in the native binary package.

[AArnott]

That would consolidate the RID knowledge in one place though, vs. each consumer needing to add that separately and update things if/when we change the RIDs we ship in the native binary package.

Ya, that sounds good!

[bording]

@AArnott I've opened https://github.com/libgit2/libgit2sharp/issues/1649 to consider the idea of providing an AssemblyLoadContext package.