Source-build: System.DllNotFoundException: Unable to load shared library 'git2-8e0b172'

[s0dyy]

  fail: Microsoft.DotNet.Darc.Operations.Operation[0]
        Something failed while cloning.
  System.Exception: Something went wrong when cloning repo https://github.com/dotnet/arcade at <default branch> into /home/s0dy/source-build/bin/src/arcade ---> System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'git2-8e0b172' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libgit2-8e0b172: cannot open shared object file: No such file or directory
     at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()
     at LibGit2Sharp.Core.NativeMethods.LoadNativeLibrary() in C:\projects\libgit2sharp\LibGit2Sharp\Core\NativeMethods.cs:line 47
     at LibGit2Sharp.Core.NativeMethods..cctor() in C:\projects\libgit2sharp\LibGit2Sharp\Core\NativeMethods.cs:line 37
     --- End of inner exception stack trace ---
     at LibGit2Sharp.Core.NativeMethods.git_clone(git_repository*& repo, String origin_url, FilePath workdir_path, GitCloneOptions& opts)
     at LibGit2Sharp.Core.Proxy.git_clone(String url, String workdir, GitCloneOptions& opts) in C:\projects\libgit2sharp\LibGit2Sharp\Core\Proxy.cs:line 354
     at LibGit2Sharp.Repository.Clone(String sourceUrl, String workdirPath, CloneOptions options) in C:\projects\libgit2sharp\LibGit2Sharp\Repository.cs:line 715
     at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /home/s0dy/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 176
     --- End of inner exception stack trace ---
     at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /home/s0dy/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 210
     at Microsoft.DotNet.DarcLib.GitHubClient.Clone(String repoUri, String commit, String targetDirectory, String gitDirectory) in /home/s0dy/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/GitHubClient.cs:line 875
     at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyAndCreateGitDir(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, String gitDirRedirect, ILogger log) in /home/s0dy/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 282
     at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyWithGitDirPath(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /home/s0dy/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 270
     at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopy(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /home/s0dy/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 225
     at Microsoft.DotNet.Darc.Operations.CloneOperation.ExecuteAsync() in /home/s0dy/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 102
/home/s0dy/source-build/tools-local/init-build.proj(169,5): error MSB3073: The command "/home/s0dy/source-build/Tools/dotnetcli/dotnet /home/s0dy/source-build/tools-local/arcade-services/artifacts/bin/Microsoft.DotNet.Darc/Release/netcoreapp2.1/Microsoft.DotNet.Darc.dll clone --repos-folder /home/s0dy/source-build/bin/src/ --git-dir-folder /home/s0dy/source-build/.git/modules/src/ --include-toolset --ignore-repos ";https://dev.azure.com/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/devdiv/DevDiv/_git/DotNet-Trusted;https://devdiv.visualstudio.com/DevDiv/_git/DotNet-Trusted;https://[email protected]/dnceng/internal/_git/dotnet-optimization;https://dev.azure.com/dnceng/internal/_git/dotnet-core-setup" --azdev-pat bogus --github-pat bogus --depth 0 --debug" exited with code 42.

I am trying to make a package for Exherbo linux (since dotnet/source-build), I get an error because libgit2-8e0b172.so is linked to openssl-1.0

I realized that some libs were related to openssl-1.0 or 1.1 depending on the system. For example alpine (1.0)

s0dy ~/source-build lddtree packages/libgit2sharp.nativebinaries/1.0.235/runtimes/alpine-x64/native/libgit2-8e0b172.so

libgit2-8e0b172.so => packages/libgit2sharp.nativebinaries/1.0.235/runtimes/alpine-x64/native/libgit2-8e0b172.so (interpreter => none)
    libcurl.so.4 => /usr/host/lib/libcurl.so.4
        libssh2.so.1 => /usr/x86_64-pc-linux-gnu/lib/libssh2.so.1
        libssl.so.1.1 => /usr/x86_64-pc-linux-gnu/lib/libssl.so.1.1
        libcrypto.so.1.1 => /usr/x86_64-pc-linux-gnu/lib/libcrypto.so.1.1
            libdl.so.2 => /usr/host/lib/libdl.so.2
                ld-linux-x86-64.so.2 => /usr/host/lib/ld-linux-x86-64.so.2
        libpthread.so.0 => /usr/x86_64-pc-linux-gnu/lib/libpthread.so.0
        libc.so.6 => /usr/x86_64-pc-linux-gnu/lib/libc.so.6
    libssl.so.1.0.0 => not found
    libcrypto.so.1.0.0 => not found
    libz.so.1 => /usr/host/lib/libz.so.1
    libc.musl-x86_64.so.1 => not found

Debian (1.1)

s0dy ~/source-build ((HEAD detached at v3.0.100-preview6-012264-SDK)) lddtree packages/libgit2sharp.nativebinaries/1.0.235/runtimes/debian.9-x64/native/libgit2-8e0b172.so
libgit2-8e0b172.so => packages/libgit2sharp.nativebinaries/1.0.235/runtimes/debian.9-x64/native/libgit2-8e0b172.so (interpreter => none)
    librt.so.1 => /usr/host/lib/librt.so.1
    libpthread.so.0 => /usr/host/lib/libpthread.so.0
        ld-linux-x86-64.so.2 => /usr/host/lib/ld-linux-x86-64.so.2
    libcurl.so.4 => /usr/host/lib/libcurl.so.4
        libssh2.so.1 => /usr/x86_64-pc-linux-gnu/lib/libssh2.so.1
    libssl.so.1.1 => /usr/host/lib/libssl.so.1.1
    libcrypto.so.1.1 => /usr/host/lib/libcrypto.so.1.1
        libdl.so.2 => /usr/host/lib/libdl.so.2
    libz.so.1 => /usr/host/lib/libz.so.1
    libc.so.6 => /usr/host/lib/libc.so.6

To work around the problem, I copy libgit2-8e0b172.so (from the debian.9-x64 folder) to the linux-x64 folder

Is there a solution to have libgit2-8e0b172.so linked to openssl-1.1 directly via linux-x64?

For older versions, I could do a "git submodule update --init --recursive". How does it work now with Darc? Is it possible to download all the repositories before starting the compilation?

[s0dyy]

Any ideas ?

[markwilkie]

@dagood or @crummel - is this something you guys could help shed some light on?

[dagood]

I'm not familiar with the new Darc workflow (I'm not active in source-build), but @crummel and @dseefeld who I'd normally ask about this aren't available to answer right now.

+cc @dleeapho @adaggarwal

You might have luck running build-source-tarball.sh on a different machine where it works, then building the resulting tarball on your target machine. This is the way to do an offline build, and might not even require git (but I don't know the current status).

I'm tentatively transferring this issue to source-build because it sounds more like a question about what the source-build infra supports than a direct bug report against Darc.

[omajid]

https://github.com/dotnet/source-build/issues/1083 is the same issue, but on Fedora.

[dseefeld]

This may potentially require a Excerbo specific fix analogous to #1083

[omajid]

@dseefeld The way I understand this issue is that arcade-services wants to do some git operations and makes use of libgit2sharp. libgit2sharp is based on the unmanaged library libgit2 which is obtained and used from nuget. The nuget package (obviously) doesn't support every version of every platform (including all the various linux distributions) on every architecture.

I think source-build/arcade-services should be more flexible here.

I see two possible options that might help:

1. Let an OS (linux distro, in this case) supply a libgit2 and arcade-services should make use of that (in addition to supporting getting libgit2 from nuget.org for common cases)

2. arcade-services should do what SourceLink does and work with git directly

What do you think?

[wfurt]

There is now version without dependency on libgit2.
I think it would be great to get that in and avoid external dependency.

[omajid]

@wfurt Wow, that's great news! Do you know where this version is? I still see a dependency on LibGit2Sharp in arcade-services here: https://github.com/dotnet/arcade-services/blob/7bcd17b24aec5101e89723f31a748ed1d2385268/src/Microsoft.DotNet.Darc/src/DarcLib/Microsoft.DotNet.DarcLib.csproj#L14

[wfurt]

Take a look at https://github.com/dotnet/sourcelink/pull/288
I don't have any practical experience but I do see a hope.

[omajid]

@wfurt That's sourcelink, though. It's what I was referring to in option 2 above where I said arcade-services should do what sourcelink does. Do arcade-services and sourcelink share code or codebases? I might take a stab at merging the changes if that's the case.

[wfurt]

I don't know. I'm just hoping we can remove dependency on libgit2 instead of trying to fix it for all distributions,

[adaggarwal]

Related - #1242

[omajid]

@tmds's fix (https://github.com/libgit2/libgit2sharp/pull/1714) works fine for me.

[omajid]

https://github.com/dotnet/arcade-services/pull/694 fixes this issue for me. What is the process for bumping our arcade-services submodule ot point to that?

[adaggarwal]

This is fixed with darc install and update to netcoreapp3.0