Runtime: Segmentation fault on dotnet restore on Debian

[TheRealPiotrP]

Steps to reproduce

% mkdir /tmp/dotnet
% cd /tmp/dotnet
% wget https://dotnetcli.blob.core.windows.net/dotnet/Sdk/rel-1.0.0/dotnet-dev-debian-x64.latest.tar.gz
% tar xf dotnet-dev-debian-x64.latest.tar.gz
% ls
dotnet  dotnet-dev-debian-x64.latest.tar.gz  host  LICENSE.txt  sdk  shared  ThirdPartyNotices.txt
% mkdir prj
% cd prj
% ../dotnet new
Created new C# project in /tmp/dotnet/prj.
% ../dotnet restore
log  : Restoring packages for /tmp/dotnet/prj/project.json...
zsh: segmentation fault  ../dotnet restore

Expected behavior

dotnet restore should succeed

Actual behavior

dotnet restore doesn't download or do anything.

Environment data

dotnet --info output:

.NET Command Line Tools (1.0.0-preview3-003892)

Product Information:
 Version:            1.0.0-preview3-003892
 Commit SHA-1 hash:  5301f08423

Runtime Environment:
 OS Name:     debian
 OS Version:  
 OS Platform: Linux
 RID:         debian.8-x64

Cause of this problem

There are two different versions of libssl linked. Maybe the build happend in an unclean environment or it the package got incrementally built.

/tmp/dotnet/shared/Microsoft.NETCore.App/1.0.1/System.Net.Http.Native.so:
        libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007f563d13c000)
/tmp/dotnet/shared/Microsoft.NETCore.App/1.0.1/System.Security.Cryptography.Native.so:
        libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fe6caf4b000)

% LD_DEBUG=bindings /tmp/dotnet/dotnet restore 2>&1 >/dev/null |grep \`X509_verify_cert"'\|STORE_CTX_init'"              
      4799:     binding file /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 [0] to /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 [0]: normal symbol `X509_STORE_CTX_init' [OPENSSL_1.0.2d]
      4799:     binding file /tmp/dotnet/shared/Microsoft.NETCore.App/1.0.1/System.Security.Cryptography.Native.so [0] to /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [0]: normal symbol `X509_verify_cert' [OPENSSL_1.0.0]
zsh: segmentation fault  LD_DEBUG=bindings /tmp/dotnet/dotnet restore 2>&1 > /dev/null | 

% gdb --args /tmp/dotnet/dotnet restore
Reading symbols from /tmp/dotnet/dotnet...(no debugging symbols found)...done.
(gdb) r
Starting program: /tmp/dotnet/dotnet restore
log  : Restoring packages for /mnt/other/Alex-Projekt/src/web/project.json...

Thread 14 "dotnet" received signal SIGSEGV, Segmentation fault.
0x00007fffe24a235d in trust_1oidany (trust=0x7fffe2789e10 <trstandard+80>, x=0x7fff3802b240, flags=0) at x509_trs.c:270
270     x509_trs.c: Datei oder Verzeichnis nicht gefunden.

(gdb) bt
#0  0x00007fffe24a235d in trust_1oidany (trust=0x7fffe2789e10 <trstandard+80>, x=0x7fff3802b240, flags=0) at x509_trs.c:270
dotnet/core-setup#1  0x00007fffe249c19c in check_trust (ctx=0x7fff5cfa8ee0) at x509_vfy.c:923
dotnet/core-setup#2  X509_verify_cert (ctx=<optimized out>) at x509_vfy.c:381
dotnet/core-setup#3  0x00007fff7cf546f5 in ?? ()
dotnet/core-setup#4  0x00007fff5cfa8ca0 in ?? ()
dotnet/core-setup#5  0xbafbcb4b7ab40f52 in ?? ()
dotnet/core-setup#6  0x0000000006042606 in ?? ()
dotnet/core-setup#7  0x00007ffff67d7f90 in vtable for InlinedCallFrame () from /tmp/dotnet/shared/Microsoft.NETCore.App/1.0.1/libcoreclr.so
dotnet/core-setup#8  0x00007fff5cfa95e8 in ?? ()
dotnet/core-setup#9  0x00007fff7d7082f0 in ?? ()
dotnet/core-setup#10 0x00007fff7d7082f0 in ?? ()

(gdb) up
dotnet/core-setup#1  0x00007fffe249c19c in check_trust (ctx=0x7fff5cfa8ee0) at x509_vfy.c:923
923     x509_vfy.c: Datei oder Verzeichnis nicht gefunden.
(gdb) p ctx->get_issuer
$2 = (int (*)(X509 **, X509_STORE_CTX *, X509 *)) 0x7fff5fd0a630 <X509_STORE_CTX_get1_issuer>
(gdb) i sy X509_STORE_CTX_get1_issuer
X509_STORE_CTX_get1_issuer in section .text of /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(gdb) p X509_STORE_CTX_get1_issuer
$3 = {int (X509 **, X509_STORE_CTX *, X509 *)} 0x7fffe24a1020 <X509_STORE_CTX_get1_issuer>

(gdb) br X509_STORE_CTX_get1_issuer
Breakpoint 1 at 0x7fff5fd0a630: X509_STORE_CTX_get1_issuer. (2 locations)
(gdb) i br
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   <MULTIPLE>
1.1                         y     0x00007fff5fd0a630 in X509_STORE_CTX_get1_issuer at x509_lu.c:612
1.2                         y     0x00007fffe24a1020 in X509_STORE_CTX_get1_issuer at x509_lu.c:612

As you can see, the function get_issuer in the context X509_STORE_CTX is not from the code segment of the libssl1.0.0 which is used for X509_verify_cert.

Solution

Rebuild System.Security.Cryptography.Native.so with libssl1.0.2.

% git https://github.com/dotnet/corefx
% cd src/Native
% ./build-native.sh release
% cp ../../bin/Linux.x64.Release/Native/System.Security.Cryptography.Native.so /tmp/dotnet/shared/Microsoft.NETCore.App/1.0.1/

[TheRealPiotrP]

Moved from https://github.com/dotnet/cli/issues/4477 on behalf of @jo-so.

@schellap @gkhanna79 can you guys take a look?

[gkhanna79]

@stephentoub @bartonjs Can you PTAL?

[bartonjs]

I'm still working on having a Debian machine to see what they look like, but my psychic debugging says that our build machines are stale somehow.

• System.Net.Http.Native doesn't link against libssl, it only links against libcurl (which links against libssl).
• System.Security.Cryptography.Native(.OpenSsl) does link against libssl, using CMake's find_package.
• If a rebuild worked for @jo-so then that means find_package on their system returned libssl.so.1.0.2 as the file to link against.
• This suggests our build machine is out of date.

Or, since I have no idea what version that tgz is associated with, Debian has upgraded, and we haven't.
It's pretty weird that they are considering the 1.0.0 and 1.0.2 as distinct SxS libraries... I would have expected symlinks which got collapsed by the loader. But I can't verify that point until my system finishes installing.

[jo-so]

libssl 1.0.2 doesn't exist in Debian 8.0. Do you have any backports installed?

https://packages.debian.org/search?suite=default&section=all&arch=arm64&searchon=names&keywords=libssl1

[bartonjs]

@jo-so I guess I'd have to ask you the same question. Using a relatively clean install of debian8:

jbarton@jsb-debian:~/hmm/shared/Microsoft.NETCore.App/1.0.1$ ldd System.Security.Cryptography.Native.so | grep ssl
        libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f3d39089000)

jbarton@jsb-debian:~/hmm/shared/Microsoft.NETCore.App/1.0.1$ ldd System.Net.Http.Native.so | grep ssl
        libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007ff5f352f000)

jbarton@jsb-debian:~/hmm/shared/Microsoft.NETCore.App/1.0.1$ ldd System.Net.Http.Native.so | grep curl
        libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f51ffb40000)
jbarton@jsb-debian:~/hmm/shared/Microsoft.NETCore.App/1.0.1$ ldd  /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep ssl
        libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f43e0cbb000)

And, using the LD_DEBUG output:

jbarton@jsb-debian:~/hmm/test$ grep X509_STORE_CTX_init logit.log
      1830:     binding file /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 [0] to /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [0]: normal symbol `X509_STORE_CTX_init' [OPENSSL_1.0.0]
jbarton@jsb-debian:~/hmm/test$ grep X509_verify_cert logit.log
      1830:     binding file /usr/lib/x86_64-linux-gnu/libcurl.so.4 [0] to /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [0]: normal symbol `X509_verify_cert_error_string' [OPENSSL_1.0.0]
      1830:     binding file /home/jbarton/hmm/shared/Microsoft.NETCore.App/1.0.1/System.Security.Cryptography.Native.so [0] to /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 [0]: normal symbol `X509_verify_cert' [OPENSSL_1.0.0]

Everything looks good on my clean debian8.

[fogzot]

libssl1.0.2 is currently in testing and unstable. It is not in Jessie/Debian 8.

$ apt-cache policy libssl1.0.2
libssl1.0.2:
  Installed: 1.0.2j-4
  Candidate: 1.0.2j-4
  Version table:
 *** 1.0.2j-4 500
        500 http://ftp.it.debian.org/debian unstable/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2j-1 500
        500 http://ftp.it.debian.org/debian testing/main amd64 Packages

[JustArchi]

This issue is misleading. It is indeed libssl1.0.0 library that is segfaulting, but on behalf of libcurl3 one. This is because dotnet depends on old versions of libcurl3 for some reason, at least in ubuntu 16.04/14.04 + debian8 packages.

apt-get install libcurl3=7.38.0-4+deb8u5 libcurl4-openssl-dev=7.38.0-4+deb8u5

This fixes it. Grab any version from jessie and install it on testing/sid.

     7.51.0-1 500
        500 http://ftp.pl.debian.org/debian sid/main amd64 Packages
     7.50.1-1 500
        500 http://ftp.pl.debian.org/debian testing/main amd64 Packages
 *** 7.38.0-4+deb8u5 500
        500 http://ftp.pl.debian.org/debian stable-proposed-updates/main amd64 Packages
        500 http://security.debian.org/debian-security stable/updates/main amd64 Packages
        100 /var/lib/dpkg/status
     7.38.0-4+deb8u4 500
        500 http://ftp.pl.debian.org/debian stable/main amd64 Packages

[fogzot]

Indeed! I can confirm that this fixes the problem un Debian testing/unstable as of today.

[JustArchi]

Nice to hear that it works for you too. I silently hope that dotnet team will put more efforts into this bug, because it has at least half a year by now.

[bartonjs]

If the problem is only on testing/unstable versions of Debian then the problem is likely that we don't build for (or support) those versions. We only have support for stable (Jessie).

If you have a Stretch (or Sid) system and would like to try rebuilding the native corefx components to match the libraries you have present, https://github.com/dotnet/corefx/issues/13221#issuecomment-262070038 mentions that on a thread of a segfault on Stretch. (Provided link itself has a link, but I'm providing the one I did for context and attribution)

[eerhardt]

@bartonjs @stephentoub - I don't believe there is any work in dotnet/core-setup for this issue. If there is work that needs to be done, can you move this issue to the appropriate repo? I'm assuming dotnet/corefx.

[aolney]

Here's what I did to fix dotnet on Debian unstable (sid)

• Modified the install script to handle Debian 9 and a bash shell
• Executed sudo dotnet-install.sh -Version 2.0.0-preview1-005791 --verbose -InstallDir /opt/dotnet
• After it runs do find /opt/dotnet -name '*.so' -type f -print | xargs ldd | grep 'not found' to see if you are missing any libraries
• For me, it was liblldb.so. I installed liblldb-3.7 and then linked the library to /usr/lib/x86_64-linux-gnu/liblldb.so
• Even after this, dotnet restore would segfault, apparently because dotnet depends on old versions of libcurl3 as discussed here.
• Resolved by adding stable.list to sources.list.d, running apt-get update, and then running apt-get install libcurl3=7.38.0-4+deb8u5 libcurl4-openssl-dev=7.38.0-4+deb8u5

[ngollan]

Since this came up high in search results when encountering the same problem, I'm on Debian testing at the time of writing, and I had that segault triggered by the sheer presence of an old libssl1.0.0 package on my system. Removing that in favour of the libssl1.0.2 package let the dotnet tool run.

[lilred]

Google App Engine's "Your first app with .NET" tutorial tries to set you up with .NET Core but fails because of a segfault on dotnet restore. Not ideal. Debian 9.4 (Stretch), dotnet 2.0.0.

[Petermarcu]

@lilred are you using this one: https://cloud.google.com/appengine/docs/flexible/dotnet/quickstart

And you are at the step of running the app locally from the zip file on Debian 9.4?

[lilred]

I'm using the interactive tutorial that's in the google console, under the three dots menu -> "Try an interactive tutorial" -> "Start a Flexible Environment Guided Walkthrough" -> ".NET". It's also the same tutorial that shows up in the App Engine tab if you don't have a default instance deployed.

I would expect 100% of the people who try to run .NET Core on App Engine for the first time to run into this. It should not be very hard to reproduce, for the same reasons.