Runtime: dotnet restore segfaults on Debian 9.0/stretch Testing

[btrzcinski]

The standard, default project.json files cause 'dotnet restore' to segfault on Debian Testing.

barnett@barnett-debian:~/csharp_proj$ dotnet new
Created new C# project in /home/barnett/csharp_proj.
barnett@barnett-debian:~/csharp_proj$ dotnet restore
log  : Restoring packages for /home/barnett/csharp_proj/project.json...
Segmentation fault

.NET Command Line Tools (1.0.0-preview2-002900)

Product Information:
 Version:            1.0.0-preview2-002900
 Commit SHA-1 hash:  f4ceb1f213

Runtime Environment:
 OS Name:     debian
 OS Version:  
 OS Platform: Linux
 RID:         debian.-x64

barnett@barnett-debian:~$ uname -a
Linux barnett-debian 4.5.0-2-amd64 dotnet/corefx#1 SMP Debian 4.5.4-1 (2016-05-16) x86_64 GNU/Linux

csharp_proj.zip

[augusteiner]

also hapenning on my tests here

[ellismg]

Is this Jessie or some other version?

[ellismg]

I just realized that "testing" in the title means you're talking about "stretch" instead of a general "I'm testing stuff on Debian".

We haven't run on Debian testing, I expect there will be a few issues we'll have to look into, mainly around the fact that the native libraries binaries we ship may not be compatible out of the box with what's in Stretch. For 1.0 we are just targeting Jessie, but we hope to document the process of bootstrapping the CLI from source for other distros within the next week few weeks.

[kolbma]

I get the Segmentation faults in the official latest Docker image during a dotnet restore.
The image is based on Debian 8 jessie.

[fogzot]

I did some tests and the official Docker images (jessie-based) work only if the kernel is old enough (3.16) but segfault if run on a system with a new kernel (4.5 or 4.6 for example): docker isolates the user space, true, but runs on the same kernel of the host. The same is true for the official jessie packages on a jessie machine: "dotnet new" (when populating the initial cache) and "dotnet restore" segfault on newer kernels.

[nmschulte]

I've been experiencing this bug for a long time now too, using Debian Sid ("unstable"). @karelz, how can I help resolve the issue? For now, I'm developing inside of a virtual machine, but that is exactly what I am trying to get away from and looking at .NET Core to begin with...

[karelz]

I would get it under debugger / collect dump and analyze it. Do you need guidance for that or do you know how to do it yourself?

@ellismg @mellinoe can you please help / point in the right direction?

[nmschulte]

core.30619.zip

nmschulte@desmas-l:~/src/shared-todo-dotnet$ dotnet --version
1.0.0-preview3-003842

nmschulte@desmas-l:~/src/shared-todo-dotnet$ dotnet

Microsoft .NET Core Shared Framework Host

  Version  : 1.0.1
  Build    : cee57bf6c981237d80aa1631cfe83cb9ba329f12

Usage: dotnet [common-options] [[options] path-to-application]

Common Options:
  --help                           Display .NET Core Shared Framework Host help.
  --version                        Display .NET Core Shared Framework Host version.

Options:
  --fx-version <version>           Version of the installed Shared Framework to use to run the application.
  --additionalprobingpath <path>   Path containing probing policy and assemblies to probe for.

Path to Application:
  The path to a .NET Core managed application, dll or exe file to execute.

If you are debugging the Shared Framework Host, set 'COREHOST_TRACE' to '1' in your environment.

To get started on developing applications for .NET Core, install .NET SDK from:
  http://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409

nmschulte@desmas-l:~/src/shared-todo-dotnet$ gdb --args dotnet restore
GNU gdb (Debian 7.11.1-2) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from dotnet...(no debugging symbols found)...done.
(gdb) run
Starting program: /home/nmschulte/.local/bin/dotnet restore
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff54f8700 (LWP 30623)]
[New Thread 0x7ffff4cf7700 (LWP 30624)]
[New Thread 0x7ffff44f6700 (LWP 30625)]
[New Thread 0x7ffff397c700 (LWP 30626)]
[New Thread 0x7fffefff4700 (LWP 30627)]
[New Thread 0x7fffef7f3700 (LWP 30628)]
[New Thread 0x7fffeeff2700 (LWP 30629)]
[New Thread 0x7fffee7b1700 (LWP 30631)]
[New Thread 0x7fff57ffd700 (LWP 30632)]
[New Thread 0x7fff577fc700 (LWP 30633)]
[New Thread 0x7fff3ec19700 (LWP 30638)]
[New Thread 0x7fff3e418700 (LWP 30639)]
[Thread 0x7fff3e418700 (LWP 30639) exited]
log  : Restoring packages for /home/nmschulte/src/shared-todo-dotnet/project.json...

Thread 12 "dotnet" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff3ec19700 (LWP 30638)]
0x00007ffff03cfddd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(gdb) log  : Writing lock file to disk. Path: /home/nmschulte/src/shared-todo-dotnet/project.lock.json
log  : Restore completed in 324ms for /home/nmschulte/src/shared-todo-dotnet/project.json.
gcore
warning: target file /proc/30619/cmdline contained unexpected null characters
Saved corefile core.30619
(gdb) quit
A debugging session is active.

    Inferior 1 [process 30619] will be killed.

Quit anyway? (y or n) y
nmschulte@desmas-l:~/src/shared-todo-dotnet$ zip core.30619.zip core.30619
  adding: core.30619 (deflated 100%)

[fogzot]

Exactly same stack trace here. It is possible to include the debug symbols in the .deb packages or in the tarball? That would produce a much better trace.

[nmschulte]

I was unable to get dotnet to work in Debian 8 Jessie/stable, Debian 9 Stretch/testing, Debian Sid/unstable, or Ubuntu 14.04.5 LTS Trusty (Kernels 4.4 and 3.19). I can capture stack traces and dumps from each, just let me know. I've been waiting for these segfaults to be dealt with for many months now; it seems I just keeping hitting one after another after another; the CoreCLR Kernel 4.6 issue is one, and this is another (or multiple others).

[karelz]

We definitely need callstack with symbols. The raw stack numbers are in general not helpful.
Do you know how to get them or do you need guidance?

[nmschulte]

I don't know how; wouldn't GDB dump them if they were available? If you're asking me to compile things w/ the symbols ("debugging enabled"), I'll need help; I can't even get the built versions of things work, so that's probably not going to happen. If it's an option I can pass to dotnet (.NET CLI), that's simple enough.

[karelz]

@ellismg who can help here?

[nmschulte]

Would it make sense to e.g. change the latest packages to be Debug builds, instead of Release builds like the release candidates and official releases use? Folks using latest are likely to be wanting bleeding edge, broken/WIP features anyway; that just seems logical to me, but I'm curious what the community thinks.

To be honest, it's rather confusing not being able to see what versions of packages are available on MyGet, or which feeds/channels are obsolete and which type of build each one is (and even, what each one is, sometimes); given that both the tooling and run-time are in development, and that the tooling is built on the same run-time, new users are easily confused! When things like omnisharp-roslyn include their _own_ build environment setup scripts, it gets even worse, :).

I appreciate the help; I'm very excited to play with the awesome things everyone's been working so hard to get out the door.

[nmschulte]

I'm not sure why this bug is in the corefx repository, apologies for keeping it alive if I shouldn't be.

When using the preview release, as opposed to the stable release, things work fine on Debian 8 Jessie/stable (amd64; Linux 3.16.0).

dotnet new && dotnet restore && dotnet run works great on Debian 8 Jessie/stable and Debian Sid/unstable.
~/src/omnisharp-roslyn$ dotnet restore works great on Debian 8 Jessie/stable, but crashes on Debian Sid/unstable.

Crash logs and dumps from Debian Sid -- dotnet crashes in a few different ways, but it seems the libcrypt/x509_verify_cert calls are the culprit. This jives with the issues I was having with NuGet.exe and the omnisharp-roslyn build script (./build.sh), where it seemed like NuGet was having issues with SSL certificates; see this NuGet output (v3.3.0) and this NuGet output(v3.4.4). NuGet.exe is built with .NET Core, correct?

debian.sid.crash.gdb.log.txt
core.27307.zip
more.gdb.runs.log.txt

[ellismg]

@nmschulte Just to be clear, the contents of ~/src/omnisharp-roslyn is just a clone of https://github.com/OmniSharp/omnisharp-roslyn, correct?

[nmschulte]

@ellismg precisely;

nmschulte@desmas-l:~/src/omnisharp-roslyn$ git show HEAD
commit 071c945f7267f2ff83657279440629bdadc2d021
Merge: c1dbcfb 9b73adc
Author: David Driscoll <[email protected]>
Date:   Sun Oct 9 21:29:03 2016 -0400

    Merge pull request dotnet/runtime#14065 from bstockus/dev

    Added Roslyn CSharp code formatting options to omnisharp.json.

The omnisharp-roslyn provided build script (./build.sh) attempts to setup a .NET CLI/Core environment, but I haven't run those in this case. Running the script fails, using the default tooling of dotnet-dev-debian-x64.1.0.0-preview2-003131.tar.gz, or "preview"/"Latest" (dotnet-dev-debian-x64.1.0.0-preview2.1-003153.tar.gz), which is why I'm here.

[fogzot]

@nmschulte dotnet runs fine on Jessie as long as you use the 3.16 kernel.Switch to a new one and you get the same libcrypt-related bug.

[nmschulte]

That's the bug I assume this story is here to address, though, right? There are no v3 kernels packaged for Debian Sid or Debian Testing (next stable; freezes 2017-02-05). Booting a custom built kernel isn't something I want to have to support (myself individually, or as part of the community) in order to use .NET Core.

[fogzot]

@nmschulte Yes. Sorry, I didn't explain myself too well. I just wanted to emphasize that everything runs OK on Jessie _as long as you don't upgrade the kernel_, else everything breaks on Jessie too.

[nmschulte]

@ellismg who can help here?

@karelz; we're looking for a stack trace w/ debug symbols yet? I've been out of the loop re: .NET Core for a few weeks now; perhaps I should try w/ newer releases (are there any?)?

[karelz]

Yes, there have been releases since May: https://www.microsoft.com/net/download/core

[janvorli]

This is a known issue that was fixed on June 26 2016. See https://github.com/dotnet/coreclr/issues/6016 and https://github.com/dotnet/coreclr/pull/6027

[nmschulte]

I did my testing well after May and June; so while the issues are resolved in the src, at the time (and maybe still now), there's no way to use it on these systems as there was (is?) no release (CoreCLR 1.0.4 has the fix) w/ it available.

So many moving parts w/ disparate versionings, all brand new (v1.0.0/preview1...), it's confusing and easy to get lost. A webpage showing the dependencies (as a graph or nested structure) w/ current version numbers/names might go a long way (for the various releases at least).

I guess this issue can be closed, though. Thank you for taking the time!

[janvorli]

@nmschulte the tarballs available at https://github.com/dotnet/cli all contain that fix. The packages that you can download from http://dot.net site also have that fix.

[karelz]

Closing as dupe of the issues mentioned above: https://github.com/dotnet/corefx/issues/8951#issuecomment-267572781
If you have evidence the problem was not addressed, feel free to reopen.

[Yutsa]

I am using Debian 9 (Stretch) and I have the same issue of dotnet segfaulting when using dotnet restore or dotnet run. I donwloaded the latest tarball here.

$ uname -a
Linux edouard 4.9.0-2-amd64 dotnet/corefx#1 SMP Debian 4.9.18-1 (2017-03-30) x86_64 GNU/Linux

[karelz]

@Yutsa can you please file a new issue and tag me & @janvorli there? It is likely different issue.
Please provide links to exact bits you installed and mention all steps you did to help us troubleshoot it, thanks!

[fogzot]

Isn't this the usual problem with libcurl? Just downgrade curl and libcurl3 to version 7.38.0-4+deb8u5 and the last tarball works correctly on Stretch.

[Yutsa]

@fogzot Oh it actually worked ! I used Jessie's version of libcurl 3 and I managed to create, restore and run the app !

I don't really understand why but I won't complain !

Any way to make a wiki or something to explain all steps needed to be able to use .NET Core on Debian Stretch ? I had to look several issues to know I had to download several packages and so on.

Thanks !

EDIT : I created a markdown file here to explain the step to follow to make it work on Stretch. I tried on a brand new Stretch VM and it works great.

[karelz]

BTW: I linked your workaround above from OS Supported Versions

If you want to move your how-to to CoreFX wiki, you can - it is writeable for everyone. We can figure out how to incorporate it into other the docs.
But maybe the steps should be rather in dotnet/core repo. We could just link them from CoreFX contributor guide if necessary. Thoughts?

[MarkusHaslinger]

Sorry to revive this, but I couldn’t find anything and am not linux savvy enough to know if this has been fixed by now or if the workaround is still required since Debian 9 is officially released?

[mellinoe]

@marce155 Are you able to try out .NET Core 2.0? It should work without a workaround, I believe.

[fogzot]

Unfortunately no, it still doesn't work. I just tried .NET Core 2.0 preview 2 with curl/libcurl 7.52.1-5 from Debian stable and dotnet new just segfaults. Rolling back to 7.38.0-4+deb8u5 ("oldstable") makes it work.

[fogzot]

Some additional information. Some parts of dotnet explicitly link with libcurl and libssl1.0.0. In Debian 9 libcurl links to libssl1.0.2 so dotnet ends loading both libraries at the same time. I guess the problem would go away if MS release a build of dotnet that uses the same SSL as libcurl, i.e., 1.0.0 for Debian 8 and 1.0.2 for Debian 9.

[MarkusHaslinger]

@mellinoe sorry for the delayed response. Sadly, I can only confirm the statement from @fogzot it didn’t work for me either with preview 2 and current Debian stable.

[JustArchi]

Yet another bump to this, preview2 doesn't work OOB and depends on oldstable libcurl. Can we do anything with that? Stretch went stable a while ago and not supporting it without workarounds feel wrong.

[mellinoe]

Could you post a bit more information about your system set-up? I created a Debian 9.0 docker container and was able to run with the latest "Linux x64" bits from https://github.com/dotnet/cli. I have this version of curl and ssl installed:

libcurl4-openssl-dev/stable,now 7.52.1-5 amd64 [installed]
libssl1.0.2/stable,now 1.0.2l-2 amd64 [installed,automatic]
libssl1.1/stable,now 1.1.0f-3 amd64 [installed,automatic]
openssl/stable,now 1.1.0f-3 amd64 [installed,automatic]

I'm able to run dotnet new, dotnet restore, and dotnet run successfully.

I'm using a docker container, so obviously this environment might not match what you guys are using. I can set up a real VM, but that will take a bit more time.

[mellinoe]

@danmosemsft @karelz We might want to re-open this and double-check that things are working as expected. I think it was closed previously because a workaround existed, but we should support this scenario by default with 2.0.

[JustArchi]

dotnet new, dotnet restore and dotnet run indeed crashed previously and no longer do, this is strange, I tried to reproduce segfault with HttpClient but that didn't work either, so I coded my own reproducible case based on libraries I use. Sorry for bringing it in, but at least it should be reproducible nicely now until you find out the real cause.

Correct output:

root@archi:/tmp/test# dotnet restore
  Restoring packages for /tmp/test/test.csproj...
  Generating MSBuild file /tmp/test/obj/test.csproj.nuget.g.props.
  Generating MSBuild file /tmp/test/obj/test.csproj.nuget.g.targets.
  Restore completed in 15.05 sec for /tmp/test/test.csproj.
root@archi:/tmp/test# dotnet run
OK

Bad output:

root@archi:/tmp/test# dotnet restore
  Restoring packages for /tmp/test/test.csproj...
  Generating MSBuild file /tmp/test/obj/test.csproj.nuget.g.props.
  Generating MSBuild file /tmp/test/obj/test.csproj.nuget.g.targets.
  Restore completed in 6.24 sec for /tmp/test/test.csproj.
root@archi:/tmp/test# dotnet run
root@archi:/tmp/test

In dmesg of bad output:

[324262.032879] traps: dotnet[27464] general protection ip:7f23e3fa8d6d sp:7f1f0bbdfbf0 error:0
[324262.032895]  in libcrypto.so.1.0.0[7f23e3e6d000+1cd000]

DotnetReproduce8951.zip

libcurl3:
  Installed: 7.38.0-4+deb8u5
  Candidate: 7.52.1-5

Installed is oldstable version that works properly. Candidate is the version that doesn't work and causes segfault.

.NET Command Line Tools (2.0.0-preview2-006497)

Product Information:
 Version:            2.0.0-preview2-006497
 Commit SHA-1 hash:  06a2093335

Runtime Environment:
 OS Name:     debian
 OS Version:
 OS Platform: Linux
 RID:         debian-x64
 Base Path:   /opt/dotnet/sdk/2.0.0-preview2-006497/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0-preview2-25407-01
  Build    : 40c565230930ead58a50719c0ec799df77bddee9

[mellinoe]

Hm. Your repro works for me (it prints out "OK" and then exits), and I have this installed:

libcurl3/stable,now 7.52.1-5 amd64 [installed,automatic]

You are saying that when you update it to my version, dotnet starts crashing? I wonder if there is some other package installed that is causing troubles.

[mellinoe]

It may also help narrow things down if you try the latest version from the master branch of CLI, as well as the release/2.0.0 branch.

master
release/2.0.0

[fogzot]

I just did some extensive tests using a clean Debian 9 docker image. On a clean install, adding curl and libcurl 7.52.1-5 is not a problem, everything works well. The problem pops up when upgrading an old Debian to version 9 or when using the Ubuntu .deb packages on Debian 9. In both cases libssl1.0.0 gets pulled in and BOOM, segfault. Note that this is NOT libssl1.0.2, the Debian 9 default but an older library with a different soname. Apparently dotnet tries to load _any_ available libssl at runtime and when both libcurl 7.52.1-5 (that pulls libssl1.0.2) and libssl1.0.0 are present it chooses to load the older one resulting in a segfault.

TL;DR: don't use Ubuntu packages on Debian 9 and remove libssl1.0.0 after upgrading from an older Debian to the latest stable.

[JustArchi]

What @fogzot explained above very likely is the correct culprit, since I couldn't reproduce the bug on clean Debian 9 either, while my faulty machine had these:

un  libssl-dev
ii  libssl1.0-dev:amd64
ii  libssl1.0.0:amd64
ii  libssl1.0.2:amd64
ii  libssl1.1:amd64

I removed libssl1.0.0 via apt-get purge, it seems that nothing depended on it apart from old libcurl3, so purge automatically triggered libcurl3 update as well. The library wasn't caught by my usual apt-get autoremove for some reason, I guess libraries are skipped from autoremove just in case something depended on them.

So yes, like fogzot said above:

Apparently dotnet tries to load any available libssl at runtime and when both libcurl 7.52.1-5 (that pulls libssl1.0.2) and libssl1.0.0 are present it chooses to load the older one resulting in a segfault.

This is correct. Which also means that you can easily reproduce my segfault if you do apt-get install libssl1.0.0 on Debian 9 - you will need to add oldstable repo to your /etc/apt/sources.list for doing so, as it isn't even available in stretch repo right away (and that's good, it won't be possible to reproduce it easily on clean Stretch image).

Whether this is worth it to investigate and fix dotnet to try to load newest libssl in case of multiple versions available, or just as a note in 2.0.0 upgrade, I'm happy with either, because finally we found the real culprit and the library is already obsolete in Stretch so there is no way something can depend on it on clean system. Just remove libssl1.0.0 if you're updating from Jessie, that's the correct fix.

[fogzot]

I have some more information.

Looking at a standard install of 1.0.x + 1.1.x runtime and 2.0 preview 2 runtime+sdk I can find only 3 .so objects that depend on libssl:

shared/Microsoft.NETCore.App/1.0.5/System.Security.Cryptography.Native.so on 1.0.0
shared/Microsoft.NETCore.App/1.1.2/System.Security.Cryptography.Native.OpenSsl.so on 1.0.0
shared/Microsoft.NETCore.App/2.0.0-preview2-25407-01/System.Net.Http.Native.so on 1.0.0 for Debian 8
shared/Microsoft.NETCore.App/2.0.0-preview2-25407-01/System.Net.Http.Native.so on 1.0.2 for Debian 9

Note that the dependencies where found using ldd but objdump doesn't show any explicit dependency on libssl. I suppose there is some kind of runtime check and that's why 2.0 preview 2 shows different dependencies on different Debian versions. (My fault. ldd just shows the full dependency tree and obviously the tree is different on Debian 8 and 9.)

This means that older frameworks won't run on Debian 9 because of missing shared libraries (is this a problem?) Trying to compile and run a netcoreapp1.1 console application on Debian 9 yields:

$ dotnet run -f netcoreapp1.1
Failed to initialize CoreCLR, HRESULT: 0x80131500

Here's a little summary:

.NET Core 1.0.x

• Runs on Debian 8
• Doesn't run on Debian 9 unless libssl1.0.0 is installed from oldstable (but this breaks 2.0 preview 2)

.NET Core 1.1.x

• Runs on Debian 8
• Doesn't run on Debian 9 unless libssl1.0.0 is installed from oldstable (but this breaks 2.0 preview 2)

.NET Core 2.0 preview 2

• Runs on Debian 8
• Runs on Debian 9, when installed from scratch
• Runs on Debian 9 when upgraded from Debian 8 only if libssl1.0.0 is removed from the system

[benscobie]

I've got GitLab installed on my Debian 9 server and dotnet new is segfaulting because it is somehow picking up libcrypto.so.1.0.0 in /opt/gitlab/embedded/lib/. If I rename this it no longer segfaults. Does anybody have any idea how dotnet is picking up this library and whether I can make it ignore it?

[karelz]

@janvorli @bartonjs any insights here?

[fogzot]

@benscobie check /etc/ld.so.conf and you environment for LD_LIBRARY_PATH and LD_PRELOAD. I don't see any other way for a console app to pickup a library in /opt.

[janvorli]

The Debian 9 uses libssl.so.1.0.2 and libcrypto.so.1.0.2 instead of the 1.0.0 versions. When I have added support for the 1.0.2 version, I have not anticipated issues when both versions are installed. But obviously the presence of the 1.0.0 version is causing problems on Debian 9. .NET Core tries to load the version 1.0.0 first and only if the OS doesn't find it, it tries to load the 1.0.2. As @fogzot has mentioned, you probably have the /opt/gitlab/embedded/lib in one of the three places and the system looks into that location first.
Now the question is if you have set it that way. Or if a gitlab installation did that automatically, which would really be intrusive.

[benscobie]

In desperation, I had stupidly symlinked libssl.so.1.0.0 in /lib/x86_64-linux-gnu/ to /opt/gitlab/embedded/lib/libssl.so.1.0.0 when I was originally fighting this issue. I've removed that and problem solved!

[roller]

I had this issue on debian stretch, gdb showed a segfault in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0.

dpkg -S /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 showed that libssl1.0.0 was the installed package. Apparently this is due to not removing obsolete packages such as php5-cli and postgresql-9.4.

After sudo apt remove libssl1.0.0 (and the obsolete packages) it works!

[simonz130]

I understand that debian 9.4 (stretch) is not supported.
If so, can you update this page?
https://www.microsoft.com/net/download/linux-package-manager/debian9/runtime-2.0.6

[danmosemsft]

@leecow for that