Runtime: Self-Contained dotnet core 1.1 app crash on Darwin 17.0.0 (macOS High Sierra)

[TingluoHuang]

We have few customers reported that they can not configure the VSTS-Agent on macOS High Sierra. Application will crash on making http call with error Segmentation fault: 11

We have asked customer to collect app crash log, please check the following issue for detail.
https://github.com/Microsoft/vsts-agent/issues/1007

Is there anyway we can help customer unblock?

Thanks,
Ting

[ellismg]

Looks like https://github.com/dotnet/core-setup/issues/2649, @bartonjs @Petermarcu

FWIW, I think this is blocking us bringing up macOS High Sierra in our VSTS labs.

[bartonjs]

Yep, looks like the same OpenSSL+LibreSSL not playing nicely together (via libcurl).

I believe backporting https://github.com/dotnet/corefx/pull/19493 would fix this. (Unless libcrypto.35.dylib (LibreSSL) claims to be "openssl/1.0" for the cURL TLS provider string, in which case we'd need to just disable talking to OpenSSL altogether on High Sierra, which is essentially our 2.0 fix).

[gkhanna79]

CC @janvorli

[bartonjs]

Fixed by https://github.com/dotnet/corefx/pull/21069 and https://github.com/dotnet/corefx/pull/21071

[TingluoHuang]

@bartonjs do you know how can i consume the fix? do i have to wait for another dotnet core 1.1.x release?

[bartonjs]

@TingluoHuang If you want a guarantee of success you'll need to wait for (at least) the next official build. If you want to continue being a trailblazer you can

• Locate System.Net.Http.dll (presumably /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.2/System.Net.Http.dll)
• Save a backup copy
• Download https://ci.dot.net/job/dotnet_corefx/job/release_1.1.0/job/osx_release/lastSuccessfulBuild/artifact/bin/build.tar.gz
• tar -xzvf build.tar.gz bin/Unix.AnyCPU.Release/System.Net.Http/System.Net.Http.dll
• Copy the extracted file over top of the shared framework copy

That will move past the HttpClient problem, but I don't think that we've yet identified that this is the only problem on High Sierra... so if you go the cutting edge route please continue to let us know what you run into.

[TingluoHuang]

@bartonjs thanks, my customer is able to apply the fix himself and able to use the build agent run a build.

[knownasilya]

What version was this released for?

[knownasilya]

I'm using 1.1.4 and tried doing the copy to /usr/local/share/dotnet/shared/Microsoft.NETCore.App/1.1.4 but that didn't help.

[janvorli]

@knownasilya looking at the 1.1.4 release date, the issue is already fixed in that release.

[knownasilya]

Hum, maybe my issue is a bit different. I don't actually get a segfault, see my minidump https://gist.github.com/knownasilya/7249b7eae0f52a1559756905862794d7

Basically it crashes dotnet cli with no real errors and not triggering a try/catch. I'm doing a POST with HttpClient to carto.com on a HighSierra machine with LibreSSL 2.2.7

It was all working until last week, and I'm not aware of any related updates that I did explicitly. The same code is working on a coworkers Windows machine.

[bartonjs]

Somehow we cross-talked between OpenSSL (libcrypto.1.0.0.dylib) and LibreSSL (libcrypto.35.dylib):

Thread 41 Crashed:
0   libcrypto.35.dylib              0x00007fff67a40584 ASN1_STRING_cmp + 20
1   libcrypto.35.dylib              0x00007fff67a37f8d ASN1_OCTET_STRING_cmp + 29
2   libcrypto.35.dylib              0x00007fff67b80268 X509_check_akid + 88
3   libcrypto.35.dylib              0x00007fff67b80149 X509_check_issued + 121
4   libcrypto.35.dylib              0x00007fff67b6aaf1 check_issued + 33
5   libcrypto.1.0.0.dylib           0x000000010ab07cfd find_issuer + 67
6   libcrypto.1.0.0.dylib           0x000000010ab06f6e X509_verify_cert + 561
7   ???                             0x000000010ed636d5 0 + 4543887061
8   ???                             0x000000010ed62db8 0 + 4543884728
9   ???                             0x000000010ed616fe 0 + 4543878910
10  libcoreclr.dylib                0x00000001054228b7 UMThunkStub + 273
11 libssl.35.dylib 0x00007fff690e6207 ssl_verify_cert_chain + 471

That's not something we should have done.

@knownasilya What do you get for curl --version?

$ curl --version
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

(My example is from Ubuntu 16.04, so yours should definitely not match mine)

[knownasilya]

$ curl --version
curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSLlibz HTTP2 UnixSockets HTTPS-proxy

[bartonjs]

That definitely doesn't include the string "openssl/1.0", so we shouldn't have registered the callback that fired.

It sounds like, you're somehow using an older copy of System.Net.Http.dll.

If you using System.Reflection; you can do something like

C# Console.WriteLine(typeof(System.Net.Http.HttpClient).GetTypeInfo().Assembly.CodeBase);

to find out where HttpClient came from.

[knownasilya]

file:///Users/iradchenko/.nuget/packages/System.Net.Http/4.3.1/runtimes/unix/lib/netstandard1.6/System.Net.Http.dll

[bartonjs]

@knownasilya Yay, sanity has been attained. The fix is in package version 4.3.3.

[knownasilya]

@bartonjs how would I update that?

[knownasilya]

@bartonjs I did dotnet add package System.Net.Http -v 4.3.3 in that project, but it's still using the global one from above.

[knownasilya]

For anyone having the same issue, I followed this answer https://stackoverflow.com/a/43687514/483616 (just don't put . in the path because that won't work cross platform)

[terrajobst]

Phew. Glad it's not us :-)

[knownasilya]

@terrajobst so I had to revert that because it breaks the app when using razor. Error is attached.

The module that it can't find changes based on machine or os, not sure. But when I reverted back to the old Http package, it worked. This error only seems to happen when using http together with razor as far as I can tell.

[terrajobst]

Unfortunately I have no idea how the Razor assembly resolver works.

@davidfowl, could you please take a look?

[knownasilya]

Would be up to pairing on the issue over vscode live share/hangouts (or whatever) if that would help.