Opentelemetry-dotnet: AddOtlpExporter seems to be causing a crash?

[atrauzzi]

Bug Report

        <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc1.1" />
        <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc1.1" />
        <PackageReference Include="OpenTelemetry" Version="1.0.0-rc1.1" />
        <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.0.0-rc1.1" />

Running on net5.0

Symptom

It seems like when I use the following in my ASP.net Core project:

            services.AddOpenTelemetryTracing((builder) =>
            {
                builder
                    .AddSource(projectName)
                    .SetSampler(new AlwaysOnSampler())
                    .AddAspNetCoreInstrumentation()
                    .AddOtlpExporter();
            });

I get a segfault with the message: Process finished with exit code 139.

Note: If I remove AddOtlpExporter() everything boots fine.

What is the expected behavior?
Normal application bootup.

What is the actual behavior?
The segfault.

[cijothomas]

Can you share the runtime information as well? which OS, whether collector was running etc. I couldn't repo in my windows environment. I did not have OT Collector running, so exporter keeps hitting exception, but its handled, and no app crash..

[atrauzzi]

Running on linux, running on .net 5 whether I'm using an ASP.net host or a worker host, collector is running as a container:

telemetry          /otelcontribcol --config=/ ...   Up      55679/tcp, 0.0.0.0:55680->55680/tcp, 0.0.0.0:9411->9411/tcp

I've had this working prior to upgrading to the latest RC version.

---

It's worth also noting that the failure isn't _after_ the application starts. The failure occurs during service registration before the host is even started. So I wouldn't necessarily suspect any connectivity issues.

[cijothomas]

Ok i couldn't repro in windows.
Will try Linux.

[atrauzzi]

Sounds good, thanks for looking into it. Also, I'm running on Ubuntu using the latest snap package.

I'm trying to think of any other variances, but this is happening so early in the application boot and without any way of debugging it as the app just crashes out... :sweat_smile:

[cijothomas]

https://github.com/open-telemetry/opentelemetry-dotnet/blob/master/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/OtlpExporter.cs#L49 - There is no try..catch. We can try to catch the exception and see if it helps?

[atrauzzi]

If there was an exception, wouldn't I be seeing it instead of a hard-crash?

[cijothomas]

mm. Let me try if this is repro ing. and suggest next steps.

[cijothomas]

If there was an exception, wouldn't I be seeing it instead of a hard-crash?

Right. it shouldn't have crashed.

[edubsi]

Sadly we just ran into the same problem. Added unhandled Task & App Exception Handling to maybe catch & log the exceptions which did not help.
For us it looks like the crash happens when the first trace/log/metric is exported.

[alanwest]

@atrauzzi are you running your application in an alpine linux container? If so, I suspect this may be the problem you're running into #1251.

@cijothomas it does not seem that the work in #1634 prevents the crash in #1251... not sure we can catch this exception.

[atrauzzi]

@alanwest - Nope, bare on an ubuntu/pop!_os machine.

I'm not getting an exception at all. My application just hard-crashes with the one line in my original description. No stacktrace, nothing to catch.

[alanwest]

Ah, yes, sorry I had missed your previous statement regarding Ubuntu. I'm able to reproduce what you're seeing. Still not yet sure why, but I can confirm that #1634 does not resolve this issue.

[cijothomas]

@alanwest Do you know if this issue is fixed with daily build containing your PR for using diff grpc client lib for netstd21?

[alanwest]

Do you know if this issue is fixed with daily build containing your PR for using diff grpc client lib for netstd21?

My hope is yes based on the testing that I've done. @atrauzzi would you be able to validate using the daily build in your environment? The change in #1662 may help assuming your application is .NET Core 3.0 or higher.

Daily builds are available from this MyGet source https://www.myget.org/F/opentelemetry/api/v3/index.json. You should be able to use dotnet new nugetconfig to generate a nuget.config file in the root of your project, and then add the following to it:

<add key="otel" value="https://www.myget.org/F/opentelemetry/api/v3/index.json" />

[atrauzzi]

I might be able to get a look in a couple of weeks, just wrestling with a deadline at the moment, that alright?

[alanwest]

I've validated that this is resolved for .NET Core 3.0 and higher apps running on Ubuntu.