Opentelemetry-dotnet: Jaeger Exporter - UDP Packet size was 9886. Exception message: Message too long

[WorldIsM]

Describe your environment. Describe any aspect of your environment relevant to the problem:

• SDK version: all OTel packages 0.2.0-alpha.275
• .NET runtime version: netcoreapp3.1
• Platform and OS version: mac os

If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on master.

Steps to reproduce.

1. Strat app with JaegerExporter
2. Add large tags to spans

What is the expected behavior?
App running in any condition of Jaeger service

What is the actual behavior?
Application crashed.

Additional context.
Unhandled exception. OpenTelemetry.Exporter.Jaeger.Implementation.JaegerExporterException: Could not send 13 spans ---> Thrift.Transport.TTransportException: Cannot flush because of socket exception. UDP Packet size was 9886. Exception message: Message too long at OpenTelemetry.Exporter.Jaeger.Implementation.JaegerThriftClientTransport.FlushAsync(CancellationToken cancellationToken) at OpenTelemetry.Exporter.Jaeger.Implementation.EmitBatchArgs.WriteAsync(Int32 seqId, Byte[] processMessage, List1 spanMessages, TProtocol oprot, CancellationToken cancellationToken)
at OpenTelemetry.Exporter.Jaeger.Implementation.JaegerUdpBatcher.SendAsync(Dictionary2 batches, CancellationToken cancellationToken) --- End of inner exception stack trace --- at OpenTelemetry.Exporter.Jaeger.Implementation.JaegerUdpBatcher.SendAsync(Dictionary2 batches, CancellationToken cancellationToken)
at OpenTelemetry.Exporter.Jaeger.Implementation.JaegerUdpBatcher.FlushAsyncInternal(Boolean lockAlreadyHeld, CancellationToken cancellationToken)
at OpenTelemetry.Exporter.Jaeger.Implementation.JaegerUdpBatcher.<.ctor>b__12_0(Object sender, ElapsedEventArgs args)
at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
`

Data looks like:
"Attributes": [ { "Key": "component", "Value": "sql" }, { "Key": "db.statement", "Value": "some long sql...." ...

[MikeGoldsmith]

Thanks for the bug report @WorldIsM. We are currently trying to integrate some upcoming .NET5 diagnostic library changes into the core project. Once this has been completed we will have time to properly triage and investigate your bug report.

[kirillta]

Same on 288

[eddynaka]

@WorldIsM / @kirillta , do you know how long is the message that you are trying to record? Since we are using UDP, we have some size limitations, so, if you create a message near 65000, it will generate the exception.

[cijothomas]

@eddynaka We need to ensure we don't throw this exception. It should be caught and logged, at the minimum - Do you mind ensuring we do this already?

[eddynaka]

@cijothomas , in my tests, even passing a string greater than 66_000, it didn't crash, just ignored that message.

@WorldIsM / @kirillta , recently we released the beta version. Can you try with that and share a sample code with us?

Thank you!

[kirillta]

I will if you update OpenTelemetry.Instrumentation.Dependencies to the beta

[eddynaka]

@kirillta , the OpenTelemetry.Instrumentation.Dependencies got divided in: Instrumentation.Http, Instrumentation.SqlClient, Instrumentation.Grpc. Below the links:
https://www.nuget.org/packages/OpenTelemetry.Instrumentation.Grpc/0.4.0-beta.2
https://www.nuget.org/packages/OpenTelemetry.Instrumentation.Http/0.4.0-beta.2
https://www.nuget.org/packages/OpenTelemetry.Instrumentation.SqlClient/0.4.0-beta.2

[cijothomas]

@cijothomas , in my tests, even passing a string greater than 66_000, it didn't crash, just ignored that message.

Are we not logging to EventSource when this occurs?

[eddynaka]

@cijothomas , in my tests, even passing a string greater than 66_000, it didn't crash, just ignored that message.

Are we not logging to EventSource when this occurs?

No, we are just throwing the exceptions. I can do that. I will open a PR saving it to EventSource

[eddynaka]

@cijothomas , looking again at the code, we are saving it: https://github.com/open-telemetry/opentelemetry-dotnet/blob/755f727df00612b9d268cf18f7ac312f53f79d0a/src/OpenTelemetry.Exporter.Jaeger/Implementation/JaegerUdpBatcher.cs#L254-L283

[WorldIsM]

@WorldIsM / @kirillta , do you know how long is the message that you are trying to record? Since we are using UDP, we have some size limitations, so, if you create a message near 65000, it will generate the exception.

9216 bytes for Mac by default

[eddynaka]

@WorldIsM , were you able to retry with our latest beta and see if the issue occurs?

[kirillta]

803 bytes

[kirillta]

Running the betas for a month and catch no errors so far

[eddynaka]

@kirillta , thanks for let us know! :)

[cijothomas]

Closing old issue as this is likely already fixed.
Please re-open this or create a new issue if any issues are to be reported.