Apm-agent-dotnet: Elastic APM 1.7.0 has significantly increased performance impact

[kanadaj]

Describe the bug
After updating from 1.6.1 to 1.7.0, we have observed a very significant degradation in application performance and resources used:

Not entirely sure why or what exactly causes it but that's a _lot_ of overhead.

[kanadaj]

@gregkalapos This one essentially makes it impossible to update to 1.7.0

[kanadaj]

Packages used:
Elastic.Apm.AspNetCore 1.6.1 -> 1.7.0
Elastic.Apm.Elasticsearch 1.0.1-beta1 -> 1.7.0
Elastic.Apm.EntityFrameworkCore 1.6.1 -> 1.7.0

Setup:

// 1.6.1
app.UseElasticApm(Configuration, new HttpDiagnosticsSubscriber(), new AspNetCoreDiagnosticsSubscriber(), new EfCoreDiagnosticsSubscriber(), ElasticsearchDiagnosticsSubscriber()));

// 1.7.0
app.UseElasticApm(Configuration, new HttpDiagnosticsSubscriber(), new AspNetCoreErrorDiagnosticsSubscriber(), new EfCoreErrorDiagnosticsSubscriber(), ElasticsearchDiagnosticsSubscriber()));

[cbo100]

Commenting as another datapoint to backup this observation. Only local testing so far, but with no other change beyond:

• 1.6.1 -> 1.7.0

Seeing 50%-150% increase in response times (20ms avg -> 30-50ms avg)

[gregkalapos]

I looked into this, this is likely caused by our stack trace capturing code.

I opened #1052 to address this.

There are 2 things that could help me a lot:

• if someone could test the code from the PR and see if it helps it'd be super helpful. I benchmarked it, it looks good, but it's always useful to have more data. (NuGet files here)
• a CPU sample would be also useful. If someone could attach one here, it'd be great.

[kanadaj]

@gregkalapos I just pushed it to our NuGet repo and CI/CD, should have you some results in an hour or so.

[kanadaj]

@gregkalapos Updating from 1.6.1 to the alpha, transaction duration is untouched, CPU usage is up by a fair amount:

The CPU impact isn't as bad as before but still quite bad. These stacktraces are fat, aren't they? And many of them don't even include the application code even at 50 lines deep:

Elastic.Apm.Model.Span in .ctor
Elastic.Apm.Model.Span in StartSpanInternal
Elastic.Apm.Model.ExecutionSegmentCommon in StartSpanOnCurrentExecutionSegment
Elastic.Apm.EntityFrameworkCore.EfCoreDiagnosticListener in OnNext
System.Diagnostics.DiagnosticListener in Write
Microsoft.EntityFrameworkCore.Diagnostics.IDiagnosticsLogger in DispatchEventData
Microsoft.EntityFrameworkCore.Diagnostics.RelationalLoggerExtensions in BroadcastCommandExecuting
Microsoft.EntityFrameworkCore.Diagnostics.RelationalLoggerExtensions in CommandReaderExecutingAsync
Microsoft.EntityFrameworkCore.Storage.RelationalCommand+<ExecuteReaderAsync>d__17 in ExecuteReaderAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlRelationalConnection+<OpenAsync>d__14 in OpenAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenAsync>d__54 in OpenAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenInternalAsync>d__58 in OpenInternalAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.MySqlConnection+<OpenAsync>d__28 in OpenAsync at line 419
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.MySqlConnection+<CreateSessionAsync>d__118 in CreateSessionAsync at line 850
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.Core.ConnectionPool+<GetSessionAsync>d__10 in GetSessionAsync at line 142
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult

[kanadaj]

Okay yeah, reducing sample rate from 0.5 helps too, but it kind of sucks since ElasticApm still doesn't have adaptive sampling of any sort.

Some thoughts:

• Maybe we need a setting to disable stack traces completely in EF Core, since traces like above are pointless unless I increase the max
• For explicit usage of the tracing API, maybe have [CallerMemberNameAttribute] and [CallerLineNumber] as an explicit alternative?

[kanadaj]

Okay, so apparently even with https://github.com/elastic/apm-agent-dotnet/pull/948, EF Core stacks often look like this:

Elastic.Apm.Model.Span in .ctor
Elastic.Apm.Model.Transaction in StartSpanInternal
Elastic.Apm.Model.ExecutionSegmentCommon in StartSpanOnCurrentExecutionSegment
Elastic.Apm.EntityFrameworkCore.EfCoreDiagnosticListener in OnNext
System.Diagnostics.DiagnosticListener in Write
Microsoft.EntityFrameworkCore.Diagnostics.IDiagnosticsLogger in DispatchEventData
Microsoft.EntityFrameworkCore.Diagnostics.RelationalLoggerExtensions in BroadcastCommandExecuting
Microsoft.EntityFrameworkCore.Diagnostics.RelationalLoggerExtensions in CommandReaderExecutingAsync
Microsoft.EntityFrameworkCore.Storage.RelationalCommand+<ExecuteReaderAsync>d__17 in ExecuteReaderAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
Pomelo.EntityFrameworkCore.MySql.Storage.Internal.MySqlRelationalConnection+<OpenAsync>d__14 in OpenAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenAsync>d__54 in OpenAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenInternalAsync>d__58 in OpenInternalAsync
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.MySqlConnection+<OpenAsync>d__28 in OpenAsync at line 419
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.MySqlConnection+<CreateSessionAsync>d__118 in CreateSessionAsync at line 850
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.Core.ConnectionPool+<GetSessionAsync>d__10 in GetSessionAsync at line 142
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.Core.ServerSession+<TryResetConnectionAsync>d__73 in TryResetConnectionAsync at line 552
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1 in SetResult
MySqlConnector.Core.ServerSession+<ReceiveReplyAsyncAwaited>d__82 in ReceiveReplyAsyncAwaited at line 807
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.Protocol.Serialization.ProtocolUtility+<<DoReadPayloadAsync>g__AddContinuation|5_0>d in <DoReadPayloadAsync>g__AddContinuation|5_0 at line 483
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.Protocol.Serialization.ProtocolUtility+<<ReadPacketAsync>g__AddContinuation|1_0>d in <ReadPacketAsync>g__AddContinuation|1_0 at line 412
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.Protocol.Serialization.BufferedByteReader+<ReadBytesAsync>d__2 in ReadBytesAsync at line 55
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Threading.Tasks.AwaitTaskContinuation in RunOrScheduleAction
System.Threading.Tasks.Task in RunContinuations
System.Threading.Tasks.Task`1 in TrySetResult
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1 in SetExistingTaskResult
MySqlConnector.Protocol.Serialization.SocketByteHandler+<DoReadBytesAsync>d__8 in DoReadBytesAsync at line 106
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1 in MoveNext
System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs in InvokeContinuation
System.Net.Sockets.SocketAsyncContext+OperationQueue`1 in ProcessSyncEventOrGetAsyncEvent
System.Net.Sockets.SocketAsyncContext in HandleEvents
System.Net.Sockets.SocketAsyncEngine in System.Threading.IThreadPoolWorkItem.Execute
System.Threading.ThreadPoolWorkQueue in Dispatch

At this point I'm not sure it's worth the cost to even try to trace these. I feel like the request is already running directly on the middleware pipeline due to async/await shenanigans yielding back.

[tbutler-qontigo]

I looked into this, this is likely caused by our stack trace capturing code.

@gabriellandau Does this mean there should be no performance degradation if we disable stack tracing completely? So safe to go to 1.7.0 as long as we don't want this functionality?
Thanks

[gabriellandau]

@tbutler-qontigo Do you mean s/ @gabriellandau / @gregkalapos /?

[tbutler-qontigo]

@tbutler-qontigo Do you mean s/ @gabriellandau / @gregkalapos /?

Apologies, yes I meant @gregkalapos - too many people beginning with "g" ;)

[gregkalapos]

Does this mean there should be no performance degradation if we disable stack tracing completely? So safe to go to 1.7.0 as long as we don't want this functionality?
Thanks

@tbutler-qontigo unfortunately not in 1.7.0, because in that version we captured the stack trace always - if you disable it, the agent won't send it, but that doesn't disable the capturing itself. Which clearly isn't good and this is also fixed in https://github.com/elastic/apm-agent-dotnet/pull/1052 - so in the next version, the answer will be yes, but in 1.7.0 it's a no unfortunately.

[tbutler-qontigo]

@gregkalapos Ok thanks, I will stick with 1.6 for the moment then.
Next question of course is do you have a timescale for 1.7.1 which fixes this? (And adds TryGetLabel? ;) )
many thanks

[vxavictor513]

1.7.1 is out. Has anyone compared its performance with 1.6.1?

[ffMathy]

We are also experiencing degraded performance.

[ffMathy]

And that's with 1.7.1.

[gregkalapos]

Is it still the increased CPU usage? Does anyone have numbers? (there are CPU usage charts above, but honestly I can't really interpret those).

In 1.7.1 I'd not expect a change on the overall CPU and memory usage compared to 1.7.0, but the impact on transaction duration should be significantly less in 1.7.1 compared to 1.7.0.

Another thing: Did anyone try to turn off stack trace capturing in 1.7.1? For this set StackTraceLimit to 0 or SpanFramesMinDuration to 0ms. It'd be nice to know the impact of that in your environments. If there is no impact (so still higher CPU usage compared to 1.6), then we'll know it's not stack trace related - but I'd expect that turning off stack trace capturing will help with the CPU overhead (let's see if my theory is wrong or not...).

From my point of view here is how this issue looks:

• 1.7.1 should improve transaction duration
• I left this open to see what we can do with the CPU impact in a later release

[vhatsura]

At least what I see, the issue is in the garbage collection due to increased memory consumption in EnhancedStackFrame.
For 20 minutes session, Rider spotted a lot of allocations inside of EnhancedStackFrame constructor.

Small Object Heap: Allocated 161.4 MB (169267832 B) of type System.Int32[] by System.Diagnostics.EnhancedStackFrame..ctor(StackFrame, ResolvedMethod, String, int, int)

at System.Diagnostics.StackFrameHelper.InitializeSourceInfo(int, bool, Exception)
at System.Diagnostics.StackFrame.BuildStackFrame(int, bool)
at System.Diagnostics.StackFrame..ctor(String, int)
at System.Diagnostics.StackFrame..ctor(String, int, int)
at System.Diagnostics.EnhancedStackFrame..ctor(StackFrame, ResolvedMethod, String, int, int) in D:\GitHub\Ben.Demystifier\src\Ben.Demystifier\EnhancedStackFrame.cs:line 19 column 15
at System.Diagnostics.EnhancedStackTrace.GetFrames(StackTrace) in D:\GitHub\Ben.Demystifier\src\Ben.Demystifier\EnhancedStackTrace.Frames.cs:line 72 column 21
at System.Diagnostics.EnhancedStackTrace..ctor(StackTrace) in D:\GitHub\Ben.Demystifier\src\Ben.Demystifier\EnhancedStackTrace.cs:line 47 column 13
at Elastic.Apm.Filters.SpanStackTraceCapturingFilter.Filter(ISpan)
at Elastic.Apm.Report.PayloadSenderV2.<ProcessQueueItems>g__TryExecuteFilter|28_0(IEnumerable, )
at Elastic.Apm.Report.PayloadSenderV2+<ProcessQueueItems>d__28.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(ref )
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start(ref )
at Elastic.Apm.Report.PayloadSenderV2.ProcessQueueItems(Object[])
at Elastic.Apm.Report.PayloadSenderV2+<WorkLoopIteration>d__25.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.Threading.Tasks.VoidTaskResult,Elastic.Apm.Report.PayloadSenderV2+<WorkLoopIteration>d__25>.ExecutionContextCallback(Object)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.Threading.Tasks.VoidTaskResult,Elastic.Apm.Report.PayloadSenderV2+<WorkLoopIteration>d__25>.MoveNext(Thread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.Threading.Tasks.VoidTaskResult,Elastic.Apm.Report.PayloadSenderV2+<WorkLoopIteration>d__25>.MoveNext()
at System.Threading.Tasks.TaskSchedulerAwaitTaskContinuation+<>c.<Run>b__2_0(Object)
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref Task, Thread)
at System.Threading.Tasks.Task.ExecuteEntry()
at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task)
at Elastic.Apm.Helpers.SingleThreadTaskScheduler.TryExecuteTaskInline(Task, bool)
at System.Threading.Tasks.TaskScheduler.TryRunInline(Task, bool)
at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(Task, bool)
at System.Threading.Tasks.TaskSchedulerAwaitTaskContinuation.Run(Task, bool)
at System.Threading.Tasks.Task.RunContinuations(Object)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1<System.__Canon>.TrySetResult()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.__Canon>.SetExistingTaskResult(Task, )
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.__Canon>.SetResult()
at Elastic.Apm.Report.PayloadSenderV2+<ReceiveBatchAsync>d__26.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.__Canon,Elastic.Apm.Report.PayloadSenderV2+<ReceiveBatchAsync>d__26>.ExecutionContextCallback(Object)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.__Canon,Elastic.Apm.Report.PayloadSenderV2+<ReceiveBatchAsync>d__26>.MoveNext(Thread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.__Canon,Elastic.Apm.Report.PayloadSenderV2+<ReceiveBatchAsync>d__26>.MoveNext()
at System.Threading.Tasks.TaskSchedulerAwaitTaskContinuation+<>c.<Run>b__2_0(Object)
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref Task, Thread)
at System.Threading.Tasks.Task.ExecuteEntry()
at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task)
at Elastic.Apm.Helpers.SingleThreadTaskScheduler.TryExecuteTaskInline(Task, bool)
at System.Threading.Tasks.TaskScheduler.TryRunInline(Task, bool)
at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(Task, bool)
at System.Threading.Tasks.TaskSchedulerAwaitTaskContinuation.Run(Task, bool)
at System.Threading.Tasks.Task.RunContinuations(Object)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1<System.Boolean>.TrySetResult()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Boolean>.SetExistingTaskResult(Task, )
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<System.Boolean>.SetResult()
at Elastic.Apm.Report.PayloadSenderV2+<TryAwaitOrTimeout>d__27.MoveNext()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.Boolean,Elastic.Apm.Report.PayloadSenderV2+<TryAwaitOrTimeout>d__27>.ExecutionContextCallback(Object)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.Boolean,Elastic.Apm.Report.PayloadSenderV2+<TryAwaitOrTimeout>d__27>.MoveNext(Thread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1<System.Boolean,Elastic.Apm.Report.PayloadSenderV2+<TryAwaitOrTimeout>d__27>.MoveNext()
at System.Threading.Tasks.TaskSchedulerAwaitTaskContinuation+<>c.<Run>b__2_0(Object)
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref Task, Thread)
at System.Threading.Tasks.Task.ExecuteEntry()
at System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task)
at Elastic.Apm.Helpers.SingleThreadTaskScheduler.<ThreadMain>b__14_0()
at Elastic.Apm.Helpers.ExceptionUtils.DoSwallowingExceptions(IApmLogger, Action, bool, String)
at Elastic.Apm.Helpers.SingleThreadTaskScheduler.ThreadMain()
at System.Threading.ThreadHelper.ThreadStart_Context(Object)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object)
at System.Threading.ThreadHelper.ThreadStart()

Even calculation was moved to the dedicated thread, memory allocations still in place, which creates pressure for the garbage collector.

[gregkalapos]

Thanks @vhatsura,

yeah, that's also what I'd expect. So I still think, if this is an issue to people, disabling stack trace capturing should help as a workaround in 1.7.1 - and then we'll see what we can do in later versions.

[gregkalapos]

Closing this now.

I opened https://github.com/elastic/apm-agent-dotnet/issues/1099 to investigate the issue with EnhancedStackFrame.

Summary on the status of the overall issue can be found above.