Runtime: ActivitySource performance issue

[ENikS]

Description

I am working on instrumenting Unity Container (a small dependency injection container) to provide runtime telemetry. I've read and searched for best practices and based on recommendations from @tarekgh chose ActivitySource pattern as most promising.

Unfortunately adding diagnostic pattern to methods slows them dramatically, best case scenario vent from 16 ns to 45 ns just by adding constructor for Activity and simple IsEnabled(...). In case of Unity, even time for allocating 56 vs 28 byte structs matter and adds overhead.

I understand these are costs associated with tracing but does is have to be so high?

I would like to propose couple of optimizations:

Add ValueActivity

This is a long shot and probably very involved to implement, but after ValueTask has been created, I think anything is possible. With support for ref returns and C# 8.x - 9.0, Activity.Current does not have to be an instance, it could be a struct. Imagine how much faster it could be!

Optimize ActivitySource

ActivitySource is very useful type and makes life easier but it costs even more to use due to two allocations it makes in Create.... Perhaps these allocations could be avoided if data is sent as in structures instead of class instances?

Theoretically ActivitySource could be used as Activity pool.

Every instance of Activity has Source reference. When disposed, instead of being discarded, it could notify parent Source and be reabsorbed into Activity pool. It could be reinitialized or even could keep some of the data for next life, when appropriate.

[msftbot[bot]]

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti
See info in area-owners.md if you want to be subscribed.

[tarekgh]

CC @noahfalk

Add ValueActivity

I don't think this will help much as because the Activity has to travel through the asynclocal context which will get boxed at that time.

use due to two allocations it makes in Create.... Perhaps these allocations could be avoided if data is sent as in structures instead of class instances?

Could you tell more what you mean the allocation in Create? and what instances you are talking about here?

Every instance of Activity has Source reference. When disposed, instead of being discarded, it could notify parent Source and be reabsorbed into Activity pool. It could be reinitialized or even could keep some of the data for next life, when appropriate.

This will need life time management for the Activity objects to decide when you can free such objects and add policy to such pool. it is interesting idea though.

[ENikS]

I was referring to these two allocations:

```c#
var aco = new ActivityCreationOptions(this, name, parentId, kind, tags, links);
var acoContext = new ActivityCreationOptions(this, name, aco.GetContext(), kind, tags, links);

> I don't think this will help much as because the Activity has to travel through the asynclocal context which will get boxed at that time.

I am not suggesting to replace Activity with `ValueActivity`, I am suggesting adding it, same as `ValueTask` did not replace `Task` type. Given samples from [this guidance](https://github.com/Microsoft/ApplicationInsights-dotnet-server/blob/b3a1d3ade8d5d1ccf8d1b0b922764a97b88ab96c/LibraryInstrumentationGuidance.md), this is the common pattern of instrumenting [the code](https://github.com/Azure/azure-service-bus-dotnet/blob/1607601d3d3c6cf7e0c2a757ccb398d803482e7c/src/Microsoft.Azure.ServiceBus/ServiceBusDiagnosticsSource.cs#L628):

```C#
            Activity activity = null;
            ... 
            if (DiagnosticListener.IsEnabled()
            {
                activity = new Activity(activityName);

                if (DiagnosticListener.IsEnabled(activityName))
                {
                    DiagnosticListener.StartActivity(activity, payload);
                }
                else
                {
                    activity.Start();
                }
            }

Activity is always created to preserve hierarchy, if at all enabled. If a struct could be used instead of instance the performance should be much better.

If it has to go through async context, it could be boxed later, but it would be once in a while, compared to every single time it is allocated now.

This is the pattern throughout entire implementation. Imagine how much overhead and GC pressure could be avoided.

[tarekgh]

I was referring to these two allocations:

I am wondering why using hierarchical Ids in the activity? just curious to know your scenario. This will limit the capabilities to have the activity context transfered or collected by telemetry collectors (e.g. appinsights or similar)

I am not suggesting to replace Activity with ValueActivity, I am suggesting adding it, same as ValueTask did not replace Task type.

Yes that is what I understood in the first place.

if it has to go through async context, it could be boxed later, but it would be once in a while, compared to every single time it is allocated now.

Anyone create Activity is starting it which will add it to the async context. so this wouldn't be once in awhile. almost will happen all the time anyone request a new Activity.

This is the pattern throughout entire implementation. Imagine how much overhead and GC pressure could be avoided.

I understand your point. I am just brainstorming with you. When using ActivitySource, the activity object is not going to get created at all except if there is a listeners. When having a listeners, there will be some cost at that time. So, in your case, is this the case you'll have listeners all the time? or you are trying to optimize when the listeners show up?

[ENikS]

Imagine typical production server scenario, an http server for example. It will always have some kind of telemetry running in background set to low verbosity level, so IsEnabled() will always be true and as result activities are always created.

Now, request comes in and DI is asked to create and provide some type, a controller for example. It starts resolving the type and all its dependencies. At one point DI container encounters a problem and according to verbosity settings needs to report it.

To properly report an event the DI requires to report entire dependency graph (quite complex sometime) from failed dependency to the original request to uniquely identify failure point. The graphs are dynamic constructs based on current registrations, already resolved instances, weather in Hong Kong and etc. and unknown at compile time, so the only way to properly report it is to walk hierarchy back to original request noting steps on the way and building the graph info. This requires an Activity for each step!

When firing the event the whole hierarchy is irrelevant, only the Current matters. Only this one will cross the async threshold and will be boxed, the rest will happily live on the stack without ever living the context.

When using ActivitySource, the activity object is not going to get created at all except if there is a listeners. When having a listeners, there will be some cost at that time. So, in your case, is this the case you'll have listeners all the time? or you are trying to optimize when the listeners show up?

As you could see from my explanation above this is not always desired behavior. If it is used as an Activity pool, it should provide other creation options.

[ENikS]

Thinking about it even further, why are we even talking about crossing async local context? Regardless where it is allocated it lives in Activity.Current and is guaranteed to not to outlive predecessors?

In this case it is absolutely irrelevant if it is on the heap or the stack. Am I missing something? Could you provide more info?

[tarekgh]

While we are discussing optimizations I have another question. In this scenario, would it be possible to use Dispose to fire Stop event on the DiagnosticListener that started it?

Yes.

[tarekgh]

Thinking about it even further, why are we even talking about crossing async local context? Regardless where it is allocated it lives in Activity.Current and is guaranteed to not to outlive predecessors?

Activity.Current is just accessing the async local to set/get the current activity object. How you'll make the following work without async local context?

C# Activity activity = ActivitySource.StartActivity(.....); Task.Run( () => Activity.Current == activity);

[ENikS]

I see your point, but this code makes no sense in DI terms.
By the time it executes the activity could be obsolete and current could be changed several times.

[noahfalk]

best case scenario vent from 16 ns to 45 ns just by adding constructor for Activity and simple IsEnabled(...)

IsEnabled() here refers to DiagnosticListener.IsEnabled()? I see you've got a few snippets above but could you add a snippet that shows the code you were benchmarking? I want to make sure we are focused on the right thing and I didn't recall we made any change in DiagnosticListener.IsEnabled() or Activity constructor so I wouldn't have expected a regression.

Perhaps these allocations could be avoided if data is sent as in structures instead of class instances?
... I was referring to these two allocations:
var aco = new ActivityCreationOptions(this, name, parentId, kind, tags, links);
var acoContext = new ActivityCreationOptions(this, name, aco.GetContext(), kind, tags, links);

ActivityCreationOptions is already a struct. The only object that is heap allocated should be Activity itself, if the ActivityListener indicated it should be sampled.

Overall I mentally bucket Activity performance into 3 tiers:

• All Activities are off - ActivitySource.HasListeners() returns false providing a fast opt out in ~1 ns
• Activities are sampled, some are needed but the current one isn't. This path should have zero heap allocations and complete the sampling operation in < 100ns for a performant sampler.
• Activities are sampled, current Activity will be logged. This path allocates and probably takes ~1000ns assuming the listener has a reasonably performant serialization+transmission mechanism.

I wouldn't be surprised if we still have some room for optimization within the different cases. In terms of Activity pooling/ValueActivity the idea has come up in the past but it appears challenging to say the least : ) Activity is expected to flow across threads to be async-local. Also Activity.Dispose() is used only as syntactic sugar to invoke Stop() at the end of a scope. After that point it is still legal for listeners to read from the Activity object in order to serialize it and there is no current API that indicates when each of the potentially many listeners is finished with that task. Even if we did add APIs or copying to do lifetime tracking that would allow it to be pooled there is a good chance the overhead of those approaches would be worse than the GC overhead of the allocation (for all the flak that allocating gets, the amortized cost to allocate and free an object that never lives past gen0 is astonishingly fast). Hope that helps!

[ENikS]

@noahfalk

This is the code I am benchmarking:
```C#
private object? ResolveContractDiagnostic(in Contract contract, ResolverOverride[] overrides)
{

        var enabled = UnityDiagnosticSource.DiagnosticListener.IsEnabled("Resolve", null);
        using Activity activity = new Activity("Resolve").AddTag("type", contract.Type.FullName)
                                                         .AddTag("name", contract.Name);
        try
        {
            if (enabled)
                UnityDiagnosticSource.DiagnosticListener.StartActivity(activity, overrides);
            else
                activity.Start();

            var container = this;
            bool? isGeneric = null;
            Contract generic = default;

            do
            {
                // Look for registration
                var manager = container._scope.Get(in contract);
                if (null != manager)
                {
                    //Registration found, check value
                    var value = manager.TryGetValue(_scope.Disposables);
                    if (!ReferenceEquals(RegistrationManager.NoValue, value)) return value;

                    return container.ResolveContract(in contract, manager, overrides);
                }
             ... removed for simplicity
            }
            while (null != (container = container.Parent));

            // No registration found, resolve unregistered
            return (bool)isGeneric ? ResolveUnregisteredGeneric(in contract, in generic, overrides) :
                contract.Type.IsArray ? ResolveArray(in contract, overrides)
                                      : ResolveUnregistered(in contract, overrides);
        }
        finally
        {
            if (enabled)
                UnityDiagnosticSource.DiagnosticListener.StopActivity(activity, null);
            else
                activity.Stop();
        }
    }

and these are the numbers:
```ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.1016 (1909/November2018Update/19H2)
AMD Ryzen Threadripper 2970WX, 1 CPU, 48 logical and 24 physical cores
.NET Core SDK=5.0.100-preview.8.20363.2
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.36102, CoreFX 5.0.20.36102), X64 RyuJIT
  DefaultJob : .NET Core 5.0.0 (CoreCLR 5.0.20.36102, CoreFX 5.0.20.36102), X64 RyuJIT


|                                   Method |         Mean |      Error |     StdDev |
|----------------------------------------- |-------------:|-----------:|-----------:|
| 'Container.Resolve<IUnityContainer >( )' | 17,186.86 ns | 318.365 ns | 549.164 ns |
| 'Container.Resolve<IServiceProvider>( )' | 16,696.03 ns | 232.667 ns | 228.511 ns |
| 'Container.R*Async<IUnityCont*Async>( )' |     19.11 ns |   0.095 ns |   0.079 ns |

The last result is the code without instrumentation

To be honest, I am quite disappointed and at loss. 899 times increase in mean time is astonishing.

These results are measured without any listeners. Please tell me I am doing something fundamentally wrong and there is a better/faster way of doing it.

As I've described in post above, this is pretty much expected scenario in production servers, where most of the Activities are unobserved but still are required to be instantiated.

[noahfalk]

There are a few things we should look at:

1. I wasn't able to repro the behavior you measured (my alternate code is shown below). Something specific about your scenario is 30x slower than my attempt to isolate just the Activity behavior.
2. What value are you extracting from creating presumably a large number of Activities that aren't being logged? I might be able to suggest a higher performing approach that still achieves your goal

If you want to post a complete self-contained repro that I could run as-is I can do so. This was my modification to your sample to wrap it in BDN and remove all the dependencies on specific Unity logic.

using System;
using System.Diagnostics;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

namespace MyBenchmarks
{
    public class ActivityBenchmark
    {
        // some dummy objects
        class Contract
        {
            public Type Type => typeof(object);
            public string Name => "The Contract!";
        }

        DiagnosticListener listener = new DiagnosticListener("Dummy");
        object[] overridesArg = new object[1];
        Contract contract = new Contract();

        [Benchmark]
        public void WorkAndActivity()
        {
            var enabled = listener.IsEnabled("Resolve", null);
            using Activity activity = new Activity("Resolve").AddTag("type", contract.Type.FullName)
                                                             .AddTag("name", contract.Name);
            try
            {
                if (enabled)
                    listener.StartActivity(activity, overridesArg);
                else
                    activity.Start();

                DoSomeWork();
            }
            finally
            {
                if (enabled)
                    listener.StopActivity(activity, null);
                else
                    activity.Stop();
            }
        }

        int DoSomeWork()
        {
            int x = 0;
            for(int i = 0; i < 100; i++)
            {
                x += i;
            }
            return x;
        }

        [Benchmark]
        public void WorkOnly() => DoSomeWork();
    }

    public class Program
    {
        public static void Main(string[] args)
        {
            var summary = BenchmarkRunner.Run<ActivityBenchmark>();
        }
    }
}

Results:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.1016 (1909/November2018Update/19H2)
Intel Core i7-9700K CPU 3.60GHz (Coffee Lake), 1 CPU, 8 logical and 8 physical cores
.NET Core SDK=5.0.100-preview.7.20366.6
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT
  DefaultJob : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT


|          Method |      Mean |    Error |   StdDev |
|---------------- |----------:|---------:|---------:|
| WorkAndActivity | 624.73 ns | 2.345 ns | 1.958 ns |
|        WorkOnly |  27.20 ns | 0.212 ns | 0.199 ns |

[ENikS]

@noahfalk

Imagine typical production server scenario, an http server for example. It will always have some kind of telemetry running in background set to low verbosity level, so IsEnabled() will always be true.

Now, request comes in and DI is asked to create and provide some type, a controller for example. It starts resolving the type and all its dependencies. At one point DI container encounters a problem and according to verbosity settings needs to report it.

To properly record an event the DI requires to report entire dependency graph (quite complex sometime) from failed dependency to the original request to uniquely identify failure point. The graphs are dynamic constructs and unknown at compile time, so the only way to properly report it is to walk hierarchy back to original request noting steps on the way and building the graph info. This requires an Activity for each step!

I’ve updated benchmarks a bit and getting results similar to yours. It is around 780 ns, but I am using preview 8 of net 5.0 and it is noticeably slower than 7.

Although it is not as shocking as my initial results, it is still quite significant. Perhaps there is something else that could be done to improve performance? I’ve noticed all that time is spent creating the Activity. Would it be possible to postpone some initializations to later time or even move it into get accessors so it is initialized on demand?

[davidfowl]

You don't want to use ActivitySource in your DI container, IMO it's too heavy for a component this low in the stack. I'd use an EventSource, it's what the default container does in Microsoft.Extensions does.

[noahfalk]

To properly record an event the DI requires to report entire dependency graph (quite complex sometime) from failed dependency to the original request to uniquely identify failure point. The graphs are dynamic constructs and unknown at compile time, so the only way to properly report it is to walk hierarchy back to original request noting steps on the way and building the graph info. This requires an Activity for each step!

There are alternative ways we could record these steps. Activity comes with some overheads you probably don't need in this task such as timestamps on start and stop, sampling support, randomized generation of W3C compliant trace ids, and tracking the current entry using async-local storage.

My general rule of thumb is that I aim to make telemetry collection not more than ~10% overhead on the useful work an app is doing. This means Activity and EventSource are reaching their limits when you need more than 150K Activities/sec and 300K EventSource messages/sec. For many scenarios those limits are no problem at all, but for a scenario like this where you are only doing 30ns of work a requirement to add low overhead diagnostics needs techniques in a different ballpark of performance. I'd look into building your own list of info and then only in the case an error occurs walk the entire list and log it. Something like:

````
public void StartOperation(string work)
{
Stack diagnosticTrace = new Stack(); // this could also be cached thread-local and maybe pre-set to a larger capacity
// memory vs. CPU usage tradeoff there
DoRecursiveWork(work, diagnosticTrace);
}

void DoRecursiveWork(string typeName, Stack diagnosticTrace)
{
diagnosticTrace.Push(typeName);
string nextRecursiveType = ...
if(failureOccured)
{
LogFailure(diagnosticTrace);
}
else
{
DoRecursiveWork(nextRecursiveType, diagnosticTrace);
}
diagnosticTrace.Pop();
}
````
I'd guess this solution is 50x faster than logging an Activity at each step and you could probably push it faster still if you wanted to. Hope that helps.

[ENikS]

Gentlemen, thank you for your feedback. I appreciate your opinions and suggestions.
This is very unfortunate that such a promising technology turned out to be so prohibitively slow.

I'd like to invite one more person @jbogard to this discussion. His excellent article inspired me to do the integration. He might want to know that it is not ready for prime time yet.

[ENikS]

@noahfalk

Would it be possible to make Activity type an abstract and move implementation into other type, similar to Source/Listener setup?

If I can derive my ResolutionFrame from Activity and implement just a limited set of features, perhaps I could meet performance requirements while keeping compatibility with Open Telemetry?

[tarekgh]

Would it be possible to make Activity type an abstract and move implementation into other type, similar to Source/Listener setup?

No, this will break all code doing new Activity(...)

[noahfalk]

He might want to know that it is not ready for prime time yet.

I'm not sure where you are setting the goal on performance, but I am suspicious you are setting it so high that no logging library in any language/platform with features similar to Activity or EventSource would meet the goal.

and implement just a limited set of features, I could meet performance requirements while keeping compatibility with Open Telemetry?

OpenTelemetry is part of the reason many of these features are required to exist. Removing them would make the proposed new thing incompatible with Open Telemetry's specification. IMO you are pushing OpenTelemetry to limits of its effective design space when you start looking at sub-microsecond telemetry. If the requirements drop below 100ns I doubt you could find any viable implementation regardless of language, platform, or optimization effort.

[ENikS]

@noahfalk

I'm not sure where you are setting the goal on performance, but I am suspicious you are setting it so high that no logging library in any language/platform with features similar to Activity or EventSource would meet the goal.

Think about this, if each activity takes almost a microsecond to just initialize, it leaves little time for a library to do anything else and puts a hard physical limit on throughput a system might have. Considering that it could be hundreds of activities to resolve reasonably complex graphs the bandwidth drops down to less than a thousand resolutions per second.

Take Kestrel server, for example. The request/response times are similar to what unity has and if instrumentation for just one activity takes 20 times longer than it takes to complete the whole request it is hard to accept.

After looking into activity code for last two day and trying to make it work, these are my observations

• the Activity is very heavy on upfront computations, instead of creating few essentials and create the rest on demand, it calculates everything (even if never used) during initialization
• design forces upfront data serialization, tags and the rest of metadata is required to be strings
• a lot of mutable collections inside implementation where simple arrays could do
• inability to add metadata in bulk, add must be called individually for each entry
• lack of value typed activity (it is a must for non async path)

The technology is essential for Unity so, if this library couldn’t be made faster I will have to create custom solution. I would much rather have standard implementation but as it is, it can’t be used. If you are not against the idea, I would love to join efforts, but it is up to you.

I am sorry I sound so critical, not my intention!

I have a deadline and my fiasco in integrating telemetry will slow it down for weeks. I am at the point where I can’t progress without the diagnostics so...

[noahfalk]

I am sorry I sound so critical, not my intention!

Not at all, no worries! What I was hoping to convey is that if you expect that we can maintain compatibility with OpenTelemetry while dramatically improving the performance of the API (lets say 5x faster), then I believe your expectation is unrealistic. If your expectation is actually something else then let me know what it is so I can try to be more helpful rather than just a naysayer : )

Considering that it could be hundreds of activities to resolve reasonably complex graphs

I don't believe there is any requirement that you must represent each step in the graph resolution with an Activity? This is a constraint you are self-imposing that I believe will make it extremely difficult for you to find a performant solution. As a comparison @davidfowl mentioned that DI in asp.net core doesn't use an Activity because they knew it could not be done within their performance requirements.

Take Kestrel server, for example. The request/response times are similar to what unity has and if instrumentation for just one activity takes 20 times longer than it takes to complete the whole request it is hard to accept

This comparison to Kestrel doesn't seem accurate. Even in the highest performing situations it achieves about 7M requests/sec on a machine with 28 hardware threads. This works out to ~4000ns request/response vs. the unity scenario that treats 30ns as the unit of work. The goal to use one Activity per-DI resolution step is over 100X more demanding than a Kestrel goal to use 1 Activity per request.

After looking into activity code for last two day and trying to make it work, these are my observations

Much of your feedback looks accurate and indeed there are definitely performance improvements possible, no dispute there. How much faster do you expect Activity would be after making those changes? How fast would it need to be for you to want to use it?

If you are not against the idea, I would love to join efforts, but it is up to you.

We are always glad to have people contribute PRs but I'd want to make sure our goals are aligned so that everyone's time is being well spent : ) From my perspective backwards compat and OpenTelemetry compat are essential requirements in any change in the official libraries. If your goal is to create a new non-compliant variation of Activity that has higher perf but sacrifices those other characteristics then I'd recommend doing your own project or a separate community project for that. Also I do care about Activity performance and would love to see it keep improving, but being honest I won't be able to commit much time to this effort beyond an occasional code review or a bit of discussion. Last .NET 5 is coming to a close so any changes would go towards .NET 6. If all of that sounds good then I'd suggest a good first steps are

1. Write up in this issue exactly what your perf goal is (xyz complete benchmark code should run with result better than abc)
2. Describe an initial change you propose to make so I can give you a heads up about any potential issues before you've sunk a bunch of time into it.

Hope that helps!

[ENikS]

Much of your feedback looks accurate and indeed there are definitely performance improvements possible, no dispute there. How much faster do you expect Activity would be after making those changes? How fast would it need to be for you to want to use it?

I will do research and will get back with numbers. I still don’t understand it that well, so it will take some time.

[noahfalk]

I will do research ...

Any thought on the "How fast would it need to be for you to want to use it?" It seems like you have a goal in mind but it has never been stated.

[ENikS]

I just don't know enough to give you an intelligent answer.

A rationale for optimism is this:

• Activity does not do anything inherently slow, like generics, for example
• Time spent during initialization is:
  
  
  
  • heap allocations (storage for collections), could be avoided if initialized via array of metadata (tags and etc.), non breaking
  
  
  • ID generation, could be postponed and spread out to get {} accessors, non breaking
  
  
  • serialization of tags and etc., metadata could be kept in original form and only serialized when required, (this is preferred form, I'd like to be able to examine attached objects in listeners without need to parse it back to a Type, for example), non breaking
  
  
• ValueActivity as you can see in referenced thread, I am hoping it could be done for non async path. Since async vs conventional path is known at compile time it is easily handled with proper type. Non breaking as well.

My most optimistic hope, it could be down to 100+ for synchronous execution and perhaps 200+ for async.

[jbogard]

As it is, I'm only using ActivitySource for operations that cross a network boundary. In terms of perf, it's not a big hit because I'm waiting for RabbitMQ to confirm a message or MongoDB to execute a query.

[ENikS]

I found a 'bug' ! According to answer to this question the Activity should fire a Stop event. Instead it quietly dies without notification:
```C#
Activity activity = null;
try
{
activity = DiagnosticListener.StartActivity(new Activity("Resolve"), null);

DiagnosticListener.Write("Resolve.Progress", null);

}
finally
{
activity.Dispose();
}

```

[tarekgh]

@ENikS you can listen to the event through the ActivityListener. Here is example https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/tests/ActivitySourceTests.cs#L61. Sorry, I overlooked the part you said in DiagnosticListener. I was just thinking there is a way to get such notification.

[ENikS]

Would it be possible to accommodate above scenario as well?
This is pretty common pattern. Perhaps an extension method that auto subscribes the listener?
```c#
Activity DiagnosticListener.StartActivity(string name, object payload);

This should be preferred pattern as it eliminates need for try/finally and still safe in case exception is throw down the line:

```C#
void DoWork()
{
  // Fire and forget
  using var activity = DiagnosticListener.StartActivity(new Activity("Resolve"), null);

  // Do work
}

[ENikS]

Well, after closer examination, I don't think it could be much better than may be 400+ ns without major design changes. There is nothing that could be done with AsyncLocal{T} get/set

Even if I postpone creating IDs in GenerateW3CId (two GUIDs taking 70 ns each) it would still be much higher than I could afford to spend on overhead. I guess I am stuck with simplified custom solution.

Thank you guys for your help! Great library and excellent work but unfortunately too feature reach for my case.
It was very useful learning experience though 😃

[noahfalk]

Cool glad we could help! Part of me was hoping you might find some awesome optimization opportunities I was missing, but still happy you got a chance to poke around and assess what potential still remains before we hit the wall on design. Thanks!