Apm-agent-dotnet: Use new Redis instrumentation in an ASP.NET Core app

[ejsmith]

APM Agent version

1.8.0

Environment

.NET Framework/Core name and version (e.g. .NET 4.6.1, NET Core 3.1.100) :
.NET 5.0

Application Target Framework(s) (e.g. net461, netcoreapp3.1):
net5.0

Describe the bug

It seems to be impossible to configure the Redis instrumentation inside of an ASP.NET Core app while registering the ConnectionMultiplexer into the DI services collection in the Startup.ConfigureServices method. Calling UseElasticApm on the ConnectionMultiplexer triggers an access to the Agent.Instance singleton therefore configuring the instance and locking it. When you later call app.UseElasticApm in the Startup.Configure method you get an Elastic.Apm.Agent.InstanceAlreadyCreatedException with a message saying The singleton APM agent has already been instantiated and can no longer be configured.

Honestly, I have to say that working with the Elastic APM agent configuration has been a pretty poor experience as I have run into several roadblocks along the way.

[ejsmith]

To reproduce, just add this code block to the ConfigureServicesExceptMvc method in the SampleAspNetCoreApp and add a reference to the Elastic.Apm.StackExchange.Redis project.

const string redisConnection = @"localhost,abortConnect=false";
var redisConnectionMultiplexer = ConnectionMultiplexer.Connect(redisConnection);
redisConnectionMultiplexer.UseElasticApm();
services.AddSingleton(redisConnectionMultiplexer);

[russcam]

Thank you for the honest feedback @ejsmith, it's very much appreciated.

I think this situation could be improved if the agent is accessed lazily inside of the ElasticApmProfiler such that it's retrieved only when the connection multiplexer asks for a profiling session. With this setup, it could still be possible that the agent is initialized by the redis integration _before_ the app initializes it in app.UseElasticApm if a call is made to redis beforehand, but it would allow the setup in your comment to work.

Registering a delegate to construct the ConnectionMultiplexer singleton should work, if the first usage of ConnectionMultiplexer happens after app.UseElasticApm

c# services.AddSingleton(_ => { const string redisConnection = @"localhost,abortConnect=false"; var redisConnectionMultiplexer = ConnectionMultiplexer.Connect(redisConnection); redisConnectionMultiplexer.UseElasticApm(); return redisConnectionMultiplexer; });

Would you be able to try this?

[ejsmith]

I tried that and it does work in the sample app, but does not work for my application. I guess something is causing that singleton to be requested before the spot where I am calling app.UseElasticApm.

[russcam]

@ejsmith is there a public repository and branch that I can take a look at?

[ejsmith]

@russcam no, this is a closed source app. I tried to take a copy of ElasticApmProfiler and make it use a Lazy<IApmAgent> to see if that would work, but of course there are all kinds of internals being used. I'm sorry, but it makes me so frustrated when libraries like this cheat in their own extensions and make it damn near impossible for the community to create their own instrumentations or modify existing ones.

[ejsmith]

Thanks @russcam for getting on a Zoom call with me and helping me figure out this issue. For anyone else looking at this, Russ is going to make the agent access in the ElasticApmProfiler lazy and he helped me find where my app was aggressively loading a dependency and causing the agent to be configured before the other app.UseElasticApm call was made.

[russcam]

Thanks for your time earlier @ejsmith.

I've opened https://github.com/elastic/apm-agent-dotnet/pull/1192 to lazily access the agent in the redis integration when accessed through the Agent.Instance static property.

An approach that might be of interest to other folks that do need to use Redis early in startup, and before the agent is initialized, it would be possible right now to register the redis integration _after_ initialization of the agent in app configure. For example, in ConfigureServices

```c#
services.AddSingleton(_ => {
const string redisConnection = @"localhost,abortConnect=false";
var redisConnectionMultiplexer = ConnectionMultiplexer.Connect(redisConnection);
return redisConnectionMultiplexer;
});

Then, in `Configure`

```c#
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
   // Some call to redis happens **before** the agent is initialized

    // agent initialized here
    app.UseAllElasticApm(Configuration);

    var muxer = app.ApplicationServices.GetService<ConnectionMultiplexer>();
    // register redis integration
    muxer.UseElasticApm();

    // other app setup
}

It does mean that those initial redis calls won't be traced, but would avoid the Elastic.Apm.Agent.InstanceAlreadyCreatedException.

As part of https://github.com/elastic/apm-agent-dotnet/pull/1161, Elastic.Apm.Agent.InstanceAlreadyCreatedException is removed and there will be an error log instead.

[russcam]

1192 has been merged and will go out in a future release. I'm going to close this as I think it addresses the order of initialization issue.