Orleans: Docker + Consul + Memory Storage - how to ?

[pchalamet]

Hello,

I'm hitting the wall while trying to get a silo running under docker with only Consul and Memory Storage. The configuration I use is in this gist: https://gist.github.com/pchalamet/d486e846a16cbf2b871712cd846fefa3

Looks like Consul is correctly used by Orleans:

But at some point, the silo is shutting down - and I really do not understand why (full logs in the gist):

I miss something for sure in the configuration (see gist). I also found https://dotnet.github.io/orleans/Documentation/deployment/consul_deployment.html but looks like SiloHost & friends are obsolete now and UseConsulClustering does initialize everything as expected instead.

So, is there something I do the wrong way ?
Thanks for your help!

[ReubenBond]

Could you add your program.cs, too? The silo may be shutting down because the program is terminatin

[pchalamet]

Updated Silo.cs. It does work outside docker btw.

[ReubenBond]

What if you print a message after the Console Read statement to see if that's why it's shutting down?

[pchalamet]

Oh... yes you are right. The process is getting past Console.ReadLine(); normally and just shut down :-(
That's really weird. Thanks for pinpointing the problem!

[pchalamet]

I still do not know why the process is exiting this way (in Docker only) but blocking indefinitively does solve the problem for now. I will investigate this later.

Thanks a lot @ReubenBond for helping!

[ReubenBond]

Try waiting for Ctrl + C (SIGINT) instead, Console.CancelKeyPress is the event, I believe

[pchalamet]

I've already tried this (that's what is in the Kubernetes sample) and unfortunately same problem again. There is something weird that I have to investigate because I'm so upset to have spent so much time on something so ridiculous 😒 I will keep you posted if it's Orleans related.

[ReubenBond]

@pchalamet folks in Gitter may have some tips

[ReubenBond]

How does your Console.CancelKeyPress code look?

[seniorquico]

I started a small Docker sample project for reference. This, and some other projects our team has in production, have been using the following to block/wait and listen for SIGTERM from Docker:

private static async Task Main(string[] args)
{
    // Cancel all tasks on SIGINT (Ctrl+C) and SIGTERM (process exiting).
    var cancellationTokenSource = new CancellationTokenSource();
    AssemblyLoadContext.Default.Unloading += context =>
    {
        cancellationTokenSource.Cancel();
    };
    Console.CancelKeyPress += (sender, e) =>
    {
        cancellationTokenSource.Cancel();
        e.Cancel = true;
    };

    ... // start silo

    await Task.Delay(Timeout.Infinite, cancellationTokenSource.Token);

    ... // stop silo
}

[pchalamet]

I shall not work at night and mess with the semaphore (initialCount should be 0, not 1). So it does work perfectly now 😅

var sem = new SemaphoreSlim(0, 1); void cancelKeyPress(object sender, ConsoleCancelEventArgs e) { e.Cancel = true; sem.Release(); } Console.CancelKeyPress += cancelKeyPress; await sem.WaitAsync();

[pchalamet]

Thanks @seniorquico, probably better.

[JorgeCandeias]

Out of curiosity, have you tried the generic host builder yet? Here's an example from the newer samples:

public static Task Main(string[] args)
{
    return new HostBuilder()
        .UseOrleans(builder =>
        {
            builder
                .UseLocalhostClustering()
                .ConfigureApplicationParts(manager =>
                    {
                        manager.AddApplicationPart(typeof(GameGrain).Assembly).WithReferences();
                    });
        })
        .ConfigureLogging(builder =>
        {
            builder.AddConsole();
        })
        .ConfigureServices(services =>
        {
            services.Configure<ConsoleLifetimeOptions>(options =>
            {
                options.SuppressStatusMessages = true;
            });
        })
        .RunConsoleAsync();
}

This wires up Ctrl+C and SIGTERM for you and starts the silo in one go. Haven't tried it on docker yet though.

[seniorquico]

Nice, @JorgeCandeias! RunConsoleAsync registers ConsoleLifetime as a singleton IHostLifetime in the host DI container. ConsoleLifetime uses a very similar approach to my example above, but it subscribes to AppDomain.CurrentDomain.ProcessExit instead of AssemblyLoadContext.Default.Unloading. (It's quite possible my example code originated in a pre-.NET Standard 2 project.)

The ASP.NET Core docs indicate ConsoleLifetime is the default IHostLifetime registration (in their projects). Given all of my ASP.NET Core projects have worked out of the box in Docker, I'd assume this works just fine for Orleans+Docker. I'll have to update/simplify my projects! Thanks for the pointer!

References:
RunConsoleAsync (extension method)
ConsoleLifetime (the part that listens for SIGINT and SIGTERM)

[RehanSaeed]

I came here after reading Andre Lock's post on ConsoleLifetime to see if Orleans supports it.

I'd like to update my Dotnet Boxed Orleans project template using 'dotnet new' to use this new system. Is there any documentation for it?

[ReubenBond]

@RehanSaeed

I came here after reading Andre Lock's post on ConsoleLifetime to see if Orleans supports it.
[...] Is there any documentation for it?

Not yet, we have to update documentation. The documentation would largely focus on using IHostBuilder.AddOrleans(Action<ISiloBuilder>) and point to the generic host builder documentation for the rest (lifetime, etc). Most/all methods which support ISiloHostBuilder have overloads to support ISiloBuilder, so the configuration looks almost identical

EDIT: there is a sample here: https://github.com/dotnet/orleans/tree/master/Samples/2.3/GenericHost