Orleans: Correct way to run silohost and client in docker?

Created on 25 Apr 2019 · 9Comments · Source: dotnet/orleans

I'm using Orleans 2.2.4. I want to put silohost in one docker and client in another docker. Currently these two dockers are on the same linux machine.

At first, I tried 'UseLocalhostClustering' and my configuration is below:

var builder = new SiloHostBuilder()
                .UseLocalhostClustering(siloPort, gatewayPort)
                .Configure<ClusterOptions>(options =>
                {
                    options.ClusterId = clusterId;
                    options.ServiceId = serviceId;
                })
                .Configure<EndpointOptions>(options =>
                {
                    options.AdvertisedIPAddress = IPAddress.Loopback;
                })
                .ConfigureApplicationParts(parts => parts.AddApplicationPart(applicationAssembly).WithReferences())
                .ConfigureLogging(logging => logging.AddConsole())
                .ConfigureLogging(logging => logging.SetMinimumLevel(LogLevel.Warning))
                .AddMemoryGrainStorageAsDefault()
                .EnableDirectClient();

Then in docker, I get error:

"type": "System.TimeoutException",
"message": "Connection to 127.0.0.1:84 could not be established in 00:00:05",
"stackTrace": "   at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)\n   at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)\n   at Orleans.Runtime.LRU`2.Get(TKey key)\n   at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)"

Then I tried 'UseDevelopmentClustering' and my configuration is below:

var ip = GetInternalIp(); // return the machine's ip
var builder = new SiloHostBuilder()
                .UseDevelopmentClustering(new IPEndPoint(ip, siloPort))
                .ConfigureEndpoints(ip, siloPort, gatewayPort)
                .Configure<ClusterOptions>(options =>
                {
                    options.ClusterId = clusterId;
                    options.ServiceId = serviceId;
                })
                .ConfigureApplicationParts(parts => parts.AddApplicationPart(applicationAssembly).WithReferences())
                .ConfigureLogging(logging => logging.AddConsole())
                .ConfigureLogging(logging => logging.SetMinimumLevel(LogLevel.Warning))
                .AddMemoryGrainStorageAsDefault()
                .EnableDirectClient();

Then in docker, I get another error:
[04/25/2019 05:13:47] [System] : Unhandled exception: Cannot assign requested address.

If not in docker, both of them work fine.
So is there something wrong with my configuration? What's the correct way to use orleans 2.2.4 in docker? Thanks in advance!

Source

bidiudiu

Most helpful comment

It should work out of the box, I would use the GenericHost example.

And also set

 builder.Configure<ProcessExitHandlingOptions>(options =>
                {
                    options.FastKillOnProcessExit = false;
                });

SebastianStehle on 26 Apr 2019

👍3

All 9 comments

I would just resolve the ip address using the host name in your docker network.

Loopback does only work with host network mode.

SebastianStehle on 25 Apr 2019

I would just resolve the ip address using the host name in your docker network.

Loopback does only work with host network mode.

Hi Sebastian,
Thanks for your reply! So I should use 'UseDevelopmentClustering' instead of 'UseLocalhostClustering', right?

Now I'm following unreliable-deployment-on-a-cluster-of-dedicated-servers but get error in docker:

[System] [04/25/2019 08:14:09]: Start Silo Host
siloPort is 82, gatewayPort is 83.
warn: Orleans.Runtime.Silo[100405]
Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
warn: Orleans.Runtime.Silo[100405]
Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
warn: Orleans.Runtime.NoOpHostEnvironmentStatistics[100708]
No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
warn: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101307]
Remote socket closed while receiving connection preamble data from endpoint 172.30.0.1:59506.
warn: Runtime.Messaging.SiloMessageSender/SystemSender[101021]
Exception getting a sending socket to endpoint S10.190.151.48:82:0
System.TimeoutException: Connection to 10.190.151.48:82 could not be established in 00:00:05
at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)
at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)
at Orleans.Runtime.LRU`2.Get(TKey key)
at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)
warn: Runtime.Messaging.SiloMessageSender/SystemSender[101021]
Exception getting a sending socket to endpoint S10.190.151.48:82:0
System.TimeoutException: Connection to 10.190.151.48:82 could not be established in 00:00:05
at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)
at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)
at Orleans.Runtime.LRU`2.Get(TKey key)
at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)
warn: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101308]
Exception receiving connection preamble data from endpoint 172.30.0.1:59520
System.Net.Sockets.SocketException (110): Connection timed out
at Orleans.Runtime.Messaging.IncomingMessageAcceptor.ReadFromSocket(Socket sock, Int32 expected)
warn: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101308]
Exception receiving connection preamble data from endpoint 172.30.0.1:59522
System.Net.Sockets.SocketException (110): Connection timed out
at Orleans.Runtime.Messaging.IncomingMessageAcceptor.ReadFromSocket(Socket sock, Int32 expected)
warn: Runtime.Messaging.SiloMessageSender/SystemSender[101021]
Exception getting a sending socket to endpoint S10.190.151.48:82:0
System.TimeoutException: Connection to 10.190.151.48:82 could not be established in 00:00:05
at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)
at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)
at Orleans.Runtime.LRU`2.Get(TKey key)
at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)
warn: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101308]
Exception receiving connection preamble data from endpoint 172.30.0.1:59524
System.Net.Sockets.SocketException (110): Connection timed out
at Orleans.Runtime.Messaging.IncomingMessageAcceptor.ReadFromSocket(Socket sock, Int32 expected)
warn: Runtime.Messaging.SiloMessageSender/SystemSender[101021]
Exception getting a sending socket to endpoint S10.190.151.48:82:0
System.TimeoutException: Connection to 10.190.151.48:82 could not be established in 00:00:05
at Orleans.Runtime.SocketManager.Connect(Socket s, IPEndPoint endPoint, TimeSpan connectionTimeout)
at Orleans.Runtime.SocketManager.SendingSocketCreator(IPEndPoint target)
at Orleans.Runtime.LRU`2.Get(TKey key)
at Orleans.Runtime.Messaging.SiloMessageSender.GetSendingSocket(Message msg, Socket& socket, SiloAddress& targetSilo, String& error)
companybll.kg    | warn: Orleans.Runtime.CallbackData[100157]
companybll.kg    |       Response did not arrive on time in 00:00:30 for message: Request S172.30.0.3:82:293876050*stg/15/0000000f@S0000000f->S10.190.151.48:82:0*stg/29/0000001d@S0000001d #11: . Target History is: <S10.190.151.48:82:0:*stg/29/0000001d:@S0000001d>. About to break its promise.
fail: Orleans.Runtime.SiloLifecycleSubject[100450]
Lifecycle observer Orleans.Runtime.Silo failed to start due to errors at stage 8000.
System.TimeoutException: Starting MembershipOracle failed due to timeout 00:05:00
at Orleans.OrleansTaskExtentions.WithTimeout(Task taskToComplete, TimeSpan timeout, String exceptionMessage)
at Orleans.Runtime.Silo.<>c__DisplayClass72_0.<<OnRuntimeGrainServicesStart>g__StartMembershipOracle|1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Orleans.Runtime.Silo.StartAsyncTaskWithPerfAnalysis(String taskName, Func`1 task, Stopwatch stopWatch)
at Orleans.Runtime.Silo.OnRuntimeGrainServicesStart(CancellationToken ct)
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
fail: Orleans.Runtime.SiloLifecycleSubject[100450]
Lifecycle start canceled due to errors at stage 8000
System.TimeoutException: Starting MembershipOracle failed due to timeout 00:05:00
at Orleans.OrleansTaskExtentions.WithTimeout(Task taskToComplete, TimeSpan timeout, String exceptionMessage)
at Orleans.Runtime.Silo.<>c__DisplayClass72_0.<<OnRuntimeGrainServicesStart>g__StartMembershipOracle|1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Orleans.Runtime.Silo.StartAsyncTaskWithPerfAnalysis(String taskName, Func`1 task, Stopwatch stopWatch)
at Orleans.Runtime.Silo.OnRuntimeGrainServicesStart(CancellationToken ct)
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
at Orleans.LifecycleSubject.WrapExecution(CancellationToken ct, Func`2 action)
at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
fail: Orleans.Runtime.Silo[100439]
Exception during Silo.Start
Orleans.Runtime.OrleansLifecycleCanceledException: Lifecycle start canceled due to errors at stage 8000 ---> System.TimeoutException: Starting MembershipOracle failed due to timeout 00:05:00
at Orleans.OrleansTaskExtentions.WithTimeout(Task taskToComplete, TimeSpan timeout, String exceptionMessage)
at Orleans.Runtime.Silo.<>c__DisplayClass72_0.<<OnRuntimeGrainServicesStart>g__StartMembershipOracle|1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Orleans.Runtime.Silo.StartAsyncTaskWithPerfAnalysis(String taskName, Func`1 task, Stopwatch stopWatch)
at Orleans.Runtime.Silo.OnRuntimeGrainServicesStart(CancellationToken ct)
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
at Orleans.LifecycleSubject.WrapExecution(CancellationToken ct, Func`2 action)
at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
--- End of inner exception stack trace ---
at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
[04/25/2019 08:19:11] [System] : Unhandled exception: Lifecycle start canceled due to errors at stage 8000.

Here's host configuration:

IPAddress.TryParse(Environment.GetEnvironmentVariable("HostIP"), out var ip);
var builder = new SiloHostBuilder()
                .UseDevelopmentClustering(new IPEndPoint(ip, siloPort))
                .ConfigureEndpoints(siloPort, gatewayPort)
                .Configure<ClusterOptions>(options =>
                {
                    options.ClusterId = clusterId;
                    options.ServiceId = serviceId;
                })
                .ConfigureApplicationParts(parts => parts.AddApplicationPart(applicationAssembly).WithReferences())
                .ConfigureLogging(logging => logging.AddConsole())
                .ConfigureLogging(logging => logging.SetMinimumLevel(LogLevel.Warning))
                .AddMemoryGrainStorageAsDefault()
                .EnableDirectClient();

Here's client configuration:

IPEndPoint[] endpoints = { new IPEndPoint(ip, gatewayPort) };
var client = new ClientBuilder()
                    .UseStaticClustering(endpoints)
                    .ConfigureApplicationParts(parts => parts.AddApplicationPart(typeof(Generic).Assembly).WithReferences())
                    .Configure<ClusterOptions>(options =>
                    {
                        options.ClusterId = clusterId;
                        options.ServiceId = serviceId;
                    })
                .ConfigureLogging(logging => logging.AddConsole())
                .Build();

Do you have any suggestions?

bidiudiu on 25 Apr 2019

I would use a "normal" clustering provider such as AdoNet or MongoDB or Azure or whatever and then use the host name to resolve the advertised ip address with Dns.GetHostName(). This worked for me (today I just use co-hosting and have no client). It also works well in kubernetes.

SebastianStehle on 25 Apr 2019

I would use a "normal" clustering provider such as AdoNet or MongoDB or Azure or whatever and then use the host name to resolve the advertised ip address with Dns.GetHostName(). This worked for me (today I just use co-hosting and have no client). It also works well in kubernetes.

Thanks a lot! I'll have a try.
By the way, do you know how to handle 'shutdown silo gracefully' problem in docker? I find the document hasn't been completed yet.

bidiudiu on 25 Apr 2019

It should work out of the box, I would use the GenericHost example.

And also set

 builder.Configure<ProcessExitHandlingOptions>(options =>
                {
                    options.FastKillOnProcessExit = false;
                });

SebastianStehle on 26 Apr 2019

👍3

solved so close it

bidiudiu on 9 May 2019

solved so close it

I am also facing the same issue, I am trying to deploy it on Azure containers. Can you past sample code how you configure silobuilder.

I am using UseAzureStorageClustering

keshavraja on 11 Jul 2019

@keshavraja I am using UseAzureStorageClustering in docker. Here's my configuration:

        private static async Task<ISiloHost> StartSilo(string clusterId, string serviceId, int siloPort, int gatewayPort, string connectionString, Assembly applicationAssembly)
        {
            var builder = new SiloHostBuilder()
                .Configure<ProcessExitHandlingOptions>(options =>
                {
                    options.FastKillOnProcessExit = false;
                })
                .Configure<ClusterOptions>(options =>
                {
                    options.ClusterId = clusterId;
                    options.ServiceId = serviceId;
                })
                .UseAzureStorageClustering(options => options.ConnectionString = connectionString)
                .ConfigureEndpoints(Dns.GetHostName(), siloPort, gatewayPort)
                .ConfigureApplicationParts(parts => parts.AddApplicationPart(applicationAssembly).WithReferences())
                .ConfigureLogging(logging => logging.AddConsole())
                .ConfigureLogging(logging => logging.SetMinimumLevel(LogLevel.Warning))
                .EnableDirectClient();

            var host = builder.Build();
            await host.StartAsync().ConfigureAwait(false);

            return host;
        }

bidiudiu on 15 Jul 2019