Orleans: Orleans on Kubernetes, pod keeps crashing/restarting

Created on 24 Jul 2020  ·  20Comments  ·  Source: dotnet/orleans

Situation

I have a kubernetes cluster with 3 Pods

  • Database (PostgreSQL)
  • Silo Service 1 (AdoNET Clustering using PostgreSQL)
  • Silo Service 2 (AdoNET Clustering using PostgreSQL

They work and communicate as expected when running for the first time with clean database, 2 rows for each Silo are created for the membership table.

I have a volume for my database Pod, so the data gets persisted.

If i then stop/delete my kube deployment/services/pods, change some configs and start again (normal development / experimentation flow), extra 2 rows for each Silo are created in the membership table, total of 4 rows, Orleans will perform around 5 or 6 connection re-tries to the older/dead silo row IPs and will timeout, services will run fine.

Now i stop/delete and start my deployment/services/pods again, now we have 6 rows, Orleans will not stop re-trying connection with the previous 4 dead/old silos until the pod crashes and restarts and this will keep going forever.

In summary, the steps to reproduce this problem are:

*1. kubectl apply -f .. -f ... : start my deployments/pods/services with clean database

  1. navigate, test stuff, working great
  2. change config
  3. kubectl delete services,pods,etc
  4. kubectl apply -f..-f.. start again
  5. with 2 silos, now i have 4 rows in my membership table, my 2 silos will fail to connect to the older 2 rows for some time and run fine.
  6. stop and start again, now i have 2 silos and 6 membership rows, my 2 silos will fail to connect to the older 4 rows until my pods crash and restart forever.*

My pods after 2 crashes/restarts

NAME                                     READY   STATUS    RESTARTS   AGE
eventstoredb-75bfdf767-dt648             1/1     Running   0          4m2s
fhirserver-pod-6558c57dc4-pl8gr          1/1     Running   2          4m2s
patientservice-pod-788877d9f8-prs98      1/1     Running   2          4m2s
postgres-fhirdatabase-7dd856f8c4-gdg7m   1/1     Running   0          4m2s
ui-pod-85f768b7c6-cwq7l                  1/1     Running   0          4m2s

fhirserver-pod Logs before crashing (both pod logs look the same before they crash/restart)

[24-Jul-2020 18:35:04 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Initializing PostgreOrleansStorage()
[24-Jul-2020 18:35:04 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - @ EnsureCreated()
ConnectionString: Server=fhirdatabase-service;Port=5108;Database=smartfhirserver.db;User ID=admin;Password=admin
[24-Jul-2020 18:35:06 FTL] Database error
Npgsql.NpgsqlException (0x80004005): Exception while connecting
 ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (111): Connection refused 10.98.23.247:5108
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.<>c.<ConnectAsync>b__274_0(IAsyncResult iar)
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.TaskExtensions.WithCancellation(Task task, CancellationToken cancellationToken)
   at Npgsql.TaskExtensions.WithTimeout(Task task, NpgsqlTimeout timeout)
   at Npgsql.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 419
   at SMARTPlatform.Messaging.Storage.PostgreOrleansStorage.DatabaseExists(String dbName, String connectionString)
   at SMARTPlatform.Messaging.Storage.PostgreOrleansStorage.CreateDatabase(String databaseName, String connectionString)
   at SMARTPlatform.Messaging.Storage.PostgreOrleansStorage.EnsureCreated(String dbName, String connectionString)
   at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
[24-Jul-2020 18:35:06 INF] [1] Trying to make background database connections
[24-Jul-2020 18:35:16 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - @ EnsureCreated()
ConnectionString: Server=fhirdatabase-service;Port=5108;Database=smartfhirserver.db;User ID=admin;Password=admin
[24-Jul-2020 18:35:16 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Database smartfhirserverclustering.db already exists!
[24-Jul-2020 18:35:16 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Updated connection string!
[24-Jul-2020 18:35:17 INF] Entity Framework Core 3.1.6 initialized 'WriteApplicationDbContext' using provider 'Npgsql.EntityFrameworkCore.PostgreSQL' with options: MigrationsAssembly=Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
[24-Jul-2020 18:35:17 INF] Executed DbCommand (3ms) [Parameters=[], CommandType='Text', CommandTimeout='30']

                    SELECT CASE WHEN COUNT(*) = 0 THEN FALSE ELSE TRUE END
                    FROM information_schema.tables
                    WHERE table_type = 'BASE TABLE' AND table_schema NOT IN ('pg_catalog', 'information_schema')
[24-Jul-2020 18:35:17 WRN] Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
[24-Jul-2020 18:35:17 INF] User profile is available. Using '/root/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
[24-Jul-2020 18:35:17 INF] Silo starting with GC settings: ServerGC=False GCLatencyMode=Interactive
[24-Jul-2020 18:35:17 WRN] Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
[24-Jul-2020 18:35:17 WRN] Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
[24-Jul-2020 18:35:17 INF] -------------- Initializing silo on host fhirserver-pod-6558c57dc4-pl8gr MachineName fhirserver-pod-6558c57dc4-pl8gr at 10.1.2.167:11111, gen 333311717 --------------
[24-Jul-2020 18:35:17 INF] Starting silo Silo_820bf
[24-Jul-2020 18:35:18 WRN] No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
[24-Jul-2020 18:35:18 INF] Starting VirtualBucketsRingProvider on silo S10.1.2.167:11111:333311717/xA66E0673.
[24-Jul-2020 18:35:18 INF] Added Server S10.1.2.167:11111:333311717/xA66E0673. Current view: [S10.1.2.167:11111:333311717 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
[24-Jul-2020 18:35:18 INF] -NotifyLocalRangeSubscribers about old <(0 0], Size=x100000000, %Ring=100.000%> new <MultiRange: Size=x100000000, %Ring=100.000%> increased? True
[24-Jul-2020 18:35:18 INF] -------------- Started silo S10.1.2.167:11111:333311717, ConsistentHashCode A66E0673 --------------
[24-Jul-2020 18:35:18 INF] Creating key {d8d191e5-450d-4b22-9ee5-9fb14ef90173} with creation date 2020-07-24 18:35:18Z, activation date 2020-07-24 18:35:18Z, and expiration date 2020-10-22 18:35:18Z.
[24-Jul-2020 18:35:18 WRN] No XML encryptor configured. Key {d8d191e5-450d-4b22-9ee5-9fb14ef90173} may be persisted to storage in unencrypted form.
[24-Jul-2020 18:35:18 INF] Writing data to file '/root/.aspnet/DataProtection-Keys/key-d8d191e5-450d-4b22-9ee5-9fb14ef90173.xml'.
[24-Jul-2020 18:35:18 INF] Now listening on: http://[::]:80
[24-Jul-2020 18:35:18 INF] Starting Orleans Silo.
[24-Jul-2020 18:35:18 INF] Stage First (-2147483648): Orleans.Runtime.SiloOptionsLogger
[24-Jul-2020 18:35:18 INF] Stage 1999: ConnectionManager, SiloConnectionListener, GatewayConnectionListener
[24-Jul-2020 18:35:18 INF] Stage RuntimeInitialize (2000): Orleans.Runtime.InsideRuntimeClient, SiloStatusListenerManager, ClusterMembershipService, SiloConnectionMaintainer, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage 2001: MembershipAgent
[24-Jul-2020 18:35:18 INF] Stage RuntimeServices (4000): Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage RuntimeGrainServices (8000): MembershipTableManager, HostedClient, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage AfterRuntimeGrainServices (8100): MembershipAgent, SiloStatusListenerManager
[24-Jul-2020 18:35:18 INF] Stage ApplicationServices (10000): Orleans.Runtime.Versions.GrainVersionStore
[24-Jul-2020 18:35:18 INF] Stage BecomeActive (19999): MembershipAgent, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Stage Active (20000): ClusterHealthMonitor, MembershipTableCleanupAgent, GatewayConnectionListener, Orleans.Runtime.Silo
[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ActivationCountBasedPlacementOptions:
ChooseOutOf: 2

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ClusterMembershipOptions:
NumMissedTableIAmAliveLimit: 2
LivenessEnabled: True
ProbeTimeout: 00:00:05
TableRefreshTimeout: 00:01:00
DeathVoteExpirationTimeout: 00:02:00
IAmAliveTablePublishTimeout: 00:05:00
MaxJoinAttemptTime: 00:05:00
ValidateInitialConnectivity: True
UseLivenessGossip: True
NumProbedSilos: 3
NumMissedProbesLimit: 3
NumVotesForDeathDeclaration: 2
DefunctSiloExpiration: 7.00:00:00
DefunctSiloCleanupPeriod: 01:00:00
IsRunningAsUnitTest: False

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ClusterOptions:
ClusterId: smartplatform_silo_cluster
ServiceId: smartplatform_silo_service

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ConnectionOptions:
ProtocolVersion: Version1
ConnectionsPerEndpoint: 1
ConnectionRetryDelay: 00:00:01
OpenConnectionTimeout: 00:00:05

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.ConsistentRingOptions:
NumVirtualBucketsConsistentRing: 30
UseVirtualBucketsConsistentRing: True

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.EndpointOptions:
AdvertisedIPAddress: 10.1.2.167
SiloPort: 11111
GatewayPort: 30000
SiloListeningEndpoint: 0.0.0.0:11111
GatewayListeningEndpoint: 0.0.0.0:30000

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.GrainCollectionOptions:
CollectionQuantum: 00:01:00
CollectionAge: 02:00:00
DeactivationTimeout: 00:00:30

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.GrainDirectoryOptions:
CachingStrategy: Adaptive
CacheSize: 1000000
InitialCacheTTL: 00:00:30
MaximumCacheTTL: 00:04:00
CacheTTLExtensionFactor: 2
LazyDeregistrationDelay: 00:01:00

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.GrainVersioningOptions:
DefaultCompatibilityStrategy: BackwardCompatible
DefaultVersionSelectorStrategy: AllCompatibleVersions

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.LoadSheddingOptions:
LoadSheddingEnabled: False
LoadSheddingLimit: 95

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.PerformanceTuningOptions:
DefaultConnectionLimit: 200
Expect100Continue: False
UseNagleAlgorithm: False
MinDotNetThreadPoolSize: 0
MinIOThreadPoolSize: 0

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SchedulingOptions:
PerformDeadlockDetection: False
AllowCallChainReentrancy: True
MaxActiveThreads: 4
DelayWarningThreshold: 00:00:10
ActivationSchedulingQuantum: 00:00:00.1000000
TurnWarningLengthThreshold: 00:00:00.2000000
MaxPendingWorkItemsSoftLimit: 0
EnableWorkerThreadInjection: False
StoppedActivationWarningInterval: 00:01:00

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SerializationProviderOptions:
FallbackSerializationProvider:
MaxSustainedSerializationContextCapacity: 64

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SiloMessagingOptions:
SiloSenderQueues: 1
GatewaySenderQueues: 1
MaxForwardCount: 2
ClientDropTimeout: 00:01:00
ClientRegistrationRefresh: 00:05:00
MaxEnqueuedRequestsSoftLimit: 0
MaxEnqueuedRequestsHardLimit: 0
MaxEnqueuedRequestsSoftLimit_StatelessWorker: 0
MaxEnqueuedRequestsHardLimit_StatelessWorker: 0
MaxRequestProcessingTime: 02:00:00
AssumeHomogenousSilosForTesting: False
DEFAULT_SHUTDOWN_REROUTE_TIMEOUT: 00:00:10
ShutdownRerouteTimeout: 00:00:10
SystemResponseTimeout: 00:00:30
ResponseTimeout: 00:00:30
ResponseTimeoutWithDebugger: 00:30:00
DropExpiredMessages: True
BufferPoolBufferSize: 4096
BufferPoolMaxSize: 10000
BufferPoolPreallocationSize: 250
PropagateActivityId: False
LargeMessageWarningThreshold: 85000
MaxMessageHeaderSize: 10485760
MaxMessageBodySize: 104857600

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.SiloOptions:
SiloName: Silo_820bf

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.StatisticsOptions:
PerfCountersWriteInterval: 00:00:30
LogWriteInterval: 00:05:00
CollectionLevel: Info

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.TelemetryOptions:

[24-Jul-2020 18:35:18 INF] Configuration Orleans.Configuration.TypeManagementOptions:
TypeMapRefreshInterval: 00:01:00

[24-Jul-2020 18:35:18 INF] Orleans.Runtime.SiloOptionsLogger started in stage First (-2147483648) in 7.9077 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage First (-2147483648) took 8.8816 Milliseconds
[24-Jul-2020 18:35:18 INF] ConnectionManager started in stage 1999 in 0.0353 Milliseconds
[24-Jul-2020 18:35:18 INF] GatewayConnectionListener started in stage 1999 in 2.9368 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloConnectionListener started in stage 1999 in 4.5163 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage 1999 took 4.89 Milliseconds
[24-Jul-2020 18:35:18 INF] Loaded grain type summary for 7 types:
Grain class Infrastructure.Grains.Infrastructure.Grains.ResourceGrain [1059605650 (0x3F284C92)] from Infrastructure.dll implementing interfaces: SMARTPlatform.Messaging.GrainInterfaces.Fhir.IResourceGrain [-1455888765 (0xA938E683)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.LogConsistency.Orleans.LogConsistency.LogConsistentGrain<TView> [-319513858 (0xECF49AFE)] from Orleans.Core.dll implementing interfaces:
Grain class Orleans.Runtime.Development.Orleans.Runtime.Development.DevelopmentLeaseProviderGrain [940524075 (0x380F422B)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Development.IDevelopmentLeaseProviderGrain [-1234611069 (0xB6695483)]
Grain class Orleans.Runtime.Management.Orleans.Runtime.Management.ManagementGrain [1954798034 (0x7483D9D2)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.IManagementGrain [-1734666656 (0x989B1660)]
Grain class Orleans.Runtime.ReminderService.Orleans.Runtime.ReminderService.GrainBasedReminderTable [-55315191 (0xFCB3F509)] from Orleans.Runtime.dll implementing interfaces: Orleans.IReminderTableGrain [-1135060418 (0xBC585A3E)]
Grain class Orleans.Runtime.Versions.Orleans.Runtime.Versions.VersionStoreGrain [331003096 (0x13BAB4D8)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Versions.IVersionStoreGrain [-297600501 (0xEE42FA0B)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.Streams.Orleans.Streams.PubSubRendezvousGrain [1903070868 (0x716E8E94)] from Orleans.Runtime.dll implementing interfaces: Orleans.Streams.IPubSubRendezvousGrain [1746702088 (0x681C8F08)]

[24-Jul-2020 18:35:18 INF] Start InsideRuntimeClient took 5 Milliseconds
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.InsideRuntimeClient started in stage RuntimeInitialize (2000) in 6.3063 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloStatusListenerManager started in stage RuntimeInitialize (2000) in 0.0287 Milliseconds
[24-Jul-2020 18:35:18 INF] ClusterMembershipService started in stage RuntimeInitialize (2000) in 0.0944 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloConnectionMaintainer started in stage RuntimeInitialize (2000) in 63.9175 Milliseconds
[24-Jul-2020 18:35:18 INF] Silo Start()
[24-Jul-2020 18:35:18 INF] Configured .NET ServicePointManager to Expect100Continue=False, DefaultConnectionLimit=200, UseNagleAlgorithm=False to improve Azure storage performance.
[24-Jul-2020 18:35:18 INF] ConfigureThreadPoolAndServicePointSettings took 0 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.Silo started in stage RuntimeInitialize (2000) in 1.4288 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage RuntimeInitialize (2000) took 72.0834 Milliseconds
[24-Jul-2020 18:35:18 INF] MembershipAgent started in stage 2001 in 0.0314 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage 2001 took 0.0804 Milliseconds
[24-Jul-2020 18:35:18 INF] Start Message center took 0 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Start
[24-Jul-2020 18:35:18 INF] Starting AsyncAgent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer on managed thread 9
[24-Jul-2020 18:35:18 INF] Start local grain directory took 1 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Init implicit stream subscribe table took 2 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Create system targets and inject dependencies took 12 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.Silo started in stage RuntimeServices (4000) in 18.6763 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage RuntimeServices (4000) took 18.7516 Milliseconds
[24-Jul-2020 18:35:18 INF] HostedClient started in stage RuntimeGrainServices (8000) in 1.9141 Milliseconds
[24-Jul-2020 18:35:18 INF] MembershipOracle starting on host fhirserver-pod-6558c57dc4-pl8gr with SiloAddress S10.1.2.167:11111:333311717 at 2020-07-24 18:35:18.064 GMT
[24-Jul-2020 18:35:18 INF] Init grain services took 75 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Init type manager took 13 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Starting Windows perf counter stats collection with frequency=00:00:30
[24-Jul-2020 18:35:18 INF] Start silo statistics took 85 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Starting DeploymentLoadPublisher.
[24-Jul-2020 18:35:18 INF] Started DeploymentLoadPublisher.
[24-Jul-2020 18:35:18 INF] Start deployment load collector took 7 Milliseconds to finish
[24-Jul-2020 18:35:18 INF] Starting Silo Watchdog.
[24-Jul-2020 18:35:18 INF] Orleans.Runtime.Silo started in stage RuntimeGrainServices (8000) in 184.4005 Milliseconds
[24-Jul-2020 18:35:18 INF] ProcessTableUpdate (called from Refresh) membership table: 5 silos, 1 are Active, 3 are Dead, 1 are Joining, Version=<123, 123>. All silos: [SiloAddress=S10.1.2.168:11112:333311706 SiloName=Silo_095e7 Status=Joining, SiloAddress=S10.1.2.162:11111:333311335 SiloName=Silo_e5ba2 Status=Active, SiloAddress=S10.1.2.157:11111:333300013 SiloName=Silo_44d47 Status=Dead, SiloAddress=S10.1.2.158:11112:333300010 SiloName=Silo_c316b Status=Dead, SiloAddress=S10.1.2.163:11112:333311335 SiloName=Silo_c383c Status=Dead]
[24-Jul-2020 18:35:18 INF] MembershipTableManager started in stage RuntimeGrainServices (8000) in 295.7757 Milliseconds
[24-Jul-2020 18:35:18 INF] Starting lifecycle stage RuntimeGrainServices (8000) took 295.9127 Milliseconds
[24-Jul-2020 18:35:18 INF] SiloStatusListenerManager started in stage AfterRuntimeGrainServices (8100) in 0.0833 Milliseconds
[24-Jul-2020 18:35:18 INF] -Joining
[24-Jul-2020 18:35:18 INF] Added Server S10.1.2.162:11111:333311335/xF49D260A. Current view: [S10.1.2.167:11111:333311717 -> <MultiRange: Size=x65CCFB18, %Ring=39.766%>, S10.1.2.162:11111:333311335 -> <MultiRange: Size=x9A3304E8, %Ring=60.234%>]
[24-Jul-2020 18:35:18 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x100000000, %Ring=100.000%> new <MultiRange: Size=x65CCFB18, %Ring=39.766%> increased? True
[24-Jul-2020 18:35:18 INF] Expediting cluster type map refresh due to new silo, S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:19 INF] ProcessTableUpdate (called from TryUpdateMyStatusGlobalOnce) membership table: 6 silos, 1 are Active, 3 are Dead, 2 are Joining, Version=<124, 123>. All silos: [SiloAddress=S10.1.2.168:11112:333311706 SiloName=Silo_095e7 Status=Joining, SiloAddress=S10.1.2.167:11111:333311717 SiloName=Silo_820bf Status=Joining, SiloAddress=S10.1.2.162:11111:333311335 SiloName=Silo_e5ba2 Status=Active, SiloAddress=S10.1.2.157:11111:333300013 SiloName=Silo_44d47 Status=Dead, SiloAddress=S10.1.2.158:11112:333300010 SiloName=Silo_c316b Status=Dead, SiloAddress=S10.1.2.163:11112:333311335 SiloName=Silo_c383c Status=Dead]
[24-Jul-2020 18:35:19 INF] Gossiping S10.1.2.167:11111:333311717 status change to Joining to 1 partners
[24-Jul-2020 18:35:19 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:22 INF] MembershipAgent started in stage AfterRuntimeGrainServices (8100) in 3083.7579 Milliseconds
[24-Jul-2020 18:35:22 INF] Starting lifecycle stage AfterRuntimeGrainServices (8100) took 3084.4064 Milliseconds
[24-Jul-2020 18:35:22 INF] Orleans.Runtime.Versions.GrainVersionStore started in stage ApplicationServices (10000) in 0.2429 Milliseconds
[24-Jul-2020 18:35:22 INF] Starting lifecycle stage ApplicationServices (10000) took 0.3525 Milliseconds
[24-Jul-2020 18:35:22 INF] -BecomeActive
[24-Jul-2020 18:35:22 INF] Starting AsyncAgent Runtime.Messaging.GatewayClientCleanupAgent on managed thread 9
[24-Jul-2020 18:35:22 INF] Start gateway took 0 Milliseconds to finish
[24-Jul-2020 18:35:22 INF] Orleans.Runtime.Silo started in stage BecomeActive (19999) in 0.8027 Milliseconds
[24-Jul-2020 18:35:22 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:22 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:22 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:22 ERR] Exception while sending gossip notification to remote silo S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.MembershipService.MembershipSystemTarget.GossipToRemoteSilo(SiloAddress silo, MembershipTableSnapshot snapshot, SiloAddress updatedSilo, SiloStatus updatedStatus)
[24-Jul-2020 18:35:22 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:22 WRN] ProcessSiloAddEvent(S10.1.2.162:11111:333311335) failed, will be retried:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ProcessAddedSiloAsync(SiloAddress addedSilo, List`1 splitPartListSingle, List`1 splitPartListMulti)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ExecutePendingOperations().
[24-Jul-2020 18:35:22 WRN] Did not get ping response for ping #1/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:22 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #1
[24-Jul-2020 18:35:22 WRN] ProcessSiloAddEvent(S10.1.2.162:11111:333311335) failed, will NOT be retried:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.2.162:11111:333311335, will retry after 729.5555ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ProcessAddedSiloAsync(SiloAddress addedSilo, List`1 splitPartListSingle, List`1 splitPartListMulti)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ExecutePendingOperations()
[24-Jul-2020 18:35:23 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:25 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:25 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:25 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:26 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:27 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:28 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:28 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:28 WRN] Did not get ping response for ping #2/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:28 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:28 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #2
[24-Jul-2020 18:35:28 INF] Closing connections to defunct silo S10.1.2.157:11111:333300013
[24-Jul-2020 18:35:28 INF] Closing connections to defunct silo S10.1.2.163:11112:333311335
[24-Jul-2020 18:35:29 INF] Closing connections to defunct silo S10.1.2.158:11112:333300010
[24-Jul-2020 18:35:29 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:31 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:31 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:31 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:32 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:33 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:34 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:34 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:34 WRN] Did not get ping response for ping #3/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:34 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #3
[24-Jul-2020 18:35:34 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:35 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:37 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:37 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:37 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:38 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:39 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:40 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:41 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:41 WRN] Did not get ping response for ping #4/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:41 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:41 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #4
[24-Jul-2020 18:35:42 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:44 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:44 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:44 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:45 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:46 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:47 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:47 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:47 WRN] Did not get ping response for ping #5/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:47 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:47 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #5
[24-Jul-2020 18:35:48 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:50 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:50 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:50 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:51 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:52 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:53 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:53 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:53 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:53 WRN] Did not get ping response for ping #6/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:53 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #6
[24-Jul-2020 18:35:54 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:56 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:56 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:56 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:57 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:35:58 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:35:59 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:35:59 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:35:59 WRN] Did not get ping response for ping #7/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:35:59 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:35:59 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #7
[24-Jul-2020 18:36:00 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:02 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:02 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:02 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:03 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:04 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:05 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:05 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:05 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:05 WRN] Did not get ping response for ping #8/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:05 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #8
[24-Jul-2020 18:36:06 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:09 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:09 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:09 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:10 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:10 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:12 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:12 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:12 WRN] Did not get ping response for ping #9/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:12 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:12 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #9
[24-Jul-2020 18:36:13 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:15 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:15 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:15 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:16 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:17 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:18 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:18 WRN] Did not get ping response for ping #10/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:18 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:18 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:18 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #10
[24-Jul-2020 18:36:18 INF] Before collection#1: memory=14MB, #activations=0, collector=<#Activations=0, #Buckets=0, buckets=[]>.
[24-Jul-2020 18:36:18 INF] After collection#1: memory=14MB, #activations=0, collected 0 activations, collector=<#Activations=0, #Buckets=0, buckets=[]>, collection time=00:00:00.0045574.
[24-Jul-2020 18:36:18 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.2.162:11111:333311335, will retry after 667.3114ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:19 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:21 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:21 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:21 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:22 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 18:36:23 INF] About to send pings to 1 nodes in order to validate communication in the Joining state. Pinged nodes = [S10.1.2.162:11111:333311335]
[24-Jul-2020 18:36:24 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 18:36:24 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[24-Jul-2020 18:36:24 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[24-Jul-2020 18:36:24 WRN] Did not get ping response for ping #11/1 from S10.1.2.162:11111:333311335: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[24-Jul-2020 18:36:24 ERR] Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Silos which did not respond successfully are: [S10.1.2.162:11111:333311335]. Will continue attempting to validate connectivity until 07/24/2020 18:36:37. Attempt #11
[24-Jul-2020 18:36:25 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335

My Silo

.UseOrleans(builder =>
                {
                    builder
                        .UseAdoNetClustering(options =>
                        {
                            options.Invariant = "Npgsql";
                            options.ConnectionString = Environment.GetEnvironmentVariable("SILO_CLUSTERING_CS");
                        })
                        .ConfigureEndpoints(siloPort: 11111, gatewayPort: 30000, listenOnAnyHostAddress: true)
                        .Configure<ClusterOptions>(options =>
                        {
                            options.ClusterId = Environment.GetEnvironmentVariable("SILO_CLUSTER_ID");
                            options.ServiceId = Environment.GetEnvironmentVariable("SILO_SERVICE_ID");
                        })
                        .ConfigureApplicationParts(parts =>
                        {
                            parts.AddApplicationPart(typeof(ResourceGrain).Assembly).WithReferences();
                        });
                })

It seems that the old Silos are staying in the membership table forever as i stop/start kubernetes pods and causing Orleans to get stuck performing multiple re-tries for nonexistent IP's for around 5 or 6 times each, so the more rows there are in the table the longer it will take re-trying.

I noticed this can also be reproduced by using just normal docker containers in a docker-compose deployment

In this case, pods are crashing when the membership table contains 6 or more rows.

Is there any configuration i'm missing on my Orleans silo, maybe there is a special way of deploying Silos to a Kubernetes Pod ?

picture gabrielfreire

Most helpful comment

The labels also need to be set as orleans/serviceId and orleans/clusterId, unlike in the PR description

picture ReubenBond on 3 Sep 2020
👍2

All 20 comments

Do the logs end there? Are you able to provide the contents of the SQL table?

ReubenBond picture ReubenBond on 24 Jul 2020

I guess the logs end there, it was the last log i could get before the pod restarted

those are the rows at the moment 

"smartplatform_silo_cluster"    "10.1.2.173"    11112   333319975   "Silo_9ef08"    "patientservice-pod-788877d9f8-fjt6g"   3   30001       "2020-07-24 20:52:56.296"   "2020-07-24 20:52:59.073"
"smartplatform_silo_cluster"    "10.1.2.162"    11111   333311335   "Silo_e5ba2"    "fhirserver-pod-6558c57dc4-gbs7s"   6   30000   "10.1.2.173:11112@333319975,2020-07-24 20:53:11.178 GMT|10.1.2.172:11111@333319975,2020-07-24 20:53:11.451 GMT" "2020-07-24 18:28:56.08"    "2020-07-24 18:33:57.762"
"smartplatform_silo_cluster"    "10.1.2.158"    11112   333300010   "Silo_c316b"    "patientservice-pod-788877d9f8-nnqnm"   6   30001   "10.1.2.158:11112@333300010,2020-07-24 16:01:01.529 GMT"    "2020-07-24 15:20:10.976"   "2020-07-24 16:01:01.529"
"smartplatform_silo_cluster"    "10.1.2.157"    11111   333300013   "Silo_44d47"    "fhirserver-pod-6558c57dc4-9mhkl"   6   30000   "10.1.2.162:11111@333311335,2020-07-24 18:29:13.405 GMT|10.1.2.163:11112@333311335,2020-07-24 18:29:14.943 GMT" "2020-07-24 15:20:14.473"   "2020-07-24 16:00:16.858"
"smartplatform_silo_cluster"    "10.1.2.163"    11112   333311335   "Silo_c383c"    "patientservice-pod-788877d9f8-b4lvj"   6   30001   "10.1.2.163:11112@333311335,2020-07-24 18:34:22.839 GMT"    "2020-07-24 18:28:56.864"   "2020-07-24 18:34:22.839"
"smartplatform_silo_cluster"    "10.1.2.168"    11112   333311706   "Silo_095e7"    "patientservice-pod-788877d9f8-prs98"   6   30001   "10.1.2.168:11112@333311792,2020-07-24 18:36:34.871 GMT"    "2020-07-24 18:35:07.068"   "2020-07-24 18:35:09.261"
"smartplatform_silo_cluster"    "10.1.2.167"    11111   333311717   "Silo_820bf"    "fhirserver-pod-6558c57dc4-pl8gr"   6   30000   "10.1.2.167:11111@333311802,2020-07-24 18:36:43.706 GMT"    "2020-07-24 18:35:18.064"   "2020-07-24 18:35:18.983"
"smartplatform_silo_cluster"    "10.1.2.168"    11112   333311792   "Silo_d8310"    "patientservice-pod-788877d9f8-prs98"   6   30001   "10.1.2.168:11112@333311886,2020-07-24 18:38:09.110 GMT"    "2020-07-24 18:36:32.904"   "2020-07-24 18:36:34.965"
"smartplatform_silo_cluster"    "10.1.2.168"    11112   333311886   "Silo_96264"    "patientservice-pod-788877d9f8-prs98"   2   30001       "2020-07-24 18:38:07.362"   "2020-07-24 18:38:09.289"
"smartplatform_silo_cluster"    "10.1.2.167"    11111   333311802   "Silo_1aa20"    "fhirserver-pod-6558c57dc4-pl8gr"   6   30000   "10.1.2.167:11111@333311898,2020-07-24 18:38:20.395 GMT"    "2020-07-24 18:36:42.863"   "2020-07-24 18:36:43.778"
"smartplatform_silo_cluster"    "10.1.2.167"    11111   333311898   "Silo_25152"    "fhirserver-pod-6558c57dc4-pl8gr"   2   30000       "2020-07-24 18:38:18.814"   "2020-07-24 18:38:20.582"
"smartplatform_silo_cluster"    "10.1.2.172"    11111   333319975   "Silo_19c6a"    "fhirserver-pod-6558c57dc4-vb8c8"   3   30000       "2020-07-24 20:52:56.567"   "2020-07-24 20:52:58.067"

I just started my cluster to get the database contents for you and below are the silo logs

[24-Jul-2020 20:53:08 INF] Closing connections to defunct silo S10.1.2.167:11111:333311717
[24-Jul-2020 20:53:08 INF] Closing connections to defunct silo S10.1.2.168:11112:333311792
[24-Jul-2020 20:53:08 INF] Closing connections to defunct silo S10.1.2.167:11111:333311802
[24-Jul-2020 20:53:08 INF] Closing connections to defunct silo S10.1.2.163:11112:333311335
[24-Jul-2020 20:53:08 INF] Closing connections to defunct silo S10.1.2.158:11112:333300010
[24-Jul-2020 20:53:08 INF] Closing connections to defunct silo S10.1.2.168:11112:333311706
[24-Jul-2020 20:53:08 INF] Closing connections to defunct silo S10.1.2.157:11111:333300013
[24-Jul-2020 20:53:09 INF] Establishing connection to endpoint S10.1.2.162:11111:333311335
[24-Jul-2020 20:53:11 WRN] Connection attempt to endpoint S10.1.2.162:11111:333311335 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[24-Jul-2020 20:53:11 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.2.162:11111:333311335
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.2.162:11111:333311335. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.2.162:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)

after the above, it stops and connects normally, it's like an arbitrary issue, sometimes it gets stuck crashing and restarting the pods, but now, i just started it, it called a bunch of Closing connections to defunct silo and it's running great, weird

gabrielfreire picture gabrielfreire on 24 Jul 2020

Do you have a readiness probe or so? I had an issue a while ago, when kubernetes was restarting a pod before it was able to remove old entries from the membership table.

SebastianStehle picture SebastianStehle on 24 Jul 2020

hi @SebastianStehle , no i don't have anything more complex configured in my cluster, it's basic deployments with pods and services, some database volume configuration, i'm learning k8s and made the mistake of using an Orleans system as my hello world, if you have an example of how to configure such thing, it would be useful i couldn't find configuration examples for Orleans based applications anywhere =/

gabrielfreire picture gabrielfreire on 24 Jul 2020

I wonder if this happens with the latest release? There was a change which could help to prevent crash loops caused by TCP sockets being in CLOSE_WAIT for 2 minutes after they're closed.

What does kubectl describe pod <x> tell you was the last restart reason? I wonder what's killing them

ReubenBond picture ReubenBond on 15 Aug 2020

I also think that kubernetes and Orleans both have mechanisms that are interfering with each other.

Typically you want to have a relatively short liveness and readiness probe.

For example I was experiencing the following situation a while ago:

  1. Orleans hangs.
  2. Liveness probe was not responsive anymore.
  3. Silo/Pod was restarted from kubernetes.
  4. There was an member in the membership table which was not dead yet.
  5. Silo tried to communicate to the dead member.
  6. Liveness probe was not responsive during startup.
  7. Silo/Pod was restarted.
  8. -> STEP 4

It should be better now with the startup probes, but this is still a relatively new feature. But without them it is is not easy to find a good configuration.

SebastianStehle picture SebastianStehle on 15 Aug 2020

I would consider disabling liveness probes on a silo and allow other silos to vote it dead instead.
If you are going to have liveness probes, then make them some multiple of the silo liveness detection time, eg, 2 minutes.

Also, if you were to use StatefulSets, then your silo will restart with the same IP (and hopefully you use a fixed port), so it will not suffer from that issue, as it will mark its own predecessor as dead. I haven't tested running with StatefulSets.

I think there's more we can do here to make k8s hosting easier, eg allow specification of a stable identity, or consume k8s metadata from membership to unilaterally declare silos dead when k8s doesn't list them.

ReubenBond picture ReubenBond on 15 Aug 2020

If you disable liveness probe you cannot recover from deadlocks. Stateful sets do not help afaik, because the IP is not stable, only the host name. I think I asked for this before, to use host names in the membership table.

SebastianStehle picture SebastianStehle on 15 Aug 2020
👍2

@ReubenBond i don't get the reason from describe pod: that's my output:

All orleans packages v3.2.2

this time it stopped refreshing after 2 restarts

 NAME                                     READY   STATUS    RESTARTS   AGE
core-pod-85697879f5-p4v6x                1/1     Running   2          91m
eventstoredb-66469868c5-m9bt9            1/1     Running   0          91m
fhirserver-pod-6558c57dc4-v5nld          1/1     Running   2          91m
patientservice-pod-56bd9c97df-zx99c      1/1     Running   2          91m
postgres-fhirdatabase-7dd856f8c4-tnm7b   1/1     Running   0          91m
rabbitmq-77449f5bc9-s8kg9                1/1     Running   0          91m
ui-pod-89d678b7-8n25c                    1/1     Running   0          91m
gabrielfreiredev@DESKTOP-M95VEVA  ~\Documents\workspace\_SMARTPlatformApplications\Services\k8s   master ≢                                        [12:12]
❯ kubectl describe pod core-pod-85697879f5-p4v6x -n smartplatform
Name:         core-pod-85697879f5-p4v6x
Namespace:    smartplatform
Priority:     0
Node:         docker-desktop/192.168.65.3
Start Time:   Wed, 19 Aug 2020 10:40:55 +0100
Labels:       app=core-pod
              pod-template-hash=85697879f5
Annotations:  <none>
Status:       Running
IP:           10.1.3.173
IPs:
  IP:           10.1.3.173
Controlled By:  ReplicaSet/core-pod-85697879f5
Containers:
  core:
    Container ID:   docker://d68d8e7a07c34c58a546fc3a19af40302f006ee1450ac34f98aa5e4139ab71a8
    Image:          gabrielfreiredev/smartplatform-core:latest
    Image ID:       docker-pullable://gabrielfreiredev/smartplatform-core@sha256:be403ba248c26c65915d4b4955e576366eeaf8a8b3bc4658830e66a43830f896
    Ports:          80/TCP, 443/TCP, 30002/TCP, 11113/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Wed, 19 Aug 2020 10:44:19 +0100
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 19 Aug 2020 10:42:40 +0100
      Finished:     Wed, 19 Aug 2020 10:44:04 +0100
    Ready:          True
    Restart Count:  2
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment:
      SILO_CLUSTER_ID:                smartplatform_silo_cluster
      SILO_SERVICE_ID:                smartplatform_silo_service
      SILO_CLUSTERING_DATABASE_NAME:  smartfhirserverclustering.db
      SILO_CLUSTERING_CS:             Server=fhirdatabase-service;Port=5108;Database=smartfhirserverclustering.db;User ID=admin;Password=admin
      EVENTSTORE_URL:                 tcp://admin:changeit@eventstore-service:1113
      DATABASE_CS:                    Server=fhirdatabase-service;Port=5108;Database=smartplatformcore.db;User ID=admin;Password=admin
      RABBITMQ_HOST:                  rabbitmq-service
      RABBITMQ_USERNAME:              user
      RABBITMQ_PASSWORD:              password
      RABBITMQ_QUEUE:                 SMARTPlatform_Core
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ch2kb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-ch2kb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ch2kb
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

This is my membership table

"smartplatform_silo_cluster"    "10.1.3.172"    11112   335526063   "Silo_e5083"    "patientservice-pod-56bd9c97df-zx99c"   6   30001   "10.1.3.172:11112@335526150,2020-08-19 09:42:33.313 GMT"    "2020-08-19 09:41:04.527"   "2020-08-19 09:41:07.339"
"smartplatform_silo_cluster"    "10.1.3.171"    11111   335526063   "Silo_58d23"    "fhirserver-pod-6558c57dc4-v5nld"   6   30000   "10.1.3.171:11111@335526159,2020-08-19 09:42:43.439 GMT"    "2020-08-19 09:41:03.936"   "2020-08-19 09:41:18.206"
"smartplatform_silo_cluster"    "10.1.3.173"    11113   335526066   "Silo_9c794"    "core-pod-85697879f5-p4v6x" 6   30002   "10.1.3.173:11113@335526162,2020-08-19 09:42:47.419 GMT"    "2020-08-19 09:41:07.021"   "2020-08-19 09:41:18.908"
"smartplatform_silo_cluster"    "10.1.3.172"    11112   335526150   "Silo_a8694"    "patientservice-pod-56bd9c97df-zx99c"   6   30001   "10.1.3.172:11112@335526250,2020-08-19 09:44:13.713 GMT"    "2020-08-19 09:42:31.318"   "2020-08-19 09:42:33.512"
"smartplatform_silo_cluster"    "10.1.3.171"    11111   335526159   "Silo_05e9c"    "fhirserver-pod-6558c57dc4-v5nld"   6   30000   "10.1.3.171:11111@335526260,2020-08-19 09:44:25.913 GMT"    "2020-08-19 09:42:40.017"   "2020-08-19 09:42:43.605"
"smartplatform_silo_cluster"    "10.1.3.173"    11113   335526162   "Silo_168b8"    "core-pod-85697879f5-p4v6x" 6   30002   "10.1.3.173:11113@335526262,2020-08-19 09:44:29.817 GMT"    "2020-08-19 09:42:43.453"   "2020-08-19 09:42:47.609"
"smartplatform_silo_cluster"    "10.1.3.164"    11111   335523838   "Silo_f897a"    "fhirserver-pod-6558c57dc4-9qj9l"   6   30000   "10.1.3.172:11112@335526250,2020-08-19 09:44:39.366 GMT|10.1.3.171:11111@335526260,2020-08-19 09:44:41.804 GMT" "2020-08-19 09:03:59.733"   "2020-08-19 09:34:20.952"
"smartplatform_silo_cluster"    "10.1.3.167"    11112   335523842   "Silo_33a17"    "patientservice-pod-56bd9c97df-sxcpx"   6   30001   "10.1.3.172:11112@335526250,2020-08-19 09:44:40.036 GMT|10.1.3.173:11113@335526262,2020-08-19 09:44:42.611 GMT" "2020-08-19 09:04:02.951"   "2020-08-19 09:34:05.409"
"smartplatform_silo_cluster"    "10.1.3.172"    11112   335526250   "Silo_69fd3"    "patientservice-pod-56bd9c97df-zx99c"   3   30001       "2020-08-19 09:44:11.435"   "2020-08-19 11:14:23.065"
"smartplatform_silo_cluster"    "10.1.3.171"    11111   335526260   "Silo_a1d9e"    "fhirserver-pod-6558c57dc4-v5nld"   3   30000       "2020-08-19 09:44:21.52"    "2020-08-19 11:14:26.653"
"smartplatform_silo_cluster"    "10.1.3.166"    11113   335523842   "Silo_56b49"    "core-pod-85697879f5-5lv5q" 6   30002   "10.1.3.173:11113@335526262,2020-08-19 09:44:52.548 GMT|10.1.3.172:11112@335526250,2020-08-19 09:44:55.635 GMT" "2020-08-19 09:04:02.842"   "2020-08-19 09:34:22.04"
"smartplatform_silo_cluster"    "10.1.3.173"    11113   335526262   "Silo_5d646"    "core-pod-85697879f5-p4v6x" 3   30002       "2020-08-19 09:44:23.138"   "2020-08-19 11:14:30.714"

Logs

[19-Aug-2020 09:44:21 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:21 INF] Establishing connection to endpoint S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:21 INF] Establishing connection to endpoint S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:22 WRN] Noticed that silo S10.1.3.167:11112:335523842 has not updated it's IAmAliveTime table column recently. Last update was at 08/19/2020 09:34:05, now is 08/19/2020 09:44:22, no update for 00:10:16.8021592, which is more than 00:10:00.
[19-Aug-2020 09:44:22 WRN] Noticed that silo S10.1.3.166:11113:335523842 has not updated it's IAmAliveTime table column recently. Last update was at 08/19/2020 09:34:22, now is 08/19/2020 09:44:22, no update for 00:10:00.1714569, which is more than 00:10:00.
[19-Aug-2020 09:44:22 WRN] Noticed that silo S10.1.3.164:11111:335523838 has not updated it's IAmAliveTime table column recently. Last update was at 08/19/2020 09:34:20, now is 08/19/2020 09:44:22, no update for 00:10:01.2594901, which is more than 00:10:00.
[19-Aug-2020 09:44:22 WRN] Noticed that silo S10.1.3.167:11112:335523842 has not updated it's IAmAliveTime table column recently. Last update was at 08/19/2020 09:34:05, now is 08/19/2020 09:44:22, no update for 00:10:16.8042359, which is more than 00:10:00.
[19-Aug-2020 09:44:22 WRN] Noticed that silo S10.1.3.166:11113:335523842 has not updated it's IAmAliveTime table column recently. Last update was at 08/19/2020 09:34:22, now is 08/19/2020 09:44:22, no update for 00:10:00.1733793, which is more than 00:10:00.
[19-Aug-2020 09:44:22 WRN] Noticed that silo S10.1.3.164:11111:335523838 has not updated it's IAmAliveTime table column recently. Last update was at 08/19/2020 09:34:20, now is 08/19/2020 09:44:22, no update for 00:10:01.2614131, which is more than 00:10:00.
[19-Aug-2020 09:44:22 INF] ProcessTableUpdate (called from TryUpdateMyStatusGlobalOnce) membership table: 9 silos, 4 are Active, 3 are Dead, 2 are Joining, Version=<84, 83>. All silos: [SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Joining, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Joining, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Active, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead]
[19-Aug-2020 09:44:22 INF] -Finished BecomeActive.
[19-Aug-2020 09:44:22 INF] MembershipAgent started in stage BecomeActive (19999) in 5179.2571 Milliseconds
[19-Aug-2020 09:44:22 INF] Starting lifecycle stage BecomeActive (19999) took 5179.8718 Milliseconds
[19-Aug-2020 09:44:22 INF] ClusterHealthMonitor started in stage Active (20000) in 0.2467 Milliseconds
[19-Aug-2020 09:44:22 INF] MembershipTableCleanupAgent started in stage Active (20000) in 0.0956 Milliseconds
[19-Aug-2020 09:44:22 INF] GatewayConnectionListener started in stage Active (20000) in 2.1032 Milliseconds
[19-Aug-2020 09:44:22 INF] Orleans.Runtime.Silo started in stage Active (20000) in 3.0899 Milliseconds
[19-Aug-2020 09:44:22 INF] Starting lifecycle stage Active (20000) took 5.3527 Milliseconds
[19-Aug-2020 09:44:22 INF] Orleans Silo started.
[19-Aug-2020 09:44:22 INF] Application started. Press Ctrl+C to shut down.
[19-Aug-2020 09:44:22 INF] Hosting environment: Production
[19-Aug-2020 09:44:22 INF] Content root path: /app
[19-Aug-2020 09:44:22 INF] Will watch (actively ping) 3 silos: [S10.1.3.166:11113:335523842, S10.1.3.167:11112:335523842, S10.1.3.164:11111:335523838]
[19-Aug-2020 09:44:23 WRN] Connection attempt to endpoint S10.1.3.167:11112:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:23 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:23 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:23 WRN] Connection attempt to endpoint S10.1.3.164:11111:335523838 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:23 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:23 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:23 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:23 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:23 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.164:11111:335523838. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:24 INF] Closing connections to defunct silo S10.1.3.172:11112:335526063
[19-Aug-2020 09:44:24 INF] Closing connections to defunct silo S10.1.3.171:11111:335526063
[19-Aug-2020 09:44:24 INF] Closing connections to defunct silo S10.1.3.173:11113:335526066
[19-Aug-2020 09:44:24 INF] Closing connections to defunct silo S10.1.3.172:11112:335526150
[19-Aug-2020 09:44:24 WRN] Did not get ping response for ping #4/1 from S10.1.3.167:11112:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 35.9754ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:24 WRN] Did not get ping response for ping #5/1 from S10.1.3.164:11111:335523838: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.164:11111:335523838, will retry after 47.2124ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:24 WRN] Did not get ping response for ping #3/1 from S10.1.3.166:11113:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 46.119ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:24 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:24 INF] Establishing connection to endpoint S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:24 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.164:11111:335523838, will retry after 1.2754ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:24 INF] Establishing connection to endpoint S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:26 INF] Connection Local: 10.1.3.172:11112, Remote: 10.1.3.171:35140, ConnectionId: 0HM244HR11FGJ established with S10.1.3.171:11111:335526260
[19-Aug-2020 09:44:26 INF] Received cluster membership snapshot via gossip: [Version: 85, 11 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Joining, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Created, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Active, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:26 INF] Received cluster membership snapshot via gossip: [Version: 86, 11 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Joining, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Joining, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Active, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:26 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:26 WRN] Connection attempt to endpoint S10.1.3.164:11111:335523838 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:26 WRN] Connection attempt to endpoint S10.1.3.167:11112:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:26 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:26 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:26 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:26 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:26 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:26 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 997.6261ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:26 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 997.6261ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:26 INF] Received cluster membership snapshot via gossip: [Version: 87, 11 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Joining, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Active, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:26 INF] Added Server S10.1.3.171:11111:335526260/x54450C89. Current view: [S10.1.3.172:11112:335526250 -> <MultiRange: Size=x2F057510, %Ring=18.368%>, S10.1.3.171:11111:335526260 -> <MultiRange: Size=x2FF23D28, %Ring=18.729%>, S10.1.3.164:11111:335523838 -> <MultiRange: Size=x31184E09, %Ring=19.178%>, S10.1.3.167:11112:335523842 -> <MultiRange: Size=x37B113B8, %Ring=21.755%>, S10.1.3.166:11113:335523842 -> <MultiRange: Size=x383EEC07, %Ring=21.971%>]
[19-Aug-2020 09:44:26 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x434C0C1B, %Ring=26.288%> new <MultiRange: Size=x2F057510, %Ring=18.368%> increased? True
[19-Aug-2020 09:44:26 INF] Expediting cluster type map refresh due to new silo, S10.1.3.171:11111:335526260
[19-Aug-2020 09:44:26 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 905.8289ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:26 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 905.8289ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:26 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.164:11111:335523838, will retry after 904.3561ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:26 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 905.4381ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:26 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 904.6949ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:27 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:27 INF] Establishing connection to endpoint S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:27 INF] Establishing connection to endpoint S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:29 WRN] Connection attempt to endpoint S10.1.3.164:11111:335523838 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:29 WRN] Did not get ping response for ping #8/2 from S10.1.3.164:11111:335523838: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.164:11111:335523838. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:29 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:29 WRN] Did not get ping response for ping #6/2 from S10.1.3.166:11113:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:29 WRN] Connection attempt to endpoint S10.1.3.167:11112:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:29 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:29 WRN] Did not get ping response for ping #7/2 from S10.1.3.167:11112:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:29 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:29 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.164:11111:335523838. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:29 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:29 WRN] RegisterAsync - It seems we are not the owner of activation S10.1.3.171:11111:335526260*cli/5df7f08e@77c212a1, trying to forward it to S10.1.3.167:11112:335523842 (hopCount=1)
[19-Aug-2020 09:44:29 WRN] Exception registering activations in AcceptExistingRegistrations:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 771.0789ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.AcceptExistingRegistrationsAsync(List`1 singleActivations, List`1 multiActivations)
[19-Aug-2020 09:44:29 WRN] AcceptExistingRegistrations failed, will be retried:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 771.0789ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.AcceptExistingRegistrationsAsync(List`1 singleActivations, List`1 multiActivations)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ExecutePendingOperations().
[19-Aug-2020 09:44:30 WRN] RegisterAsync - It seems we are not the owner of activation S10.1.3.171:11111:335526260*cli/5df7f08e@77c212a1, trying to forward it to S10.1.3.167:11112:335523842 (hopCount=1)
[19-Aug-2020 09:44:30 WRN] Exception registering activations in AcceptExistingRegistrations:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 314.0708ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.AcceptExistingRegistrationsAsync(List`1 singleActivations, List`1 multiActivations)
[19-Aug-2020 09:44:30 INF] Connection Local: 10.1.3.172:11112, Remote: 10.1.3.173:56354, ConnectionId: 0HM244HR11FGK established with S10.1.3.173:11113:335526262
[19-Aug-2020 09:44:30 WRN] AcceptExistingRegistrations failed, will NOT be retried:
Exc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 314.0708ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.AcceptExistingRegistrationsAsync(List`1 singleActivations, List`1 multiActivations)
   at Orleans.Runtime.GrainDirectory.GrainDirectoryHandoffManager.ExecutePendingOperations()
[19-Aug-2020 09:44:30 INF] Received cluster membership snapshot via gossip: [Version: 88, 12 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Dead, SiloAddress=S10.1.3.173:11113:335526262 SiloName=Silo_5d646 Status=Created, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Active, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:30 INF] Received cluster membership snapshot via gossip: [Version: 89, 12 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Dead, SiloAddress=S10.1.3.173:11113:335526262 SiloName=Silo_5d646 Status=Joining, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Active, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:30 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 195.5008ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:30 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 195.5008ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:30 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 190.4831ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:30 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 190.4831ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:30 INF] Received cluster membership snapshot via gossip: [Version: 90, 12 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Dead, SiloAddress=S10.1.3.173:11113:335526262 SiloName=Silo_5d646 Status=Active, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Active, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:30 INF] Added Server S10.1.3.173:11113:335526262/x9D350D33. Current view: [S10.1.3.172:11112:335526250 -> <MultiRange: Size=x264AC15A, %Ring=14.958%>, S10.1.3.164:11111:335523838 -> <MultiRange: Size=x274CF1EA, %Ring=15.352%>, S10.1.3.173:11113:335526262 -> <MultiRange: Size=x2A8DE952, %Ring=16.623%>, S10.1.3.171:11111:335526260 -> <MultiRange: Size=x2A8EE7B5, %Ring=16.624%>, S10.1.3.166:11113:335523842 -> <MultiRange: Size=x2E1D95F5, %Ring=18.014%>, S10.1.3.167:11112:335523842 -> <MultiRange: Size=x2F2DE5C0, %Ring=18.429%>]
[19-Aug-2020 09:44:30 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x2F057510, %Ring=18.368%> new <MultiRange: Size=x264AC15A, %Ring=14.958%> increased? True
[19-Aug-2020 09:44:30 INF] Expediting cluster type map refresh due to new silo, S10.1.3.173:11113:335526262
[19-Aug-2020 09:44:30 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 17.9689ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:30 INF] Will watch (actively ping) 3 silos: [S10.1.3.173:11113:335526262, S10.1.3.167:11112:335523842, S10.1.3.164:11111:335523838]
[19-Aug-2020 09:44:30 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 17.9689ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:30 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.164:11111:335523838, will retry after 12.7169ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:30 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 16.0089ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:30 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 17.2172ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:30 INF] Establishing connection to endpoint S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:30 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 2.5946ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:30 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.167:11112:335523842, will retry after 3.8229ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:31 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:31 INF] Establishing connection to endpoint S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:32 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:32 WRN] Connection attempt to endpoint S10.1.3.167:11112:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:32 WRN] Connection attempt to endpoint S10.1.3.164:11111:335523838 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:32 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 2.435ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:32 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:32 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:32 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.164:11111:335523838. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:32 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.164:11111:335523838. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:33 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:33 INF] Establishing connection to endpoint S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:33 INF] Establishing connection to endpoint S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:33 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.164:11111:335523838, will retry after 0.4656ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:35 WRN] Connection attempt to endpoint S10.1.3.164:11111:335523838 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:35 WRN] Did not get ping response for ping #11/3 from S10.1.3.164:11111:335523838: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.164:11111:335523838. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:35 WRN] Silo S10.1.3.164:11111:335523838 failed 3 probes and is suspected of being dead. Publishing a death vote.
[19-Aug-2020 09:44:35 INF] -Putting my vote to mark silo S10.1.3.164:11111:335523838 as DEAD #2. Previous suspect list is [], trying to update to [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:35.880 GMT>], eTag=, freshVotes is []
[19-Aug-2020 09:44:36 INF] Closing connections to defunct silo S10.1.3.171:11111:335526159
[19-Aug-2020 09:44:36 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:36 WRN] Connection attempt to endpoint S10.1.3.167:11112:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:36 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:36 WRN] Did not get ping response for ping #10/3 from S10.1.3.167:11112:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:36 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:36 WRN] Silo S10.1.3.167:11112:335523842 failed 3 probes and is suspected of being dead. Publishing a death vote.
[19-Aug-2020 09:44:36 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:36 INF] -Putting my vote to mark silo S10.1.3.167:11112:335523842 as DEAD #2. Previous suspect list is [], trying to update to [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:36.930 GMT>], eTag=, freshVotes is []
[19-Aug-2020 09:44:37 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:37 INF] Establishing connection to endpoint S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:37 INF] Establishing connection to endpoint S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:38 WRN] Connection attempt to endpoint S10.1.3.164:11111:335523838 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:38 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.164:11111:335523838. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:39 WRN] Did not get ping response for ping #14/4 from S10.1.3.164:11111:335523838: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.164:11111:335523838, will retry after 630.9873ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:39 WRN] Silo S10.1.3.164:11111:335523838 failed 4 probes and is suspected of being dead. Publishing a death vote.
[19-Aug-2020 09:44:39 INF] -Putting my vote to mark silo S10.1.3.164:11111:335523838 as DEAD #2. Previous suspect list is [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:35.880 GMT>], trying to update to [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:39.366 GMT>], eTag=, freshVotes is [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:35.880 GMT>]
[19-Aug-2020 09:44:40 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:40 WRN] Connection attempt to endpoint S10.1.3.167:11112:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:40 WRN] Did not get ping response for ping #13/4 from S10.1.3.167:11112:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:40 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.167:11112:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:40 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:40 WRN] Silo S10.1.3.167:11112:335523842 failed 4 probes and is suspected of being dead. Publishing a death vote.
[19-Aug-2020 09:44:40 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:40 INF] -Putting my vote to mark silo S10.1.3.167:11112:335523842 as DEAD #2. Previous suspect list is [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:36.930 GMT>], trying to update to [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:40.036 GMT>], eTag=, freshVotes is [<S10.1.3.172:11112:335526250, 2020-08-19 09:44:36.930 GMT>]
[19-Aug-2020 09:44:40 INF] Closing connections to defunct silo S10.1.3.173:11113:335526162
[19-Aug-2020 09:44:41 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:41 INF] Establishing connection to endpoint S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:41 INF] Establishing connection to endpoint S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:41 INF] Received cluster membership snapshot via gossip: [Version: 95, 12 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Dead, SiloAddress=S10.1.3.173:11113:335526262 SiloName=Silo_5d646 Status=Active, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Active, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Dead, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:41 INF] Will watch (actively ping) 4 silos: [S10.1.3.166:11113:335523842, S10.1.3.173:11113:335526262, S10.1.3.171:11111:335526260, S10.1.3.167:11112:335523842]
[19-Aug-2020 09:44:41 INF] Removed Server S10.1.3.164:11111:335523838/xE9F438D7. Current view: [S10.1.3.172:11112:335526250 -> <MultiRange: Size=x2B7AC70E, %Ring=16.984%>, S10.1.3.166:11113:335523842 -> <MultiRange: Size=x335F3F97, %Ring=20.067%>, S10.1.3.171:11111:335526260 -> <MultiRange: Size=x342FCFBC, %Ring=20.385%>, S10.1.3.167:11112:335523842 -> <MultiRange: Size=x35D7D23F, %Ring=21.032%>, S10.1.3.173:11113:335526262 -> <MultiRange: Size=x371E5760, %Ring=21.531%>]
[19-Aug-2020 09:44:41 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x264AC15A, %Ring=14.958%> new <MultiRange: Size=x2B7AC70E, %Ring=16.984%> increased? True
[19-Aug-2020 09:44:41 WRN] The target silo became unavailable for message: Request S10.1.3.172:11112:335526250*stg/22/00000016@S00000016->S10.1.3.164:11111:335523838*stg/22/00000016@S00000016 #111. Target History is: <S10.1.3.164:11111:335523838:*stg/22/00000016:@S00000016>. See https://aka.ms/orleans-troubleshooting for troubleshooting help. About to break its promise.
[19-Aug-2020 09:44:41 WRN] The target silo became unavailable for message: Request S10.1.3.172:11112:335526250*stg/17/00000011@S00000011->S10.1.3.164:11111:335523838*stg/17/00000011@S00000011 #115. Target History is: <S10.1.3.164:11111:335523838:*stg/17/00000011:@S00000011>. See https://aka.ms/orleans-troubleshooting for troubleshooting help. About to break its promise.
[19-Aug-2020 09:44:41 INF] Catalog is deactivating 0 activations due to a failure of silo S10.1.3.164:11111:335523838/xE9F438D7, since it is a primary directory partition to these grain ids.
[19-Aug-2020 09:44:41 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.164:11111:335523838
Orleans.Runtime.SiloUnavailableException: The target silo became unavailable for message: Request S10.1.3.172:11112:335526250*stg/17/00000011@S00000011->S10.1.3.164:11111:335523838*stg/17/00000011@S00000011 #115. Target History is: <S10.1.3.164:11111:335523838:*stg/17/00000011:@S00000011>. See https://aka.ms/orleans-troubleshooting for troubleshooting help.
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:42 WRN] Connection attempt to endpoint S10.1.3.164:11111:335523838 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.164:11111. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:42 INF] Received cluster membership snapshot via gossip: [Version: 96, 12 silos, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526066 SiloName=Silo_9c794 Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Dead, SiloAddress=S10.1.3.173:11113:335526262 SiloName=Silo_5d646 Status=Active, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.171:11111:335526063 SiloName=Silo_58d23 Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Dead, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Dead, SiloAddress=S10.1.3.172:11112:335526063 SiloName=Silo_e5083 Status=Dead, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active]
[19-Aug-2020 09:44:42 INF] Will watch (actively ping) 3 silos: [S10.1.3.166:11113:335523842, S10.1.3.173:11113:335526262, S10.1.3.171:11111:335526260]
[19-Aug-2020 09:44:42 INF] Removed Server S10.1.3.167:11112:335523842/x00AE0315. Current view: [S10.1.3.172:11112:335526250 -> <MultiRange: Size=x3139F791, %Ring=19.229%>, S10.1.3.171:11111:335526260 -> <MultiRange: Size=x4154DDF2, %Ring=25.520%>, S10.1.3.166:11113:335523842 -> <MultiRange: Size=x4560B6DB, %Ring=27.101%>, S10.1.3.173:11113:335526262 -> <MultiRange: Size=x481073A2, %Ring=28.150%>]
[19-Aug-2020 09:44:42 WRN] The target silo became unavailable for message: Request S10.1.3.172:11112:335526250*stg/22/00000016@S00000016->S10.1.3.167:11112:335523842*stg/22/00000016@S00000016 #110. Target History is: <S10.1.3.167:11112:335523842:*stg/22/00000016:@S00000016>. See https://aka.ms/orleans-troubleshooting for troubleshooting help. About to break its promise.
[19-Aug-2020 09:44:42 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x2B7AC70E, %Ring=16.984%> new <MultiRange: Size=x3139F791, %Ring=19.229%> increased? True
[19-Aug-2020 09:44:42 WRN] The target silo became unavailable for message: Request S10.1.3.172:11112:335526250*stg/17/00000011@S00000011->S10.1.3.167:11112:335523842*stg/17/00000011@S00000011 #114. Target History is: <S10.1.3.167:11112:335523842:*stg/17/00000011:@S00000011>. See https://aka.ms/orleans-troubleshooting for troubleshooting help. About to break its promise.
[19-Aug-2020 09:44:42 INF] Catalog is deactivating 0 activations due to a failure of silo S10.1.3.167:11112:335523842/x00AE0315, since it is a primary directory partition to these grain ids.
[19-Aug-2020 09:44:42 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.167:11112:335523842
Orleans.Runtime.SiloUnavailableException: The target silo became unavailable for message: Request S10.1.3.172:11112:335526250*stg/17/00000011@S00000011->S10.1.3.167:11112:335523842*stg/17/00000011@S00000011 #114. Target History is: <S10.1.3.167:11112:335523842:*stg/17/00000011:@S00000011>. See https://aka.ms/orleans-troubleshooting for troubleshooting help.
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:43 WRN] Connection attempt to endpoint S10.1.3.167:11112:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.167:11112. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:43 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:43 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:43 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:43 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:43 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:43 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 998.3995ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:43 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.1.3.166:11113:335523842, will retry after 998.3995ms
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectionEntry.ThrowIfRecentConnectionFailure()
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:44 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:46 WRN] Noticed that silo S10.1.3.166:11113:335523842 has not updated it's IAmAliveTime table column recently. Last update was at 08/19/2020 09:34:22, now is 08/19/2020 09:44:46, no update for 00:10:24.1971582, which is more than 00:10:00.
[19-Aug-2020 09:44:46 INF] ProcessTableUpdate (called from Refresh) membership table: 9 silos, 4 are Active, 5 are Dead, Version=<97, 97>. All silos: [SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Active, SiloAddress=S10.1.3.173:11113:335526262 SiloName=Silo_5d646 Status=Active, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Dead, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Dead]
[19-Aug-2020 09:44:46 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:46 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:46 WRN] Did not get ping response for ping #15/1 from S10.1.3.166:11113:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:46 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:46 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:46 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
[19-Aug-2020 09:44:47 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:49 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:49 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:49 WRN] Did not get ping response for ping #18/2 from S10.1.3.166:11113:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:49 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:50 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:51 INF] Closing connections to defunct silo S10.1.3.164:11111:335523838
[19-Aug-2020 09:44:52 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:52 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:52 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:52 INF] Closing connections to defunct silo S10.1.3.167:11112:335523842
[19-Aug-2020 09:44:53 INF] Establishing connection to endpoint S10.1.3.166:11113:335523842
[19-Aug-2020 09:44:55 WRN] Connection attempt to endpoint S10.1.3.166:11113:335523842 failed with exception Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
[19-Aug-2020 09:44:55 WRN] An exception was thrown by PublishStatistics.UpdateRuntimeStatistics(). Ignoring.
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.DeploymentLoadPublisher.PublishStatistics(Object _)
[19-Aug-2020 09:44:55 WRN] Did not get ping response for ping #21/3 from S10.1.3.166:11113:335523842: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
   at Orleans.Runtime.MembershipService.SiloHealthMonitor.Probe(Int32 diagnosticProbeNumber, CancellationToken cancellation)
[19-Aug-2020 09:44:55 WRN] Silo S10.1.3.166:11113:335523842 failed 3 probes and is suspected of being dead. Publishing a death vote.
[19-Aug-2020 09:44:55 ERR] Exception when trying to get GrainInterfaceMap for silos S10.1.3.166:11113:335523842
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.1.3.166:11113:335523842. See InnerException
 ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.1.3.166:11113. Error: HostUnreachable
   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken)
   at Orleans.Internal.OrleansTaskExtentions.MakeCancellable[T](Task`1 task, CancellationToken cancellationToken)
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   --- End of inner exception stack trace ---
   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint)
   at Orleans.Runtime.Messaging.OutboundMessageQueue.<SendMessage>g__SendAsync|10_0(ValueTask`1 c, Message m)
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
   at Orleans.Runtime.TypeManager.GetTargetSiloGrainInterfaceMap(SiloAddress siloAddress)
[19-Aug-2020 09:44:55 INF] -Going to mark silo S10.1.3.166:11113:335523842 as DEAD in the table #1. I am the last voter: #freshVotes=1, myVoteIndex = -1, NumVotesForDeathDeclaration=2 , #activeSilos=4, suspect list=[<S10.1.3.173:11113:335526262, 2020-08-19 09:44:52.548 GMT>]
[19-Aug-2020 09:44:55 INF] Catalog is deactivating 0 activations due to a failure of silo S10.1.3.166:11113:335523842/x28BEB115, since it is a primary directory partition to these grain ids.
[19-Aug-2020 09:44:55 INF] Will watch (actively ping) 2 silos: [S10.1.3.173:11113:335526262, S10.1.3.171:11111:335526260]
[19-Aug-2020 09:44:55 INF] ProcessTableUpdate (called from DeclareDead) membership table: 9 silos, 3 are Active, 6 are Dead, Version=<100, 100>. All silos: [SiloAddress=S10.1.3.173:11113:335526262 SiloName=Silo_5d646 Status=Active, SiloAddress=S10.1.3.172:11112:335526250 SiloName=Silo_69fd3 Status=Active, SiloAddress=S10.1.3.171:11111:335526260 SiloName=Silo_a1d9e Status=Active, SiloAddress=S10.1.3.172:11112:335526150 SiloName=Silo_a8694 Status=Dead, SiloAddress=S10.1.3.171:11111:335526159 SiloName=Silo_05e9c Status=Dead, SiloAddress=S10.1.3.173:11113:335526162 SiloName=Silo_168b8 Status=Dead, SiloAddress=S10.1.3.164:11111:335523838 SiloName=Silo_f897a Status=Dead, SiloAddress=S10.1.3.167:11112:335523842 SiloName=Silo_33a17 Status=Dead, SiloAddress=S10.1.3.166:11113:335523842 SiloName=Silo_56b49 Status=Dead]
[19-Aug-2020 09:44:55 INF] Gossiping S10.1.3.166:11113:335523842 status change to Dead to 2 partners
[19-Aug-2020 09:44:55 INF] Removed Server S10.1.3.166:11113:335523842/x28BEB115. Current view: [S10.1.3.172:11112:335526250 -> <MultiRange: Size=x438C24F7, %Ring=26.386%>, S10.1.3.171:11111:335526260 -> <MultiRange: Size=x5274E112, %Ring=32.210%>, S10.1.3.173:11113:335526262 -> <MultiRange: Size=x69FEF9F7, %Ring=41.405%>]
[19-Aug-2020 09:44:55 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x3139F791, %Ring=19.229%> new <MultiRange: Size=x438C24F7, %Ring=26.386%> increased? True
[19-Aug-2020 09:44:55 INF] Silo S10.1.3.172:11112:335526250 is rejecting message to known-dead silo S10.1.3.166:11113:335523842: Request S10.1.3.172:11112:335526250*stg/13/0000000d@S0000000d->S10.1.3.166:11113:335523842*stg/10/0000000a@S0000000a #148
[19-Aug-2020 09:44:55 ERR] Directory.RegisterAsync S10.1.3.172:11112:335526250*cli/0d1ea8ae@954e8a9b failed.
Orleans.Runtime.OrleansMessageRejectionException: Target silo is known to be dead
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
[19-Aug-2020 09:44:55 ERR] OnClientRefreshTimer has thrown an exceptions.
Orleans.Runtime.OrleansMessageRejectionException: Target silo is known to be dead
   at Orleans.Internal.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
   at Orleans.Internal.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
   at Orleans.Runtime.ClientObserverRegistrar.OnRefreshClients()
gabrielfreire picture gabrielfreire on 19 Aug 2020

hey @ReubenBond , i'm very excited to see that the Kubernetes PR was merged, great job, is it available in any preview release ? if not, will it be ?

thanks for that, amazing work.

gabrielfreire picture gabrielfreire on 3 Sep 2020

Hi @gabrielfreire, I'm glad to hear it, yes it is available in a preview release: https://www.nuget.org/packages/Microsoft.Orleans.Hosting.Kubernetes

The intention is to have it marked as -beta when 3.3.0 final is released

ReubenBond picture ReubenBond on 3 Sep 2020
🚀1

awesome!!! thanks

gabrielfreire picture gabrielfreire on 3 Sep 2020

Hi @ReubenBond
I believe there is a bug somewhere as i'm getting restarts again but it seems to be for another reason

below i provide some information, the logs and configuration are the same for all services(co-hosted orleans silo in asp.net core apps)

but they use different ports:

  • service1: 30000/11111
  • service2: 30001/11112
  • service3: 30002/11113

membership table after a couple of restarts

"smartplatform_silo_cluster"    "10.1.4.84" 11113   336829793   "core-pod-dbdcb6d9f-hvrph"  "core-pod-dbdcb6d9f-hvrph"  6   30002   "10.1.4.82:11111@336829815,2020-09-03 11:50:23.112 GMT" "2020-09-03 11:49:54.007"   "2020-09-03 11:50:02.972"
"smartplatform_silo_cluster"    "10.1.4.82" 11111   336829815   "fhirserver-pod-7c7599f7b7-mxrjf"   "fhirserver-pod-7c7599f7b7-mxrjf"   6   30000   "10.1.4.83:11112@336829872,2020-09-03 11:51:15.883 GMT" "2020-09-03 11:50:16.401"   "2020-09-03 11:50:23.272"
"smartplatform_silo_cluster"    "10.1.4.83" 11112   336829872   "patientservice-pod-6c64f9dbbf-jrghr"   "patientservice-pod-6c64f9dbbf-jrghr"   6   30001   "10.1.4.84:11113@336829892,2020-09-03 11:51:41.870 GMT" "2020-09-03 11:51:12.578"   "2020-09-03 11:51:16.07"
"smartplatform_silo_cluster"    "10.1.4.83" 11112   336830254   "patientservice-pod-6c64f9dbbf-jrghr"   "patientservice-pod-6c64f9dbbf-jrghr"   6   30001   "10.1.4.84:11113@336830321,2020-09-03 11:58:51.967 GMT" "2020-09-03 11:57:35.468"   "2020-09-03 11:57:38.965"
"smartplatform_silo_cluster"    "10.1.4.84" 11113   336829892   "core-pod-dbdcb6d9f-hvrph"  "core-pod-dbdcb6d9f-hvrph"  6   30002   "10.1.4.82:11111@336829919,2020-09-03 11:52:06.982 GMT" "2020-09-03 11:51:33.291"   "2020-09-03 11:51:42.069"
"smartplatform_silo_cluster"    "10.1.4.84" 11113   336830321   "core-pod-dbdcb6d9f-hvrph"  "core-pod-dbdcb6d9f-hvrph"  3   30002       "2020-09-03 11:58:42.186"   "2020-09-03 11:58:52.171"
"smartplatform_silo_cluster"    "10.1.4.82" 11111   336829919   "fhirserver-pod-7c7599f7b7-mxrjf"   "fhirserver-pod-7c7599f7b7-mxrjf"   6   30000   "10.1.4.83:11112@336829958,2020-09-03 11:52:42.012 GMT" "2020-09-03 11:52:00.384"   "2020-09-03 11:52:07.07"
"smartplatform_silo_cluster"    "10.1.4.83" 11112   336829719   "patientservice-pod-6c64f9dbbf-jrghr"   "patientservice-pod-6c64f9dbbf-jrghr"   6   30001   "10.1.4.82:11111@336829718,2020-09-03 11:48:47.372 GMT" "2020-09-03 11:48:40.406"   "2020-09-03 11:48:43.986"
"smartplatform_silo_cluster"    "10.1.4.82" 11111   336829718   "fhirserver-pod-7c7599f7b7-mxrjf"   "fhirserver-pod-7c7599f7b7-mxrjf"   6   30000   "10.1.4.84:11113@336829721,2020-09-03 11:48:49.868 GMT" "2020-09-03 11:48:39.608"   "2020-09-03 11:48:47.474"
"smartplatform_silo_cluster"    "10.1.4.83" 11112   336829958   "patientservice-pod-6c64f9dbbf-jrghr"   "patientservice-pod-6c64f9dbbf-jrghr"   6   30001   "10.1.4.84:11113@336829988,2020-09-03 11:53:15.882 GMT" "2020-09-03 11:52:38.684"   "2020-09-03 11:52:42.182"
"smartplatform_silo_cluster"    "10.1.4.84" 11113   336829988   "core-pod-dbdcb6d9f-hvrph"  "core-pod-dbdcb6d9f-hvrph"  6   30002   "10.1.4.82:11111@336830040,2020-09-03 11:54:06.478 GMT" "2020-09-03 11:53:09.211"   "2020-09-03 11:53:15.972"
"smartplatform_silo_cluster"    "10.1.4.84" 11113   336829721   "core-pod-dbdcb6d9f-hvrph"  "core-pod-dbdcb6d9f-hvrph"  6   30002   "10.1.4.82:11111@336829739,2020-09-03 11:49:04.892 GMT" "2020-09-03 11:48:42.396"   "2020-09-03 11:48:50.268"
"smartplatform_silo_cluster"    "10.1.4.82" 11111   336829739   "fhirserver-pod-7c7599f7b7-mxrjf"   "fhirserver-pod-7c7599f7b7-mxrjf"   6   30000   "10.1.4.83:11112@336829772,2020-09-03 11:49:36.086 GMT" "2020-09-03 11:48:59.696"   "2020-09-03 11:49:04.976"
"smartplatform_silo_cluster"    "10.1.4.83" 11112   336829772   "patientservice-pod-6c64f9dbbf-jrghr"   "patientservice-pod-6c64f9dbbf-jrghr"   6   30001   "10.1.4.84:11113@336829793,2020-09-03 11:50:02.811 GMT" "2020-09-03 11:49:33.097"   "2020-09-03 11:49:36.173"
"smartplatform_silo_cluster"    "10.1.4.82" 11111   336830040   "fhirserver-pod-7c7599f7b7-mxrjf"   "fhirserver-pod-7c7599f7b7-mxrjf"   6   30000   "10.1.4.83:11112@336830072,2020-09-03 11:54:37.170 GMT" "2020-09-03 11:54:01.198"   "2020-09-03 11:54:06.672"
"smartplatform_silo_cluster"    "10.1.4.83" 11112   336830072   "patientservice-pod-6c64f9dbbf-jrghr"   "patientservice-pod-6c64f9dbbf-jrghr"   6   30001   "10.1.4.84:11113@336830098,2020-09-03 11:55:07.980 GMT" "2020-09-03 11:54:33.4" "2020-09-03 11:54:37.274"
"smartplatform_silo_cluster"    "10.1.4.84" 11113   336830098   "core-pod-dbdcb6d9f-hvrph"  "core-pod-dbdcb6d9f-hvrph"  6   30002   "10.1.4.82:11111@336830195,2020-09-03 11:56:42.587 GMT" "2020-09-03 11:54:59.216"   "2020-09-03 11:55:08.066"
"smartplatform_silo_cluster"    "10.1.4.82" 11111   336830195   "fhirserver-pod-7c7599f7b7-mxrjf"   "fhirserver-pod-7c7599f7b7-mxrjf"   6   30000   "10.1.4.83:11112@336830254,2020-09-03 11:57:38.808 GMT" "2020-09-03 11:56:36.168"   "2020-09-03 11:56:42.676"

My config

                    // Set kubernetes environment variables
                    Environment.SetEnvironmentVariable("ORLEANS_SERVICE_ID", ctx.Configuration.GetSection("OrleansConfiguration:SiloClustering:ServiceId").Value);
                    Environment.SetEnvironmentVariable("ORLEANS_CLUSTER_ID", ctx.Configuration.GetSection("OrleansConfiguration:SiloClustering:ClusterId").Value);

                    var _isKubernetesHosted = false;
                    var _podIp = Environment.GetEnvironmentVariable("POD_IP");
                    var _podName = Environment.GetEnvironmentVariable("POD_NAME");
                    var _podNamespace = Environment.GetEnvironmentVariable("POD_NAMESPACE");

                    if (!string.IsNullOrEmpty(_podIp) &&
                        !string.IsNullOrEmpty(_podName) &&
                        !string.IsNullOrEmpty(_podNamespace))
                    {
                        Log.Information("|------- KUBERNETES HOSTING ------- |");
                        Log.Information($"| - POD NAME:        {_podName}     |");
                        Log.Information($"| - POD IP:          {_podIp}       |");
                        Log.Information($"| - POD NAMESPACE:   {_podNamespace}|");
                        Log.Information("|------- KUBERNETES HOSTING ------- |");
                        _isKubernetesHosted = true;
                    }

                    // MS Orleans Silo host
                    // See: https://dotnet.github.io/orleans/Documentation/index.html
                    if (_isKubernetesHosted)
                        builder.UseKubernetesHosting();

                    builder
                        .UseAdoNetClustering(options =>......

Pods

core-pod-dbdcb6d9f-hvrph                 0/1     CrashLoopBackOff   5          13m
eventstoredb-66469868c5-6zsrz            1/1     Running            0          93m
fhirserver-pod-7c7599f7b7-mxrjf          1/1     Running            6          13m
patientservice-pod-6c64f9dbbf-jrghr      0/1     CrashLoopBackOff   5          13m
postgres-fhirdatabase-7dd856f8c4-lrvwq   1/1     Running            0          93m
rabbitmq-77449f5bc9-zlvr9                1/1     Running            0          93m
ui-pod-556f648bd7-svzgw                  1/1     Running            0          24m

Describe patientservice-pod

kubectl describe pod patientservice-pod-6c64f9dbbf-jrghr -n smartplatform
Name:         patientservice-pod-6c64f9dbbf-jrghr
Namespace:    smartplatform
Priority:     0
Node:         docker-desktop/192.168.65.3
Start Time:   Thu, 03 Sep 2020 12:48:33 +0100
Labels:       app=patientservice-pod
              pod-template-hash=6c64f9dbbf
Annotations:  <none>
Status:       Running
IP:           10.1.4.83
IPs:
  IP:           10.1.4.83
Controlled By:  ReplicaSet/patientservice-pod-6c64f9dbbf
Containers:
  patientserviceapi:
    Container ID:   docker://a936615625d6dd416f0f5e00eb151c4812742e4ce6152878667ccc7bdd75d2fe
    Image:          gabrielfreiredev/smartplatform-patientserviceapi:latest
    Image ID:       docker-pullable://gabrielfreiredev/smartplatform-patientserviceapi@sha256:4e016ad14434af60b53e34c9b92bdb1af0f1fa44e441490420dcddd824b53f5b
    Ports:          80/TCP, 443/TCP, 30001/TCP, 11112/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 03 Sep 2020 12:52:36 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Thu, 03 Sep 2020 12:51:10 +0100
      Finished:     Thu, 03 Sep 2020 12:52:09 +0100
    Ready:          True
    Restart Count:  3
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment:
      OrleansConfiguration__SiloClustering__ClusterId:                           smartplatform_silo_cluster
      OrleansConfiguration__SiloClustering__ServiceId:                           smartplatform_silo_service
      OrleansConfiguration__SiloClustering__ClusteringDatabaseName:              smartfhirserverclustering.db
      OrleansConfiguration__SiloClustering__ClusteringDatabaseConnectionString:  Server=fhirdatabase-service;Port=5108;Database=smartfhirserverclustering.db;User ID=xxx;Password=xxx
      ConnectionStrings__DefaultConnection:                                      Server=fhirdatabase-service;Port=5108;Database=smartfhirserver.db;User ID=xxx;Password=xxx
      RabbitMQConnection__HostUrl:                                               rabbitmq-service
      RabbitMQConnection__Username:                                              user
      RabbitMQConnection__Password:                                              xxx
      RabbitMQConnection__Queue:                                                 SMARTPlatform_PatientService
      KubernetesClusterSettings__ServiceName:                                    patient
      EventStoreDB__ConnectionString:                                            tcp://admin:XXX@eventstore-service:1113
      FHIR_SERVER_URL:                                                           http://xxx
      POD_NAMESPACE:                                                             smartplatform (v1:metadata.namespace)
      POD_NAME:                                                                  patientservice-pod-6c64f9dbbf-jrghr (v1:metadata.name)
      POD_IP:                                                                     (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ch2kb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-ch2kb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ch2kb
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                     Message
  ----     ------     ----                 ----                     -------
  Normal   Scheduled  <unknown>            default-scheduler        Successfully assigned smartplatform/patientservice-pod-6c64f9dbbf-jrghr to docker-desktop
  Warning  BackOff    40s (x3 over 2m6s)   kubelet, docker-desktop  Back-off restarting failed container
  Normal   Pulling    28s (x4 over 4m28s)  kubelet, docker-desktop  Pulling image "gabrielfreiredev/smartplatform-patientserviceapi:latest"
  Normal   Pulled     26s (x4 over 4m25s)  kubelet, docker-desktop  Successfully pulled image "gabrielfreiredev/smartplatform-patientserviceapi:latest"
  Normal   Created    26s (x4 over 4m25s)  kubelet, docker-desktop  Created container patientserviceapi
  Normal   Started    26s (x4 over 4m25s)  kubelet, docker-desktop  Started container patientserviceapi

Logs

[03-Sep-2020 11:48:56 INF] |------- KUBERNETES HOSTING ------- |
[03-Sep-2020 11:48:56 INF] | - POD NAME:        fhirserver-pod-7c7599f7b7-mxrjf     |
[03-Sep-2020 11:48:56 INF] | - POD IP:          10.1.4.82       |
[03-Sep-2020 11:48:56 INF] | - POD NAMESPACE:   smartplatform|
[03-Sep-2020 11:48:56 INF] |------- KUBERNETES HOSTING ------- |
[03-Sep-2020 11:48:57 INF] Entity Framework Core 3.1.7 initialized 'WriteApplicationDbContext' using provider 'Npgsql.EntityFrameworkCore.PostgreSQL' with options: MigrationsAssembly=Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
[03-Sep-2020 11:48:58 INF] Executed DbCommand (52ms) [Parameters=[], CommandType='Text', CommandTimeout='30']

                    SELECT CASE WHEN COUNT(*) = 0 THEN FALSE ELSE TRUE END
                    FROM information_schema.tables
                    WHERE table_type = 'BASE TABLE' AND table_schema NOT IN ('pg_catalog', 'information_schema')
[03-Sep-2020 11:48:58 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Initializing PostgreOrleansStorage()
[03-Sep-2020 11:48:58 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - @ EnsureCreated()
ConnectionString: Server=fhirdatabase-service;Port=5108;Database=smartfhirserver.db;User ID=admin;Password=admin
[03-Sep-2020 11:48:58 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Database smartfhirserverclustering.db already exists!
[03-Sep-2020 11:48:58 INF] POSTGRESQL_ORLEANS_CLUSTERING_STORAGE - Updated connection string!
[03-Sep-2020 11:48:58 WRN] Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
[03-Sep-2020 11:48:58 INF] User profile is available. Using '/root/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
[03-Sep-2020 11:48:59 INF] Silo starting with GC settings: ServerGC=False GCLatencyMode=Interactive
[03-Sep-2020 11:48:59 WRN] Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
[03-Sep-2020 11:48:59 WRN] Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
[03-Sep-2020 11:48:59 INF] -------------- Initializing silo on host fhirserver-pod-7c7599f7b7-mxrjf MachineName fhirserver-pod-7c7599f7b7-mxrjf at 10.1.4.82:11111, gen 336829739 --------------
[03-Sep-2020 11:48:59 INF] Starting silo fhirserver-pod-7c7599f7b7-mxrjf
[03-Sep-2020 11:48:59 WRN] No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
[03-Sep-2020 11:48:59 INF] Starting VirtualBucketsRingProvider on silo S10.1.4.82:11111:336829739/xFDA760E1.
[03-Sep-2020 11:48:59 INF] Added Server S10.1.4.82:11111:336829739/xFDA760E1. Current view: [S10.1.4.82:11111:336829739 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
[03-Sep-2020 11:48:59 INF] -NotifyLocalRangeSubscribers about old <(0 0], Size=x100000000, %Ring=100.000%> new <MultiRange: Size=x100000000, %Ring=100.000%> increased? True
[03-Sep-2020 11:48:59 INF] -------------- Started silo S10.1.4.82:11111:336829739, ConsistentHashCode FDA760E1 --------------
[03-Sep-2020 11:49:00 INF] Creating key {70667e0b-6a01-4050-b0fd-a8a44fd13324} with creation date 2020-09-03 11:49:00Z, activation date 2020-09-03 11:49:00Z, and expiration date 2020-12-02 11:49:00Z.
[03-Sep-2020 11:49:00 WRN] No XML encryptor configured. Key {70667e0b-6a01-4050-b0fd-a8a44fd13324} may be persisted to storage in unencrypted form.
[03-Sep-2020 11:49:00 INF] Writing data to file '/root/.aspnet/DataProtection-Keys/key-70667e0b-6a01-4050-b0fd-a8a44fd13324.xml'.
[03-Sep-2020 11:49:03 INF] Now listening on: http://[::]:80
[03-Sep-2020 11:49:03 INF] Starting Orleans Silo.
[03-Sep-2020 11:49:03 INF] Stage First (-2147483648): Orleans.Runtime.SiloOptionsLogger
[03-Sep-2020 11:49:03 INF] Stage 1999: ConnectionManager, SiloConnectionListener, GatewayConnectionListener
[03-Sep-2020 11:49:03 INF] Stage RuntimeInitialize (2000): Orleans.Runtime.InsideRuntimeClient, SiloStatusListenerManager, ClusterMembershipService, SiloConnectionMaintainer, Orleans.Runtime.Silo
[03-Sep-2020 11:49:03 INF] Stage 2001: MembershipAgent
[03-Sep-2020 11:49:03 INF] Stage RuntimeServices (4000): Orleans.Runtime.Silo
[03-Sep-2020 11:49:03 INF] Stage RuntimeGrainServices (8000): MembershipTableManager, HostedClient, Orleans.Runtime.Silo
[03-Sep-2020 11:49:03 INF] Stage AfterRuntimeGrainServices (8100): KubernetesClusterAgent, MembershipAgent, SiloStatusListenerManager
[03-Sep-2020 11:49:03 INF] Stage ApplicationServices (10000): Orleans.Runtime.Versions.GrainVersionStore
[03-Sep-2020 11:49:03 INF] Stage BecomeActive (19999): MembershipAgent, IncomingRequestMonitor, Orleans.Runtime.Silo
[03-Sep-2020 11:49:03 INF] Stage Active (20000): ClusterHealthMonitor, MembershipTableCleanupAgent, GatewayConnectionListener, Orleans.Runtime.Silo
[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.ActivationCountBasedPlacementOptions:
ChooseOutOf: 2

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.ClusterMembershipOptions:
NumMissedTableIAmAliveLimit: 2
LivenessEnabled: True
ProbeTimeout: 00:00:05
TableRefreshTimeout: 00:01:00
DeathVoteExpirationTimeout: 00:02:00
IAmAliveTablePublishTimeout: 00:05:00
MaxJoinAttemptTime: 00:05:00
ValidateInitialConnectivity: True
UseLivenessGossip: True
NumProbedSilos: 3
NumMissedProbesLimit: 3
NumVotesForDeathDeclaration: 2
DefunctSiloExpiration: 7.00:00:00
DefunctSiloCleanupPeriod: 01:00:00
IsRunningAsUnitTest: False

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.ClusterOptions:
ClusterId: smartplatform_silo_cluster
ServiceId: smartplatform_silo_service

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.ConnectionOptions:
ProtocolVersion: Version1
ConnectionsPerEndpoint: 1
ConnectionRetryDelay: 00:00:01
OpenConnectionTimeout: 00:00:05

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.ConsistentRingOptions:
NumVirtualBucketsConsistentRing: 30
UseVirtualBucketsConsistentRing: True

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.CustomStorageLogConsistencyOptions-CustomStorage:
PrimaryCluster:

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.EndpointOptions:
AdvertisedIPAddress: 10.1.4.82
SiloPort: 11111
GatewayPort: 30000
SiloListeningEndpoint: 0.0.0.0:11111
GatewayListeningEndpoint: 0.0.0.0:30000

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.GrainCollectionOptions:
CollectionQuantum: 00:01:00
CollectionAge: 02:00:00
DeactivationTimeout: 00:00:30

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.GrainDirectoryOptions:
CachingStrategy: Adaptive
CacheSize: 1000000
InitialCacheTTL: 00:00:30
MaximumCacheTTL: 00:04:00
CacheTTLExtensionFactor: 2
LazyDeregistrationDelay: 00:01:00

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.GrainVersioningOptions:
DefaultCompatibilityStrategy: BackwardCompatible
DefaultVersionSelectorStrategy: AllCompatibleVersions

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.LoadSheddingOptions:
LoadSheddingEnabled: False
LoadSheddingLimit: 95

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.PerformanceTuningOptions:
DefaultConnectionLimit: 200
Expect100Continue: False
UseNagleAlgorithm: False
MinDotNetThreadPoolSize: 0
MinIOThreadPoolSize: 0

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.SchedulingOptions:
PerformDeadlockDetection: False
AllowCallChainReentrancy: True
MaxActiveThreads: 4
DelayWarningThreshold: 00:00:10
ActivationSchedulingQuantum: 00:00:00.1000000
TurnWarningLengthThreshold: 00:00:00.2000000
MaxPendingWorkItemsSoftLimit: 0
EnableWorkerThreadInjection: False
StoppedActivationWarningInterval: 00:01:00

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.SerializationProviderOptions:
FallbackSerializationProvider:
MaxSustainedSerializationContextCapacity: 64

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.SiloMessagingOptions:
SiloSenderQueues: 1
GatewaySenderQueues: 1
MaxForwardCount: 2
ClientDropTimeout: 00:01:00
ClientRegistrationRefresh: 00:05:00
ClientGatewayShutdownNotificationTimeout: 00:00:05
MaxEnqueuedRequestsSoftLimit: 0
MaxEnqueuedRequestsHardLimit: 0
MaxEnqueuedRequestsSoftLimit_StatelessWorker: 0
MaxEnqueuedRequestsHardLimit_StatelessWorker: 0
MaxRequestProcessingTime: 02:00:00
AssumeHomogenousSilosForTesting: False
DEFAULT_SHUTDOWN_REROUTE_TIMEOUT: 00:00:10
ShutdownRerouteTimeout: 00:00:10
SystemResponseTimeout: 00:00:30
GrainWorkloadAnalysisPeriod: 00:00:05
RequestProcessingWarningTime: 00:00:05
RequestQueueDelayWarningTime: 00:00:10
ResponseTimeout: 00:00:30
ResponseTimeoutWithDebugger: 00:30:00
DropExpiredMessages: True
BufferPoolBufferSize: 4096
BufferPoolMaxSize: 10000
BufferPoolPreallocationSize: 250
PropagateActivityId: False
LargeMessageWarningThreshold: 85000
MaxMessageHeaderSize: 10485760
MaxMessageBodySize: 104857600

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.SiloOptions:
SiloName: fhirserver-pod-7c7599f7b7-mxrjf

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.StatisticsOptions:
PerfCountersWriteInterval: 00:00:30
LogWriteInterval: 00:05:00
CollectionLevel: Info

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.TelemetryOptions:

[03-Sep-2020 11:49:03 INF] Configuration Orleans.Configuration.TypeManagementOptions:
TypeMapRefreshInterval: 00:01:00

[03-Sep-2020 11:49:03 INF] Orleans.Runtime.SiloOptionsLogger started in stage First (-2147483648) in 10.0208 Milliseconds
[03-Sep-2020 11:49:03 INF] Starting lifecycle stage First (-2147483648) took 10.9058 Milliseconds
[03-Sep-2020 11:49:03 INF] ConnectionManager started in stage 1999 in 0.0236 Milliseconds
[03-Sep-2020 11:49:03 INF] GatewayConnectionListener started in stage 1999 in 55.8687 Milliseconds
[03-Sep-2020 11:49:03 INF] SiloConnectionListener started in stage 1999 in 57.558 Milliseconds
[03-Sep-2020 11:49:03 INF] Starting lifecycle stage 1999 took 57.9536 Milliseconds
[03-Sep-2020 11:49:03 INF] Loaded grain type summary for 9 types:
Grain class Infrastructure.Grains.Infrastructure.Grains.ResourceGrain [1059605650 (0x3F284C92)] from Infrastructure.dll implementing interfaces: SMARTPlatform.Messaging.GrainInterfaces.Fhir.IResourceGrain [-1455888765 (0xA938E683)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.LogConsistency.Orleans.LogConsistency.LogConsistentGrain<TView> [-319513858 (0xECF49AFE)] from Orleans.Core.dll implementing interfaces:
Grain class Orleans.Runtime.Development.Orleans.Runtime.Development.DevelopmentLeaseProviderGrain [940524075 (0x380F422B)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Development.IDevelopmentLeaseProviderGrain [-1234611069 (0xB6695483)]
Grain class Orleans.Runtime.Management.Orleans.Runtime.Management.ManagementGrain [1954798034 (0x7483D9D2)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.IManagementGrain [-1734666656 (0x989B1660)]
Grain class Orleans.Runtime.ReminderService.Orleans.Runtime.ReminderService.GrainBasedReminderTable [-55315191 (0xFCB3F509)] from Orleans.Runtime.dll implementing interfaces: Orleans.IReminderTableGrain [-1135060418 (0xBC585A3E)]
Grain class Orleans.Runtime.Versions.Orleans.Runtime.Versions.VersionStoreGrain [331003096 (0x13BAB4D8)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Versions.IVersionStoreGrain [-297600501 (0xEE42FA0B)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.Streams.Orleans.Streams.PubSubRendezvousGrain [1903070868 (0x716E8E94)] from Orleans.Runtime.dll implementing interfaces: Orleans.Streams.IPubSubRendezvousGrain [1746702088 (0x681C8F08)]
Grain class SMARTPlatform.Messaging.EventSourcing.SMARTPlatform.Messaging.EventSourcing.EventSourcedGrain<TState> [1851261123 (0x6E5800C3)] from SMARTPlatform.Messaging.dll implementing interfaces: Orleans.MultiCluster.ILogConsistencyProtocolParticipant [-287197741 (0xEEE1B5D3)]
Grain class SMARTPlatform.Messaging.EventSourcing.SMARTPlatform.Messaging.EventSourcing.EventStoreGrain [1571370876 (0x5DA9377C)] from SMARTPlatform.Messaging.dll implementing interfaces: SMARTPlatform.Messaging.GrainInterfaces.IEventStoreGrain [-175072744 (0xF5909A18)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]

[03-Sep-2020 11:49:03 INF] Start InsideRuntimeClient took 6 Milliseconds
[03-Sep-2020 11:49:03 INF] Orleans.Runtime.InsideRuntimeClient started in stage RuntimeInitialize (2000) in 7.2105 Milliseconds
[03-Sep-2020 11:49:03 INF] SiloStatusListenerManager started in stage RuntimeInitialize (2000) in 0.0315 Milliseconds
[03-Sep-2020 11:49:03 INF] ClusterMembershipService started in stage RuntimeInitialize (2000) in 0.106 Milliseconds
[03-Sep-2020 11:49:03 INF] SiloConnectionMaintainer started in stage RuntimeInitialize (2000) in 0.6647 Milliseconds
[03-Sep-2020 11:49:03 INF] Silo Start()
[03-Sep-2020 11:49:03 INF] Configured .NET ServicePointManager to Expect100Continue=False, DefaultConnectionLimit=200, UseNagleAlgorithm=False to improve Azure storage performance.
[03-Sep-2020 11:49:03 INF] ConfigureThreadPoolAndServicePointSettings took 0 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Orleans.Runtime.Silo started in stage RuntimeInitialize (2000) in 1.4624 Milliseconds
[03-Sep-2020 11:49:03 INF] Starting lifecycle stage RuntimeInitialize (2000) took 9.7665 Milliseconds
[03-Sep-2020 11:49:03 INF] MembershipAgent started in stage 2001 in 0.0359 Milliseconds
[03-Sep-2020 11:49:03 INF] Starting lifecycle stage 2001 took 0.0799 Milliseconds
[03-Sep-2020 11:49:03 INF] Start Message center took 0 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Start
[03-Sep-2020 11:49:03 INF] Starting AsyncAgent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer on managed thread 10
[03-Sep-2020 11:49:03 INF] Start local grain directory took 1 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Init implicit stream subscribe table took 4 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Create system targets and inject dependencies took 85 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Orleans.Runtime.Silo started in stage RuntimeServices (4000) in 93.2466 Milliseconds
[03-Sep-2020 11:49:03 INF] Starting lifecycle stage RuntimeServices (4000) took 93.317 Milliseconds
[03-Sep-2020 11:49:03 INF] MembershipOracle starting on host fhirserver-pod-7c7599f7b7-mxrjf with SiloAddress S10.1.4.82:11111:336829739 at 2020-09-03 11:48:59.696 GMT
[03-Sep-2020 11:49:03 INF] HostedClient started in stage RuntimeGrainServices (8000) in 1.8037 Milliseconds
[03-Sep-2020 11:49:03 INF] Init grain services took 1 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Init type manager took 94 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Starting Windows perf counter stats collection with frequency=00:00:30
[03-Sep-2020 11:49:03 INF] Start silo statistics took 0 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Starting DeploymentLoadPublisher.
[03-Sep-2020 11:49:03 INF] Started DeploymentLoadPublisher.
[03-Sep-2020 11:49:03 INF] Start deployment load collector took 8 Milliseconds to finish
[03-Sep-2020 11:49:03 INF] Starting Silo Watchdog.
[03-Sep-2020 11:49:03 INF] Orleans.Runtime.Silo started in stage RuntimeGrainServices (8000) in 192.1385 Milliseconds
[03-Sep-2020 11:49:03 INF] ProcessTableUpdate (called from Refresh) membership table: 3 silos, 1 are Active, 2 are Dead, Version=<8, 8>. All silos: [SiloAddress=S10.1.4.84:11113:336829721 SiloName=core-pod-dbdcb6d9f-hvrph Status=Active, SiloAddress=S10.1.4.83:11112:336829719 SiloName=patientservice-pod-6c64f9dbbf-jrghr Status=Dead, SiloAddress=S10.1.4.82:11111:336829718 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Dead]
[03-Sep-2020 11:49:03 WRN] Silo fhirserver-pod-7c7599f7b7-mxrjf restarted on same host fhirserver-pod-7c7599f7b7-mxrjf New silo address = S10.1.4.82:11111:336829739 Previous silo address = S10.1.4.82:11111:336829718
[03-Sep-2020 11:49:03 INF] MembershipTableManager started in stage RuntimeGrainServices (8000) in 298.7714 Milliseconds
[03-Sep-2020 11:49:03 INF] Starting lifecycle stage RuntimeGrainServices (8000) took 298.8484 Milliseconds
[03-Sep-2020 11:49:03 INF] SiloStatusListenerManager started in stage AfterRuntimeGrainServices (8100) in 0.0937 Milliseconds
[03-Sep-2020 11:49:03 INF] -Joining
[03-Sep-2020 11:49:03 INF] Added Server S10.1.4.84:11113:336829721/xF9EB13ED. Current view: [S10.1.4.84:11113:336829721 -> <MultiRange: Size=x539264A0, %Ring=32.645%>, S10.1.4.82:11111:336829739 -> <MultiRange: Size=xAC6D9B60, %Ring=67.355%>]
[03-Sep-2020 11:49:03 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=x100000000, %Ring=100.000%> new <MultiRange: Size=xAC6D9B60, %Ring=67.355%> increased? True
[03-Sep-2020 11:49:03 INF] Expediting cluster type map refresh due to new silo, S10.1.4.84:11113:336829721
[03-Sep-2020 11:49:03 INF] ProcessTableUpdate (called from TryUpdateMyStatusGlobalOnce) membership table: 4 silos, 1 are Active, 2 are Dead, 1 are Joining, Version=<9, 8>. All silos: [SiloAddress=S10.1.4.84:11113:336829721 SiloName=core-pod-dbdcb6d9f-hvrph Status=Active, SiloAddress=S10.1.4.82:11111:336829739 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Joining, SiloAddress=S10.1.4.82:11111:336829718 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Dead, SiloAddress=S10.1.4.83:11112:336829719 SiloName=patientservice-pod-6c64f9dbbf-jrghr Status=Dead]
[03-Sep-2020 11:49:03 INF] Gossiping S10.1.4.82:11111:336829739 status change to Joining to 1 partners
[03-Sep-2020 11:49:03 INF] Establishing connection to endpoint S10.1.4.84:11113:336829721
[03-Sep-2020 11:49:03 WRN] Task [Id=1, Status=RanToCompletion] in WorkGroup [SystemTarget: S10.1.4.82:11111:336829739*stg/20/00000014@S00000014] took elapsed time 0:00:00.2144302 for execution, which is longer than 00:00:00.2000000. Running on thread 17
[03-Sep-2020 11:49:04 INF] Connected to endpoint S10.1.4.84:11113:336829721
[03-Sep-2020 11:49:04 INF] Connection Local: 10.1.4.82:47112, Remote: 10.1.4.84:11113, ConnectionId: 0HM2FVTDLHO0A established with S10.1.4.84:11113:336829721
[03-Sep-2020 11:49:04 INF] MembershipAgent started in stage AfterRuntimeGrainServices (8100) in 907.3475 Milliseconds
[03-Sep-2020 11:49:04 WRN] Silo S10.1.4.84:11113:336829721/core-pod-dbdcb6d9f-hvrph/Active does not correspond to any known pod. Marking it as dead.
[03-Sep-2020 11:49:04 INF] Going to mark silo S10.1.4.84:11113:336829721 dead as a result of a call to TryKill
[03-Sep-2020 11:49:04 INF] ProcessTableUpdate (called from DeclareDead) membership table: 4 silos, 0 are Active, 3 are Dead, 1 are Joining, Version=<10, 10>. All silos: [SiloAddress=S10.1.4.82:11111:336829739 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Joining, SiloAddress=S10.1.4.83:11112:336829719 SiloName=patientservice-pod-6c64f9dbbf-jrghr Status=Dead, SiloAddress=S10.1.4.82:11111:336829718 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Dead, SiloAddress=S10.1.4.84:11113:336829721 SiloName=core-pod-dbdcb6d9f-hvrph Status=Dead]
[03-Sep-2020 11:49:04 INF] KubernetesClusterAgent started in stage AfterRuntimeGrainServices (8100) in 1228.1704 Milliseconds
[03-Sep-2020 11:49:04 INF] Starting lifecycle stage AfterRuntimeGrainServices (8100) took 1228.251 Milliseconds
[03-Sep-2020 11:49:04 INF] Orleans.Runtime.Versions.GrainVersionStore started in stage ApplicationServices (10000) in 0.2285 Milliseconds
[03-Sep-2020 11:49:04 INF] Starting lifecycle stage ApplicationServices (10000) took 0.3103 Milliseconds
[03-Sep-2020 11:49:04 INF] IncomingRequestMonitor started in stage BecomeActive (19999) in 0.0881 Milliseconds
[03-Sep-2020 11:49:04 INF] Starting AsyncAgent Runtime.Messaging.GatewayClientCleanupAgent on managed thread 17
[03-Sep-2020 11:49:04 INF] Start gateway took 0 Milliseconds to finish
[03-Sep-2020 11:49:04 INF] Orleans.Runtime.Silo started in stage BecomeActive (19999) in 0.5553 Milliseconds
[03-Sep-2020 11:49:04 INF] -BecomeActive
[03-Sep-2020 11:49:04 INF] Catalog is deactivating 0 activations due to a failure of silo S10.1.4.84:11113:336829721/xF9EB13ED, since it is a primary directory partition to these grain ids.
[03-Sep-2020 11:49:04 INF] Removed Server S10.1.4.84:11113:336829721/xF9EB13ED. Current view: [S10.1.4.82:11111:336829739 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
[03-Sep-2020 11:49:04 INF] -NotifyLocalRangeSubscribers about old <MultiRange: Size=xAC6D9B60, %Ring=67.355%> new <MultiRange: Size=x100000000, %Ring=100.000%> increased? True
[03-Sep-2020 11:49:04 INF] Waiting for Kubernetes monitoring to be enabled
[03-Sep-2020 11:49:04 INF] ProcessTableUpdate (called from TryUpdateMyStatusGlobalOnce) membership table: 4 silos, 1 are Active, 3 are Dead, Version=<11, 10>. All silos: [SiloAddress=S10.1.4.82:11111:336829739 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Active, SiloAddress=S10.1.4.83:11112:336829719 SiloName=patientservice-pod-6c64f9dbbf-jrghr Status=Dead, SiloAddress=S10.1.4.82:11111:336829718 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Dead, SiloAddress=S10.1.4.84:11113:336829721 SiloName=core-pod-dbdcb6d9f-hvrph Status=Dead]
[03-Sep-2020 11:49:04 INF] -Finished BecomeActive.
[03-Sep-2020 11:49:04 INF] Enabling Kubernetes monitoring
[03-Sep-2020 11:49:04 INF] MembershipAgent started in stage BecomeActive (19999) in 79.974 Milliseconds
[03-Sep-2020 11:49:04 INF] Starting lifecycle stage BecomeActive (19999) took 80.0567 Milliseconds
[03-Sep-2020 11:49:04 INF] Woke up after slumber
[03-Sep-2020 11:49:04 INF] ClusterHealthMonitor started in stage Active (20000) in 0.1466 Milliseconds
[03-Sep-2020 11:49:04 INF] MembershipTableCleanupAgent started in stage Active (20000) in 0.1159 Milliseconds
[03-Sep-2020 11:49:05 INF] GatewayConnectionListener started in stage Active (20000) in 87.1509 Milliseconds
[03-Sep-2020 11:49:05 INF] Orleans.Runtime.Silo started in stage Active (20000) in 2.5985 Milliseconds
[03-Sep-2020 11:49:05 INF] Starting lifecycle stage Active (20000) took 89.6793 Milliseconds
[03-Sep-2020 11:49:05 INF] Orleans Silo started.
[03-Sep-2020 11:49:05 INF] Application started. Press Ctrl+C to shut down.
[03-Sep-2020 11:49:05 INF] Hosting environment: Production
[03-Sep-2020 11:49:05 INF] Content root path: /app
[03-Sep-2020 11:49:14 INF] Closing connection with remote endpoint 10.1.4.84:11113
[03-Sep-2020 11:49:14 WRN] Exception while processing messages from remote endpoint 10.1.4.84:11113: Microsoft.AspNetCore.Connections.ConnectionAbortedException: The Socket transport's send loop completed gracefully.
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Orleans.Runtime.Messaging.Connection.ProcessIncoming()
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The Socket transport's send loop completed gracefully.
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Orleans.Runtime.Messaging.Connection.ProcessIncoming()
[03-Sep-2020 11:49:26 INF] Connection Local: 10.1.4.82:11111, Remote: 10.1.4.83:46930, ConnectionId: 0HM2FVTDLHO0B established with S10.1.4.83:11112:336829719
[03-Sep-2020 11:49:28 INF] Closing connection with remote endpoint 10.1.4.83:46930. Exception: Microsoft.AspNetCore.Connections.ConnectionAbortedException: Connection closed
Microsoft.AspNetCore.Connections.ConnectionAbortedException: Connection closed
[03-Sep-2020 11:49:28 WRN] Exception while processing messages from remote endpoint 10.1.4.83:46930: Microsoft.AspNetCore.Connections.ConnectionResetException: Connection reset by peer
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   at Orleans.Networking.Shared.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|7_0(SocketError e)
   at Orleans.Networking.Shared.SocketAwaitableEventArgs.GetResult()
   at Orleans.Networking.Shared.SocketConnection.ProcessReceives()
   at Orleans.Networking.Shared.SocketConnection.DoReceive()
   --- End of inner exception stack trace ---
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Orleans.Runtime.Messaging.Connection.ProcessIncoming()
Microsoft.AspNetCore.Connections.ConnectionResetException: Connection reset by peer
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   at Orleans.Networking.Shared.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|7_0(SocketError e)
   at Orleans.Networking.Shared.SocketAwaitableEventArgs.GetResult()
   at Orleans.Networking.Shared.SocketConnection.ProcessReceives()
   at Orleans.Networking.Shared.SocketConnection.DoReceive()
   --- End of inner exception stack trace ---
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Orleans.Runtime.Messaging.Connection.ProcessIncoming()
[03-Sep-2020 11:49:28 INF] Connection Local: 10.1.4.82:11111, Remote: 10.1.4.83:46930, ConnectionId: 0HM2FVTDLHO0B terminated
[03-Sep-2020 11:49:35 INF] Connection Local: 10.1.4.82:11111, Remote: 10.1.4.83:47018, ConnectionId: 0HM2FVTDLHO0C established with S10.1.4.83:11112:336829772
[03-Sep-2020 11:49:35 INF] Received cluster membership snapshot via gossip: [Version: 12, 5 silos, SiloAddress=S10.1.4.84:11113:336829721 SiloName=core-pod-dbdcb6d9f-hvrph Status=Dead, SiloAddress=S10.1.4.83:11112:336829719 SiloName=patientservice-pod-6c64f9dbbf-jrghr Status=Dead, SiloAddress=S10.1.4.83:11112:336829772 SiloName=patientservice-pod-6c64f9dbbf-jrghr Status=Joining, SiloAddress=S10.1.4.82:11111:336829739 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Active, SiloAddress=S10.1.4.82:11111:336829718 SiloName=fhirserver-pod-7c7599f7b7-mxrjf Status=Dead]
[03-Sep-2020 11:49:35 INF] Pausing Kubernetes monitoring
[03-Sep-2020 11:49:46 INF] Closing connection with remote endpoint 10.1.4.83:47018. Exception: Microsoft.AspNetCore.Connections.ConnectionAbortedException: Connection closed
Microsoft.AspNetCore.Connections.ConnectionAbortedException: Connection closed
[03-Sep-2020 11:49:46 INF] Connection Local: 10.1.4.82:11111, Remote: 10.1.4.83:47018, ConnectionId: 0HM2FVTDLHO0C terminated

Then it restarts!

gabrielfreire picture gabrielfreire on 3 Sep 2020

You can't use it that way: you have to set the labels on the pod spec, not by trying to set the env vars at startup. The feature queries Kubernetes for those labels in order to discover other pods in the cluster.

ReubenBond picture ReubenBond on 3 Sep 2020
👍1

The labels also need to be set as orleans/serviceId and orleans/clusterId, unlike in the PR description

ReubenBond picture ReubenBond on 3 Sep 2020
👍2

Working beautifully !

gabrielfreire picture gabrielfreire on 3 Sep 2020

@ReubenBond does that mean

      labels:
        orleans/serviceId: myServiceName 
        orleans/clusterId: myClusterName

Or

      labels:
        serviceId: orleans/myServiceName 
        clusterId: orleans/myClusterName
freever picture freever on 29 Sep 2020

is there any documentation on this labels thing? I didn't find any and faced similar restart problem

maxormo picture maxormo on 2 Oct 2020

@freever After changing it to your first example, my pods started working again

LiamMorrow picture LiamMorrow on 27 Oct 2020

@LiamMorrow thanks I will try that!

freever picture freever on 27 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

DixonDs picture DixonDs  ·  4Comments
bwanner picture bwanner  ·  5Comments
galvesribeiro picture galvesribeiro  ·  4Comments
JorgeCandeias picture JorgeCandeias  ·  3Comments
scharada picture scharada  ·  3Comments