Akka.net: Akka.Remote starts listening on different port than specified

Created on 23 Mar 2020  路  47Comments  路  Source: akkadotnet/akka.net

Hi guys,

I've returned to one of my pet projects (lighthouse alternative), after some time off. In the project I have a bootstrapper, that generates HOCON fallbacks, from which the configuration is being built.

var config = ConfigurationFactory.Empty;

foreach (var cfg in fallbacks)
{
  config = config.WithFallback(cfg);
}

return ActorSystem.Create("System", config);

Upon startup, I'm noticing that:

  • The actorsystem only logs the 1st fallback of the configuration rather than the entire HOCON
  • Akka REMOTE attempts to start listening on a different port than specified in HOCON, appears to be a default value from Remote.conf (2552)
  • Akka.Cluster seed nodes' fallback config appears to have been ignored and "No seed-nodes configured" message was logged

HOCON, pseudocode and log outputs located here

  • Which Akka.Net version you are using:

    • 1.4.3 (bump from 1.3.15)

  • On which platform you are using Akka.Net

    • Windows

    • Linux containers

akka-remote confirmed bug
Source
picture mmisztal1980

All 47 comments

I've went back in time, and checked out a repro #3953 that @Aaronontheweb was trying out the other day for #3943 . I've applied the fallback configs and the cluster was self-formed... :/ 

public class Bugfix3943Specs
    {
        public static string[] Hocons =
             {
                 "akka : {\r\n actor : {\r\n provider : cluster }}",
                 "akka : {\r\n stdout-loglevel : INFO\r\n loglevel : INFO\r\n log-config-on-start : on\r\n loggers : [\"Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog\"] \r\nactor : {\r\n debug : {\r\n receive : on\r\n autoreceive : on\r\n lifecycle : on\r\n event-stream : on\r\n unhandled : on\r\n } } }",
                 "akka : {\r\n remote : {\r\n dot-netty : {\r\n tcp : {\r\n log-transport : true\r\n transport-class : \"Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote\"\r\n transport-protocol : tcp\r\n hostname : 0.0.0.0\r\n public-hostname : localhost\r\n port : 9000\r\n } } } }",
                 "akka : {\r\n cluster : {\r\n log-info : on\r\n seed-nodes : [\"akka.tcp://System@localhost:9000\"]\r\n roles : []\r\n role : {}\r\n } }"
             };

        [Fact]
        public async Task Bugfix3943_should_start_ActorSystem()
        {
            var config = ConfigurationFactory.ParseString("");

            foreach (var hocon in Hocons)
            {
                config = config.WithFallback(global::Akka.Configuration.ConfigurationFactory.ParseString(hocon));
            }

            var system = ActorSystem.Create("System", config);

            var tcs = new TaskCompletionSource<Done>();
            Cluster.Get(system).RegisterOnMemberUp(() =>
            {
                tcs.SetResult(Done.Instance);
            });
            var cts = new CancellationTokenSource();
            cts.Token.Register(tcs.SetCanceled);
            cts.CancelAfter(TimeSpan.FromSeconds(5));
            await tcs.Task;
        }
    }
mmisztal1980 picture mmisztal1980 on 23 Mar 2020

The behavior seems to be gone after I've terminated a process that was blocking one of the default ports.... , I'll reopen if the issue persists

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

Thanks! Let us know if you run into any more trouble.

Aaronontheweb picture Aaronontheweb on 24 Mar 2020

@Aaronontheweb I'm actually working on reproducing this again :D

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

What鈥檚 changed?

Aaronontheweb picture Aaronontheweb on 24 Mar 2020

@Aaronontheweb
I've investigated some more and found something.

ReproSpec.cs
Requires Akka.Logger.Serilog 1.4.1

using System;
using System.Threading;
using System.Threading.Tasks;
using Akka.Actor;
using Akka.Configuration;
using Xunit;

namespace Akka.Cluster.Tests
{
    public class ReproSpec
    {
        public static string[] Hocons =
        {
            "akka : {\r\n actor : {\r\n provider : cluster }}",
            "akka : {\r\n stdout-loglevel : INFO\r\n loglevel : INFO\r\n log-config-on-start : on\r\n loggers : [\"Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog\"] \r\nactor : {\r\n debug : {\r\n receive : on\r\n autoreceive : on\r\n lifecycle : on\r\n event-stream : on\r\n unhandled : on\r\n } } }",
            "akka : {\r\n remote : {\r\n dot-netty : {\r\n tcp : {\r\n log-transport : true\r\n transport-class : \"Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote\"\r\n transport-protocol : tcp\r\n hostname : 0.0.0.0\r\n public-hostname : core-cluster-svc-0.core-cluster-svc\r\n port : 5000\r\n } } } }",
            "akka : {\r\n cluster : {\r\n log-info : on\r\n seed-nodes : [\"akka.tcp://[email protected]:5000\",\"akka.tcp://[email protected]:5000\",\"akka.tcp://[email protected]:5000\"]\r\n roles : [seed]\r\n role : { } } }"
        };

        [Fact()]
        public async Task Should_start_ActorSystem()
        {
            var config = ConfigurationFactory.ParseString("");

            foreach (var hocon in Hocons)
            {
                config = config.WithFallback(global::Akka.Configuration.ConfigurationFactory.ParseString(hocon));
            }

            var system = ActorSystem.Create("System", config);

            var tcs = new TaskCompletionSource<Done>();
            Cluster.Get(system).RegisterOnMemberUp(() =>
            {
                tcs.SetResult(Done.Instance);
            });
            var cts = new CancellationTokenSource();
            cts.Token.Register(tcs.SetCanceled);
            cts.CancelAfter(TimeSpan.FromSeconds(5));
            await tcs.Task;
        }
    }
}

Upon execution, a NullReferenceException was thrown from RemoteActorRefProvider:L190 when trying to obtain Transport.DefaultAddress which in turn attempted to retrieve RemoteInternals.Transport (and apparently RemoteActorRefProvider.RemoteInternals == null).

I'm wondering what the issue is, the fallback responsible for configuring akka.remote is:

akka : {
  remote : {
    dot-netty : {
      tcp : {
        log-transport : true
        transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
        transport-protocol : tcp
        hostname : 0.0.0.0
        public-hostname : core-cluster-svc-0.core-cluster-svc
        port : 5000
      }
    }
  }
}

Correct me if I'm wrong (just in case) but, in HOCON terms:

akka {
    dot-netty.tcp {
       (...)
    }
}

should be equivalent to:

akka {
    dot-netty {
      tcp {
         (...)
      }
    }
}

Or perhaps my approach to composing the config (from multiple fallbacks) is completely bonkers?

I'm wondering what causes this :/

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

@Aaronontheweb I've done some more debugging:

RemoteActorRefProvider.cs

        public virtual void Init(ActorSystemImpl system)
        {
            _system = system;

            _local.Init(system); // <-- NullReferenceException

            _actorRefResolveThreadLocalCache = ActorRefResolveThreadLocalCache.For(system);
            _actorPathThreadLocalCache = ActorPathThreadLocalCache.For(system);

            _remotingTerminator =
                _system.SystemActorOf(
                    RemoteSettings.ConfigureDispatcher(Props.Create(() => new RemotingTerminator(_local.SystemGuardian))),
                    "remoting-terminator");

            _internals = CreateInternals(); // <-- This is where the requested Internals are being initialized

            _remotingTerminator.Tell(RemoteInternals);

            Transport.Start();
            _remoteWatcher = CreateRemoteWatcher(system);
            _remoteDeploymentWatcher = CreateRemoteDeploymentWatcher(system);
        }

Race condition?

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

Got an NRE inside the LocalActorRefProvider? Shouldn't be any kind of race condition there - it's just doing the same stuff we need to create a normal local ActorSystem.

Aaronontheweb picture Aaronontheweb on 24 Mar 2020

Ok, so why is it trying to get the internals before they've been initialized?

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

If it's a handled NRE exception, then that would be because of

https://github.com/akkadotnet/akka.net/blob/b62842e299533daad755f1baa70ea12394c61c1a/src/core/Akka/Serialization/Serialization.cs#L114-L123

Aaronontheweb picture Aaronontheweb on 24 Mar 2020

I should note - I haven't looked at the logs yet, but @Arkatufus is taking a look at this issue.

Aaronontheweb picture Aaronontheweb on 24 Mar 2020
🎉1

From your logs, looks like the system is binding to hostname without displaying itself as localhost for some reason.

Although the socket bind error is from Petabridge.Cmd

Aaronontheweb picture Aaronontheweb on 24 Mar 2020

Checking

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

hmmm yes, just noticed that too, [email protected]:2552 - seemed weird, but it's not an issue at the moment.

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

Could be related - the logging system has to grab the transport info at startup.

Aaronontheweb picture Aaronontheweb on 24 Mar 2020

I tried your debugging code and I can't quite reproduce the bug, it did fail because it couldn't connect to the seed because the seed is missing.

Arkatufus picture Arkatufus on 24 Mar 2020

@Arkatufus the HOCON & logs were collected from 1 of 3 pods running in a k8s StatefulSet, specifically from core-cluster-svc-0

akka : {
  cluster : {
    log-info : on
    seed-nodes : [
       "akka.tcp://[email protected]:5000",
       "akka.tcp://[email protected]:5000",
       "akka.tcp://[email protected]:5000"
   ]
   roles : [seed]
   role : { } 
  } 
}

What do you think is missing?

mmisztal1980 picture mmisztal1980 on 24 Mar 2020

This is what I got when I tried to run your code above:
#4353_Log.txt

This is the code I use to try and reproduce the bug:
https://gist.github.com/Arkatufus/f5362f1e497fc82aeb3d7816a786b8da

So the error came from one of the k8 seed node?

Arkatufus picture Arkatufus on 24 Mar 2020

Yes, actually the entire deployment is failing at the moment

kubectl get pods
NAME                 READY     STATUS             RESTARTS   AGE
core-cluster-svc-0   0/1       CrashLoopBackOff   20         22h
core-cluster-svc-1   0/1       CrashLoopBackOff   20         22h
core-cluster-svc-2   0/1       CrashLoopBackOff   20         22h

位 kubectl logs core-cluster-svc-0
[INFO][03/24/2020 22:43:53][Thread 0004][[akka://System/system/log1-SerilogLogger#128029824]] SerilogLogger started
crit: Core.IMicroService[0]
      core-cluster-svc has failed to start: System.NullReferenceException: Object reference not set to an instance of an object.
         at Akka.Cluster.ClusterSettings.<>c.<.ctor>b__2_1(KeyValuePair`2 kv)
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs)
         at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 keyComparer, IEqualityComparer`1 valueComparer)
         at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
         at Akka.Cluster.ClusterSettings..ctor(Config config, String systemName)
         at Akka.Cluster.Cluster..ctor(ActorSystemImpl system)
         at Akka.Cluster.ClusterExtension.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.<>c__DisplayClass56_0.<RegisterExtension>b__1()
         at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
         at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
         at System.Lazy`1.CreateValue()
         at System.Lazy`1.get_Value()
         at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension(Type extensionType, Object& extension)
         at Akka.Actor.Internal.ActorSystemImpl.GetExtension(IExtensionId extensionId)
         at Akka.Actor.ExtensionIdProvider`1.Get(ActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.Get(ActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.RegisterExtension(IExtensionId extension)
         at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
         at Akka.Cluster.Cluster.Get(ActorSystem system)
         at Akka.Cluster.ClusterActorRefProvider.CreateRemoteWatcher(ActorSystemImpl system)
         at Akka.Remote.RemoteActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Cluster.ClusterActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Actor.Internal.ActorSystemImpl.Start()
         at Core.Akka.ActorSystemBuilder.Build()
         at Core.Cluster.Service.StartupAkkaExtensions.AddActorSystem(IServiceCollection services, IConfiguration configuration) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.Akka.cs:line 29
         at Core.Cluster.Service.Startup.OnConfigureServices(IServiceCollection services, IConfiguration configuration, IMicroService microservice) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.cs:line 27
         at Core.Hosting.MicroServiceStartup.ConfigureServices(IServiceCollection services)
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
         at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
         at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass15_0.<BuildStartupServicesFilterPipeline>g__RunPipeline|0(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
         at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
         at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
         at Core.Hosting.MicroService`1.InitializeAsync(IConfigurationRoot configuration, String[] args)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Akka.Cluster.ClusterSettings.<>c.<.ctor>b__2_1(KeyValuePair`2 kv)
   at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource element)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs)
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 keyComparer, IEqualityComparer`1 valueComparer)
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   at Akka.Cluster.ClusterSettings..ctor(Config config, String systemName)
   at Akka.Cluster.Cluster..ctor(ActorSystemImpl system)
   at Akka.Cluster.ClusterExtension.CreateExtension(ExtendedActorSystem system)
   at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.CreateExtension(ExtendedActorSystem system)
   at Akka.Actor.Internal.ActorSystemImpl.<>c__DisplayClass56_0.<RegisterExtension>b__1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension(Type extensionType, Object& extension)
   at Akka.Actor.Internal.ActorSystemImpl.GetExtension(IExtensionId extensionId)
   at Akka.Actor.ExtensionIdProvider`1.Get(ActorSystem system)
   at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.Get(ActorSystem system)
   at Akka.Actor.Internal.ActorSystemImpl.RegisterExtension(IExtensionId extension)
   at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
   at Akka.Cluster.Cluster.Get(ActorSystem system)
   at Akka.Cluster.ClusterActorRefProvider.CreateRemoteWatcher(ActorSystemImpl system)
   at Akka.Remote.RemoteActorRefProvider.Init(ActorSystemImpl system)
   at Akka.Cluster.ClusterActorRefProvider.Init(ActorSystemImpl system)
   at Akka.Actor.Internal.ActorSystemImpl.Start()
   at Core.Akka.ActorSystemBuilder.Build()
   at Core.Cluster.Service.StartupAkkaExtensions.AddActorSystem(IServiceCollection services, IConfiguration configuration) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.Akka.cs:line 29
   at Core.Cluster.Service.Startup.OnConfigureServices(IServiceCollection services, IConfiguration configuration, IMicroService microservice) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.cs:line 27
   at Core.Hosting.MicroServiceStartup.ConfigureServices(IServiceCollection services)
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass15_0.<BuildStartupServicesFilterPipeline>g__RunPipeline|0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Core.Hosting.MicroService`1.InitializeAsync(IConfigurationRoot configuration, String[] args)
   at Core.Hosting.MicroService`1.RunAsync(IConfigurationRoot configuration, CancellationToken cancellationToken, String[] args)
   at Core.Cluster.Service.Program.Main(String[] args) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Program.cs:line 14
   at Core.Cluster.Service.Program.<Main>(String[] args)
Aborted
mmisztal1980 picture mmisztal1980 on 24 Mar 2020

Ok, let me read the logs, thanks.

Arkatufus picture Arkatufus on 24 Mar 2020

It is failing at this line:
https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka.Cluster/ClusterSettings.cs#L78

Can you try adding seed.min-nr-of-members = 1 in your akka.cluster.role hocon settings?

Arkatufus picture Arkatufus on 25 Mar 2020

sure, let me give it a try

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

encountered some tech issues with my stack, I'll get back to you after I get this to run under the setting you've specified.

The line you've indicated makes sense, but also seems like an interesting observation concerning: https://getakka.net/articles/configuration/akka.cluster.html

The observation would be, that having :

akka {
  cluster {
    role {
        // empty
    }
  }
}

is not allowed - correct?

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

It should be allowed, I'm trying to find out if it is a bug in hocon, I was hoping that by providing a value, it can be a temporary bandaid until I can find out the real culprit.

Arkatufus picture Arkatufus on 25 Mar 2020

Found an interesting thing

This is the HOCON that's being parsed using ConfigurationFactory.ParseString()

akka {
    cluster {
        log-info = on
        seed-nodes = ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
        roles = ["seed"]
        role["seed"].min-nr-of-members = 2
    }
}

ConfigurationFactory.ParseString(hocon).ToString() yields:

  akka : {
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
      roles : [seed]
      role : {
        min-nr-of-members : 2
      }
    }
  }

Can you repro that on your end?

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

Common culprit might be the GetObject call inside ClusterSettings

Aaronontheweb picture Aaronontheweb on 25 Mar 2020

just noticed that compared to the docs, I seem to be missing a . character in front of the [] brackets:
role.["crawler"].min-nr-of-members = 3

retesting

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

Yep, that was it

  akka : {
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
      roles : [seed]
      role : {
        seed : {
          min-nr-of-members : 2
        }
      }
    }
  }

retesting

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

Its supposed to be akka.cluster.role.seed.min-nr-of-members : 2, not akka.cluster.role["seed"].min-nr-of-members : 2

Arkatufus picture Arkatufus on 25 Mar 2020
👍1

ETA 10m for my CI to process the cfg change so I can redeploy it.

In the mean time, I really think I should write a MNTK test with this config to verify its correctness. @Aaronontheweb is there a sample I can reference for that?

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

Well, I tried to run the line of code that was failing inside ClusterSettings against your configuration and it ran fine, so the only explanation is that the akka.cluster.role object inside the configuration file somehow got deleted.

Arkatufus picture Arkatufus on 25 Mar 2020

AFAIK I'm not explicitly deleting anything from the configuration

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

The deployment crashed again, log output similiar to the old one:


Pod stdout log

位 kubectl logs core-cluster-svc-0
[INFO][03/25/2020 00:29:12][Thread 0005][[akka://System/system/log1-SerilogLogger#526976188]] SerilogLogger started
crit: Core.IMicroService[0]
      core-cluster-svc has failed to start: System.NullReferenceException: Object reference not set to an instance of an object.
         at Akka.Cluster.ClusterSettings.<>c.<.ctor>b__2_1(KeyValuePair`2 kv)
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs)
         at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 keyComparer, IEqualityComparer`1 valueComparer)
         at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
         at Akka.Cluster.ClusterSettings..ctor(Config config, String systemName)
         at Akka.Cluster.Cluster..ctor(ActorSystemImpl system)
         at Akka.Cluster.ClusterExtension.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.<>c__DisplayClass56_0.<RegisterExtension>b__1()
         at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
         at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
         at System.Lazy`1.CreateValue()
         at System.Lazy`1.get_Value()
         at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension(Type extensionType, Object& extension)
         at Akka.Actor.Internal.ActorSystemImpl.GetExtension(IExtensionId extensionId)
         at Akka.Actor.ExtensionIdProvider`1.Get(ActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.Get(ActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.RegisterExtension(IExtensionId extension)
         at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
         at Akka.Cluster.Cluster.Get(ActorSystem system)
         at Akka.Cluster.ClusterActorRefProvider.CreateRemoteWatcher(ActorSystemImpl system)
         at Akka.Remote.RemoteActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Cluster.ClusterActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Actor.Internal.ActorSystemImpl.Start()
         at Core.Akka.ActorSystemBuilder.Build()
         at Core.Cluster.Service.StartupAkkaExtensions.AddActorSystem(IServiceCollection services, IConfiguration configuration) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.Akka.cs:line 29
         at Core.Cluster.Service.Startup.OnConfigureServices(IServiceCollection services, IConfiguration configuration, IMicroService microservice) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.cs:line 27
         at Core.Hosting.MicroServiceStartup.ConfigureServices(IServiceCollection services)
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
         at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
         at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass15_0.<BuildStartupServicesFilterPipeline>g__RunPipeline|0(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
         at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
         at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
         at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
         at Core.Hosting.MicroService`1.InitializeAsync(IConfigurationRoot configuration, String[] args)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Akka.Cluster.ClusterSettings.<>c.<.ctor>b__2_1(KeyValuePair`2 kv)
   at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource element)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs)
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 keyComparer, IEqualityComparer`1 valueComparer)
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   at Akka.Cluster.ClusterSettings..ctor(Config config, String systemName)
   at Akka.Cluster.Cluster..ctor(ActorSystemImpl system)
   at Akka.Cluster.ClusterExtension.CreateExtension(ExtendedActorSystem system)
   at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.CreateExtension(ExtendedActorSystem system)
   at Akka.Actor.Internal.ActorSystemImpl.<>c__DisplayClass56_0.<RegisterExtension>b__1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension(Type extensionType, Object& extension)
   at Akka.Actor.Internal.ActorSystemImpl.GetExtension(IExtensionId extensionId)
   at Akka.Actor.ExtensionIdProvider`1.Get(ActorSystem system)
   at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.Get(ActorSystem system)
   at Akka.Actor.Internal.ActorSystemImpl.RegisterExtension(IExtensionId extension)
   at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
   at Akka.Cluster.Cluster.Get(ActorSystem system)
   at Akka.Cluster.ClusterActorRefProvider.CreateRemoteWatcher(ActorSystemImpl system)
   at Akka.Remote.RemoteActorRefProvider.Init(ActorSystemImpl system)
   at Akka.Cluster.ClusterActorRefProvider.Init(ActorSystemImpl system)
   at Akka.Actor.Internal.ActorSystemImpl.Start()
   at Core.Akka.ActorSystemBuilder.Build()
   at Core.Cluster.Service.StartupAkkaExtensions.AddActorSystem(IServiceCollection services, IConfiguration configuration) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.Akka.cs:line 29
   at Core.Cluster.Service.Startup.OnConfigureServices(IServiceCollection services, IConfiguration configuration, IMicroService microservice) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Startup.cs:line 27
   at Core.Hosting.MicroServiceStartup.ConfigureServices(IServiceCollection services)
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass15_0.<BuildStartupServicesFilterPipeline>g__RunPipeline|0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Core.Hosting.MicroService`1.InitializeAsync(IConfigurationRoot configuration, String[] args)
   at Core.Hosting.MicroService`1.RunAsync(IConfigurationRoot configuration, CancellationToken cancellationToken, String[] args)
   at Core.Cluster.Service.Program.Main(String[] args) in /bld/core-cluster/core-cluster-svc/src/Core.Cluster.Service/Program.cs:line 14
   at Core.Cluster.Service.Program.<Main>(String[] args)
Aborted

The akka.cluster HOCON piece was:

akka : {
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
      roles : [seed]
      role : {
        seed : {
          min-nr-of-members : 2
        }
      }
    }
  }

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

ok, I'll try to reproduce it, I'll post my findings here.

Arkatufus picture Arkatufus on 25 Mar 2020

I still can't reproduce it. Are you sure all of the pods are using the same configuration?
Here's the code I've used to try to reproduce the bug:
https://gist.github.com/Arkatufus/028f958a8366f2e58b3e1b6fbcb73f5b

Arkatufus picture Arkatufus on 25 Mar 2020

Yes, all pods have the same config, only the public hostname is varying from pod to pod.

In your HOCON:

"akka : {
  cluster : {
    log-info : on
    seed-nodes : [
      ""akka.tcp://AkkaSpec@localhost:5000"",
      ""akka.tcp://System@localhost:5001"",
      ""akka.tcp://System@localhost:5002""
    ]
    roles : [seed]
    role : { } 
  } 
}"

The 1st seed node refers to a different actor system than the other 2, is that intentional?

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

I've run a successful test with docker-compose

The HOCON appears to by syntaxically correct

位 docker-compose up
Recreating core-cluster_core-cluster-svc-2_1 ... done
Starting core-cluster_core-cluster-svc-1_1   ... done
Attaching to core-cluster_core-cluster-svc-1_1, core-cluster_core-cluster-svc-2_1
core-cluster-svc-1_1  | [INFO][03/25/2020 15:06:06][Thread 0007][akka://System/system/log1-SerilogLogger] SerilogLogger started
core-cluster-svc-2_1  | [INFO][03/25/2020 15:06:06][Thread 0009][akka://System/system/log1-SerilogLogger] SerilogLogger started
core-cluster-svc-1_1  | info: Core.IMicroService[0]
core-cluster-svc-1_1  |       core-cluster-svc initialized successfully: Core.Hosting.MicroService`1[Core.Cluster.Service.Startup]
core-cluster-svc-2_1  | info: Core.IMicroService[0]
core-cluster-svc-2_1  |       core-cluster-svc initialized successfully: Core.Hosting.MicroService`1[Core.Cluster.Service.Startup]
core-cluster-svc-1_1  | [15:06:06 INF] ClusterService started
core-cluster-svc-1_1  | [15:06:06 INF]   akka : {
core-cluster-svc-1_1  |     actor : {
core-cluster-svc-1_1  |       provider : cluster
core-cluster-svc-1_1  |       debug : {
core-cluster-svc-1_1  |         receive : on
core-cluster-svc-1_1  |         autoreceive : on
core-cluster-svc-1_1  |         lifecycle : on
core-cluster-svc-1_1  |         event-stream : on
core-cluster-svc-1_1  |         unhandled : on
core-cluster-svc-1_1  |       }
core-cluster-svc-1_1  |     }
core-cluster-svc-1_1  |     remote : {
core-cluster-svc-1_1  |       dot-netty : {
core-cluster-svc-1_1  |         tcp : {
core-cluster-svc-1_1  |           log-transport : true
core-cluster-svc-1_1  |           transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
core-cluster-svc-1_1  |           transport-protocol : tcp
core-cluster-svc-1_1  |           hostname : 0.0.0.0
core-cluster-svc-1_1  |           public-hostname : core-cluster-svc-1
core-cluster-svc-1_1  |           port : 5000
core-cluster-svc-1_1  |         }
core-cluster-svc-1_1  |       }
core-cluster-svc-1_1  |     }
core-cluster-svc-1_1  |     cluster : {
core-cluster-svc-1_1  |       log-info : on
core-cluster-svc-1_1  |       seed-nodes : ["akka.tcp://System@core-cluster-svc-1:5000","akka.tcp://System@core-cluster-svc-2:5000"]
core-cluster-svc-1_1  |       roles : [seed]
core-cluster-svc-1_1  |       role : {
core-cluster-svc-1_1  |         seed : {
core-cluster-svc-1_1  |           min-nr-of-members : 2
core-cluster-svc-1_1  |         }
core-cluster-svc-1_1  |       }
core-cluster-svc-1_1  |     }
core-cluster-svc-1_1  |   }
core-cluster-svc-1_1  |
core-cluster-svc-2_1  | [15:06:06 INF] ClusterService started
core-cluster-svc-2_1  | [15:06:07 INF]   akka : {
core-cluster-svc-2_1  |     actor : {
core-cluster-svc-2_1  |       provider : cluster
core-cluster-svc-2_1  |       debug : {
core-cluster-svc-2_1  |         receive : on
core-cluster-svc-2_1  |         autoreceive : on
core-cluster-svc-2_1  |         lifecycle : on
core-cluster-svc-2_1  |         event-stream : on
core-cluster-svc-2_1  |         unhandled : on
core-cluster-svc-2_1  |       }
core-cluster-svc-2_1  |     }
core-cluster-svc-2_1  |     remote : {
core-cluster-svc-2_1  |       dot-netty : {
core-cluster-svc-2_1  |         tcp : {
core-cluster-svc-2_1  |           log-transport : true
core-cluster-svc-2_1  |           transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
core-cluster-svc-2_1  |           transport-protocol : tcp
core-cluster-svc-2_1  |           hostname : 0.0.0.0
core-cluster-svc-2_1  |           public-hostname : core-cluster-svc-2
core-cluster-svc-2_1  |           port : 5000
core-cluster-svc-2_1  |         }
core-cluster-svc-2_1  |       }
core-cluster-svc-2_1  |     }
core-cluster-svc-2_1  |     cluster : {
core-cluster-svc-2_1  |       log-info : on
core-cluster-svc-2_1  |       seed-nodes : ["akka.tcp://System@core-cluster-svc-1:5000","akka.tcp://System@core-cluster-svc-2:5000"]
core-cluster-svc-2_1  |       roles : [seed]
core-cluster-svc-2_1  |       role : {
core-cluster-svc-2_1  |         seed : {
core-cluster-svc-2_1  |           min-nr-of-members : 2
core-cluster-svc-2_1  |         }
core-cluster-svc-2_1  |       }
core-cluster-svc-2_1  |     }
core-cluster-svc-2_1  |   }
core-cluster-svc-2_1  |
core-cluster-svc-1_1  | [15:06:12 INF] This node is UP
core-cluster-svc-2_1  | [15:06:12 INF] This node is UP
mmisztal1980 picture mmisztal1980 on 25 Mar 2020

@mmisztal1980 so adding the min-nr-of-members fixed the issue?

Aaronontheweb picture Aaronontheweb on 25 Mar 2020

it worked in docker-compose but I'm still seeing crashes in k8s (NRE), I'm trying to determine the difference

mmisztal1980 picture mmisztal1980 on 25 Mar 2020
👍1

@mmisztal1980 please check https://github.com/akkadotnet/akka.net/pull/4358, I can't guarantee a 100% fix, but I think I fixed the major ones

Arkatufus picture Arkatufus on 25 Mar 2020

@mmisztal1980 try tonight's nightly, 1.4.3-beta637207629595381843 - that has this fix included.

Aaronontheweb picture Aaronontheweb on 25 Mar 2020

on it

mmisztal1980 picture mmisztal1980 on 25 Mar 2020

@Aaronontheweb initial result is negative, I'll do some more testing tomorrow


Stacktrace

      core-cluster-svc has failed to start: System.NullReferenceException: Object reference not set to an instance of an object.
         at Akka.Configuration.Config.WithFallback(Config fallback)
         at Akka.Configuration.Config.GetConfig(String path)
         at Akka.Configuration.Config.GetConfig(String path)
         at Akka.Cluster.ClusterSettings..ctor(Config config, String systemName)
         at Akka.Cluster.Cluster..ctor(ActorSystemImpl system)
         at Akka.Cluster.ClusterExtension.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.<>c__DisplayClass56_0.<RegisterExtension>b__1()
         at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
         at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
         at System.Lazy`1.CreateValue()
         at System.Lazy`1.get_Value()
         at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension(Type extensionType, Object& extension)
         at Akka.Actor.Internal.ActorSystemImpl.GetExtension(IExtensionId extensionId)
         at Akka.Actor.ExtensionIdProvider`1.Get(ActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.Get(ActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.RegisterExtension(IExtensionId extension)
         at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
         at Akka.Cluster.Cluster.Get(ActorSystem system)
         at Akka.Cluster.ClusterActorRefProvider.CreateRemoteWatcher(ActorSystemImpl system)
         at Akka.Remote.RemoteActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Cluster.ClusterActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Actor.Internal.ActorSystemImpl.Start()

mmisztal1980 picture mmisztal1980 on 26 Mar 2020

@Aaronontheweb @Arkatufus I'm being cautiously optimistic right now, I'm going to retest with the role settings enabled - I've found an issue in my HELM setup and fixed it, so this is the result of a combination of factors


kubectl logs

位 kubectl log core-cluster-svc-0
log is DEPRECATED and will be removed in a future version. Use logs instead.
[INFO][03/26/2020 09:47:43][Thread 0010][akka://System/system/log1-SerilogLogger] SerilogLogger started
info: Core.IMicroService[0]
      core-cluster-svc initialized successfully in Kubernetes namespace default: Core.Hosting.MicroService`1[Core.Cluster.Service.Startup].
[09:47:44 INF] ClusterService started
[09:47:44 INF]   akka : {
    actor : {
      provider : cluster
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
    }
    remote : {
      dot-netty : {
        tcp : {
          log-transport : true
          transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
          transport-protocol : tcp
          hostname : 0.0.0.0
          public-hostname : core-cluster-svc-0.core-cluster-svc
          port : 5000
        }
      }
    }
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
      roles : [seed]
    }
  }
[09:47:47 INF] This node is UP

mmisztal1980 picture mmisztal1980 on 26 Mar 2020

Looks like we're in business :)

位 kubectl get pods
NAME                 READY     STATUS    RESTARTS   AGE
core-cluster-svc-0   1/1       Running   0          17m
core-cluster-svc-1   1/1       Running   0          17m
core-cluster-svc-2   1/1       Running   0          17m


kubectl logs core-cluster-svc-0

位 kubectl logs core-cluster-svc-0
[INFO][03/26/2020 10:07:02][Thread 0005][akka://System/system/log1-SerilogLogger] SerilogLogger started
info: Core.IMicroService[0]
      core-cluster-svc initialized successfully in Kubernetes namespace default: Core.Hosting.MicroService`1[Core.Cluster.Service.Startup].
[10:07:02 INF] ClusterService started
[10:07:03 INF]   akka : {
    actor : {
      provider : cluster
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
    }
    remote : {
      dot-netty : {
        tcp : {
          log-transport : true
          transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
          transport-protocol : tcp
          hostname : 0.0.0.0
          public-hostname : core-cluster-svc-0.core-cluster-svc
          port : 5000
        }
      }
    }
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
      roles : [seed]
      role : {
        seed : {
          min-nr-of-members : 2
        }
      }
    }
  }

[10:07:09 INF] This node is UP


kubectl logs core-cluster-svc-1

位 kubectl logs core-cluster-svc-1
[INFO][03/26/2020 10:07:03][Thread 0010][akka://System/system/log1-SerilogLogger] SerilogLogger started
info: Core.IMicroService[0]
      core-cluster-svc initialized successfully in Kubernetes namespace default: Core.Hosting.MicroService`1[Core.Cluster.Service.Startup].
[10:07:03 INF] ClusterService started
[10:07:04 INF]   akka : {
    actor : {
      provider : cluster
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
    }
    remote : {
      dot-netty : {
        tcp : {
          log-transport : true
          transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
          transport-protocol : tcp
          hostname : 0.0.0.0
          public-hostname : core-cluster-svc-1.core-cluster-svc
          port : 5000
        }
      }
    }
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
      roles : [seed]
      role : {
        seed : {
          min-nr-of-members : 2
        }
      }
    }
  }

[10:07:09 INF] This node is UP


kubectl logs core-cluster-svc-2

位 kubectl logs core-cluster-svc-2
[INFO][03/26/2020 10:07:04][Thread 0005][akka://System/system/log1-SerilogLogger] SerilogLogger started
info: Core.IMicroService[0]
      core-cluster-svc initialized successfully in Kubernetes namespace default: Core.Hosting.MicroService`1[Core.Cluster.Service.Startup].
[10:07:04 INF] ClusterService started
[10:07:04 INF]   akka : {
    actor : {
      provider : cluster
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
    }
    remote : {
      dot-netty : {
        tcp : {
          log-transport : true
          transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
          transport-protocol : tcp
          hostname : 0.0.0.0
          public-hostname : core-cluster-svc-2.core-cluster-svc
          port : 5000
        }
      }
    }
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000","akka.tcp://[email protected]:5000"]
      roles : [seed]
      role : {
        seed : {
          min-nr-of-members : 2
        }
      }
    }
  }

[10:07:12 INF] This node is UP


Healthcheck

位 kubectl port-forward svc/core-cluster-svc 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
GET localhost:8080/healthz
{
  "status": "Healthy",
  "results": {
    "AkkaClusterHealthCheck": {
      "status": "Healthy",
      "description": "OK",
      "data": {}
    }
  }
}

mmisztal1980 picture mmisztal1980 on 26 Mar 2020

So is there still a bug on our end?

Aaronontheweb picture Aaronontheweb on 26 Mar 2020

The fix by @Arkatufus seems to have helped, so I'm more than happy to close the issue.

What I'm wondering about though, are the NRE stack-traces I've posted above. Specifically I'm wondering about Akka.Cluster.ClusterSettings.<>c.<.ctor>b__2_1(KeyValuePair2 kv)`, was it failing because of the failure to merge the fallbacks?


Stacktrace

System.NullReferenceException: Object reference not set to an instance of an object.
         at Akka.Cluster.ClusterSettings.<>c.<.ctor>b__2_1(KeyValuePair`2 kv)
         at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource element)
         at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 items, MutationInput origin, KeyCollisionBehavior collisionBehavior)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs, Boolean avoidToHashMap)
         at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 pairs)
         at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 keyComparer, IEqualityComparer`1 valueComparer)
         at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
         at Akka.Cluster.ClusterSettings..ctor(Config config, String systemName)
         at Akka.Cluster.Cluster..ctor(ActorSystemImpl system)
         at Akka.Cluster.ClusterExtension.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.CreateExtension(ExtendedActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.<>c__DisplayClass56_0.<RegisterExtension>b__1()
         at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
         at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
         at System.Lazy`1.CreateValue()
         at System.Lazy`1.get_Value()
         at Akka.Actor.Internal.ActorSystemImpl.TryGetExtension(Type extensionType, Object& extension)
         at Akka.Actor.Internal.ActorSystemImpl.GetExtension(IExtensionId extensionId)
         at Akka.Actor.ExtensionIdProvider`1.Get(ActorSystem system)
         at Akka.Actor.ExtensionIdProvider`1.Akka.Actor.IExtensionId.Get(ActorSystem system)
         at Akka.Actor.Internal.ActorSystemImpl.RegisterExtension(IExtensionId extension)
         at Akka.Actor.ActorSystemWithExtensions.WithExtension[T,TI](ActorSystem system)
         at Akka.Cluster.Cluster.Get(ActorSystem system)
         at Akka.Cluster.ClusterActorRefProvider.CreateRemoteWatcher(ActorSystemImpl system)
         at Akka.Remote.RemoteActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Cluster.ClusterActorRefProvider.Init(ActorSystemImpl system)
         at Akka.Actor.Internal.ActorSystemImpl.Start()

mmisztal1980 picture mmisztal1980 on 26 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

RoBiK75 picture RoBiK75  路  5Comments
vasily-kirichenko picture vasily-kirichenko  路  4Comments
seungyongshim picture seungyongshim  路  6Comments
carol-braileanu picture carol-braileanu  路  4Comments
Horusiath picture Horusiath  路  3Comments