akka.net 1.3.12 -> 1.3.13 regression?

[avber]

• Which Akka.Net version you are using 1.3.12 -> 1.3.13
• On which platform you are using Akka.Net Windows, CentOS 7

I have the following network configuration:
192.168.150.0/24 192.168.150.61 - CentOS

192.168.150.252 - gateway with NAT

10.0.80.0/24 10.0.80.131 Windows

Both 192.168.150.61 and 10.0.80.131 join a cluster. PBM shows two nodes are up.

Application on 10.0.80.131 is a WPF application. It's using a .NET standard 2 dll with Akka.net actors.

Application on 10.0.80.131 is deploying an actor to 192.168.150.61. The deployed actor sends a message to the WPF application that cannot be deserialized.

I get different log entries when using 1.3.12 and 1.3.13

1.3.13

[DEBUG][5/2/19 5:43:31 AM][Thread 0012][LocalActorRefProvider(akka://tess-system)] Received command [Akka.Remote.DaemonMsgCreate] to RemoteSystemDaemon on [akka://tess-system]
[DEBUG][5/2/19 5:43:31 AM][Thread 0003][[akka://tess-system/system/remote-watcher#624151396]] Watching: [akka://tess-system/remote -> akka.tcp://[email protected]:4071/user]
Note 2 addresses: 10.0.80.131:4071 / [::ffff:192.168.150.252]:54193

1.3.12

[DEBUG][5/2/19 7:50:07 AM][Thread 0013][LocalActorRefProvider(akka://tess-system)] Received command [Akka.Remote.DaemonMsgCreate] to RemoteSystemDaemon on [akka://tess-system]
[DEBUG][5/2/19 7:50:07 AM][Thread 0012][[akka://tess-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Ftess-system%4010.0.80.131%3A4071-3/endpointWriter#1521407411]] Disassociated [akka.tcp://[email protected]:4055] -> akka.tcp://[email protected]:4071
[WARNING][5/2/19 7:50:07 AM][Thread 0012][[akka://tess-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Ftess-system%4010.0.80.131%3A4071-3#643178914]] Association with remote system akka.tcp://[email protected]:4071 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level, Boolean needToThrow)
at Akka.Actor.ActorCell.<>c__DisplayClass106_0.b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)]
[ERROR][5/2/19 7:50:07 AM][Thread 0012][akka://tess-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Ftess-system%4010.0.80.131%3A4071-3/endpointWriter] Disassociated
Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level, Boolean needToThrow)
at Akka.Actor.ActorCell.<>c__DisplayClass106_0.b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[DEBUG][5/2/19 7:50:07 AM][Thread 0011][[akka://tess-system/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Ftess-system%4010.0.80.131%3A4071-3/endpointWriter#1521407411]] Disassociated [akka.tcp://[email protected]:4055] -> akka.tcp://[email protected]:4071

P.S. might be related to #2254

[Aaronontheweb]

@avber are your CentOS nodes running on Mono or .NET Core Linux?

[Aaronontheweb]

And, I'm sorry, but is the issue here that there's a disassociation or that the log entries are simply different? I don't fully understand what the problem is.

[avber]

@Aaronontheweb .NET Core 2.1 Linux (self contained)

The issue is that 1.3.13 prints (and, I guess, uses) incorrect address of the remote system ( [::ffff:192.168.150.252]:54193 )

1.3.13 prints the error message that is not present in the 1.3.12 log:
was disassociated because the ProtocolStateActor failed: Unknown

[Aaronontheweb]

Looks like your system is defaulting to IPv6 addresses mapped to IPv4 (that's indicated by the ::ffff: syntax .) We didn't change anything in 1.3.13 that would do this by default - everything else still uses IPv4 addresses by default unless you explicitly enable them.

However, we did make a change to Akka.IO in 1.3.13 that uses IPv6 by default now: https://github.com/akkadotnet/akka.net/pull/3684 - none of that code gets used in Akka.Remote using the DotNetty transport, so there is no impact there.

This looks like a configuration issue - what's your Akka.Remote configuration look like on both nodes?

[avber]

Nothing fancy there

WPF app

remote {
    dot-netty.tcp {
        transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
        # Maximum frame size: ~40 MB
        maximum-frame-size = 40000000b


                    transport-protocol = tcp
        #will be populated with a dynamic host-name at runtime
        #public-hostname = ""
        #hostname = "0.0.0.0"
        #public-port = 4071
        port = 4071

    }

---

CentOS app

remote {
dot-netty.tcp {
transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
# Maximum frame size: 40 MB
maximum-frame-size = 40000000b

        applied-adapters = []
        transport-protocol = tcp
        #will be populated with a dynamic host-name at runtime if left uncommented
        #public-hostname = "10.0.80.149"
        hostname = "0.0.0.0"
        #public-port = 4055
        port = 4055
    }
}

Actor systems are instantiated by
var system = Global.ActorSystem = ActorSystem.Create(sysName, config.WithFallback(HoconLoader.GetIpAddressConfig()));

public static Config GetIpAddressConfig()
{
var name = Dns.GetHostName();
var Address = Dns.GetHostEntry(name).AddressList.FirstOrDefault(x => x.AddressFamily == AddressFamily.InterNetwork); //IPv4

        var str = @"

akka {
remote {
dot-netty.tcp {
public-hostname = ""{0}""
}
}
}
";

      str = str.Replace("{0}", Address?.ToString());
        var ipconfig = ConfigurationFactory.ParseString(str);

        return ipconfig;
    }

[snekbaev]

Not sure if it helps, at least with Azure WebApps for remoting I was using something like:

remote {
                dot-netty.tcp {
                    port = 0
                    enforce-ip-family = true
                    dns-use-ipv6 = false
                    public-hostname = "xyz.azurewebsites.net"
                    hostname = "127.0.0.1"
                }
            }

to force it to use strictly IPv4

[snekbaev]

@Aaronontheweb if I remember correctly I didn't change the default behaviour in #3684, i.e. it was using IPv6 already, whereas I needed to be able to force it to use IPv4 for Azure WebApps.

[avber]

Closing as it doesn't affect normal app flow

[ddobric]

Don't understand, why this issue is closed? I have same or similar issue when deploying Akka.NET application as Docker container to Azure.
When running same container locally on Windows (as Linux container) all works fine.

Following is reported in log:

~~~
[DEBUG][06/11/2019 13:30:40][Thread 0001][EventStream(HtmCluster)] Logger log1-DefaultLogger [DefaultLogger] started
[DEBUG][06/11/2019 13:30:40][Thread 0001][EventStream(HtmCluster)] StandardOutLogger being removed
[DEBUG][06/11/2019 13:30:40][Thread 0001][EventStream(HtmCluster)] Default Loggers started
[INFO][06/11/2019 13:30:40][Thread 0001][remoting] Starting remoting
[DEBUG][06/11/2019 13:30:40][Thread 0009][remoting] Starting prune timer for endpoint manager...
[INFO][06/11/2019 13:30:41][Thread 0001][remoting] Remoting started; listening on addresses : [akka.tcp://[email protected]:8081]
[INFO][06/11/2019 13:30:41][Thread 0001][remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:8081]
[INFO][06/11/2019 13:30:41][Thread 0001][ActorSystem(HtmCluster)] akka : {
debug : {
receive : on
autoreceive : on
lifecycle : on
event-stream : on
unhandled : on
}
loglevel : DEBUG
actor : {
provider : "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
}
log-config-on-start : on
remote : {
connection-timeout : "120 s"
transport-failure-detector : {
heartbeat-interval : 1000s
acceptable-heartbeat-pause : 6000s
}
dot-netty : {
tcp : {
maximum-frame-size : 32000000b
dns-use-ipv6 : false
port : 8081
public-hostname : nodeX.westeurope.azurecontainer.io
hostname : wk-caas-c297c6ed33e84959acbebf8240ec2752-c1e979fb02d0e830c9dd14
}
}
}
}

Press any key to stop AKKA.NET Node
~~~

[avber]

Well, I think your situation is different from mine. Did you try @snekbaev 's suggestion?

[ddobric]

Thanks for feedback. I created new issue.