akka.net 🚀 - DotNetty Remote Transport Issues with .NET Core 2.1

We don't officially support .NET Core 2.1 yet. Heck, we aren't even on netstandard 2.0 yet (although work is underway). But thanks for confirming that there are indeed issues :)

Danthar on 13 Jun 2018

Technically, everything we do should be working on .NET Core 2.1 since .NET Standard 1.6 can be consumed upstream. We'll investigate. Might be a DotNetty issue since v0.4.8 started using some of the unsafe APIs.

Aaronontheweb on 13 Jun 2018

It may be also valid for Windows as our implementation does not work anymore since we moved to .NET Core 2.1. We still need to investigate but the problem is exactly the same: remote actors cannot join anymore. We tested both in Docker and out of container.

xlegalles on 22 Jun 2018

Forget my comment: it was not related to AKKA. So it means that we are able to use AKKA.Net with 2.1 under Windows.

xlegalles on 25 Jun 2018

We are currently running a 1.3.8 cluster with .net core 2.1 in k8s with only a few minor issues.

annymsMthd on 1 Jul 2018

@annymsMthd what were those minor issues?

Lutando on 2 Jul 2018

Sometimes we have a node that starts kicking out other nodes after we deploy even though it's not leader. It just starts marking nodes as down and even though they restart it just targets them again and marks them. When we reset the node that is doing the marking everything works. We only see this when we deploy though so it isn't a show stopper.

annymsMthd on 3 Jul 2018

We have a similar behavior. Might be worth investigating.

Lutando on 3 Jul 2018

Experiencing the same issue. Net core 2.1 on Linux. The problem is random ; restarting often works

Cephei on 19 Jul 2018

Could it be that it get solved if instead of using DotNetty 0.4.8 we upgrade to 0.5.0?

pmorelli92 on 25 Aug 2018

@pmorelli92 just tried to launch my cluster under docker for Windows (linux), with DotNetty.Handlers 0.5.0 . Nodes still fail to connect.

mmisztal1980 on 10 Sep 2018

Based on some of the quality issues we've had with DotNetty recently, I've been exploring a replacement for it built on Akka.Streams.

But that is a ways off still. In the short run, I'll look more closely at the .NET Core 2.1 issues you all have been reporting. Strong chance it's something fucked up with the runtime itself and DotNetty has to patch around it in order to get it to work. Wouldn't be the first time this has happened. Part of the price of being on the bleeding edge.

RE: the cluster issues. We'll look into that too - is that caused by nodes having connectivity problems or does this appear to be a 100% programmatic issue?

Aaronontheweb on 10 Sep 2018

@Aaronontheweb I've examined my hocon multiple times (hocon below is the final configuration) and to be honest I don't see anything wrong with it. (keep in mind that it's been a while).

Hocon obtained from: core-cluster-svc

  akka : {
    stdout-loglevel : DEBUG
    loglevel : INFO
    log-config-on-start : on
    loggers : ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
    actor : {
      provider : cluster
      debug : {
        receive : on
        autoreceive : on
        lifecycle : on
        event-stream : on
        unhandled : on
      }
    }
    remote : {
      dot-netty : {
        tcp : {
          transport-class : "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
          transport-protocol : tcp
          hostname : 0.0.0.0
          public-hostname : core-cluster-svc
          port : 9000
        }
      }
    }
    cluster : {
      log-info : on
      seed-nodes : ["akka.tcp://System@core-cluster-svc-1:9000","akka.tcp://System@core-cluster-svc-2:9000","akka.tcp://System@core-cluster-svc:9000"]
      roles : [cluster-seed]
    }
  }

version: '3.4'

services:
  core-cluster-svc:
    image: core-cluster-svc
    build:
      context: .
      dockerfile: src/Core.Cluster.Service/Dockerfile
  core-cluster-svc-1:
    image: core-cluster-svc
    depends_on: 
      - core-cluster-svc
  core-cluster-svc-2:
    image: core-cluster-svc
    depends_on: 
      - core-cluster-svc

Please note that Core.Cluster.Service's purpose is similar to that of Lighthouse's, however I've made it run on top of .net core 2.1 & had it utilize constructs from Microsoft.Extensions.Hosting & Microsoft.Extensions.Configuration

I've performed connectivity check between:

core-cluster-svc & core-cluster-svc1
core-cluster-svc & core-cluster-svc2

The checks have been completed via telnet:

I've attached to the running container via: docker exec -it <core-cluster-svc_containerId> /bin/bash

And then executed:

apt-get update
apt-get install telnet
telnet core-cluster-svc-1 9000
telnet core-cluster-svc-2 9000

Both connections were successful as I'd expect them to be, since docker-compose creates a docker-network with dns support.

mmisztal1980 on 10 Sep 2018

@annymsMthd and @Lutando Are you have a code demo to test it?. I will run it under Kubernetes.

alexsandro-xpt on 10 Sep 2018

@Aaronontheweb

I've set akka.remote.dot-netty.tcp.log-transport = true and set the overall logging level to DEBUG.

Here's the output I've captured:
log.txt

The below solution is being run under Docker for Windows (linux containers).

version: '2.4'

services:
  dmon-telemetry-db:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.0
    healthcheck:
      test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
      interval: 30s
      timeout: 30s
      retries: 3
  dmon-telemetry-db-dashboard:
    depends_on: 
      dmon-telemetry-db:
          condition: service_healthy
    image: docker.elastic.co/kibana/kibana:6.3.0
  core-cluster-svc: # This is an equivalent of Lighthouse running under .net core 2.1. Instance 1/3
    depends_on: 
      dmon-telemetry-db:
          condition: service_healthy
    image: core-cluster-svc
    build:
      context: .
      dockerfile: src/Core.Cluster.Service/Dockerfile
  core-cluster-svc-1: # Instance 2/3
    image: core-cluster-svc
    depends_on: 
      - core-cluster-svc
  core-cluster-svc-2: # Instance 3/3
    image: core-cluster-svc
    depends_on: 
      - core-cluster-svc

Hope that helps a bit.

mmisztal1980 on 12 Sep 2018

@mmisztal1980

[35mcore-cluster-svc-2_1           |[0m Serilog: 2018-09-12T20:23:28.0208726Z Exception caught while converting property value: DotNetty.Common.Utilities.IllegalReferenceCountException: Illegal reference count of 0 for this object
[35mcore-cluster-svc-2_1           |[0m    at DotNetty.Buffers.ThrowHelper.ThrowIllegalReferenceCountException(Int32 count)
[35mcore-cluster-svc-2_1           |[0m    at DotNetty.Buffers.UnpooledHeapByteBuffer.get_Capacity()
[35mcore-cluster-svc-2_1           |[0m    at DotNetty.Buffers.AbstractByteBuffer.ToString()
[35mcore-cluster-svc-2_1           |[0m    at Serilog.Capturing.PropertyValueConverter.CreatePropertyValue(Object value, Destructuring destructuring, Int32 depth)
[35mcore-cluster-svc-2_1           |[0m    at Serilog.Capturing.PropertyValueConverter.CreatePropertyValue(Object value, Destructuring destructuring)

Sure looks like DotNetty catching fire under the covers. Problem with their reference counting system...

Aaronontheweb on 12 Sep 2018

👍1

So I'll need to investigate why this happens on .NET Core 2.1 but not elsewhere (AFAIK) - I'll begin an investigation and I'm going to upgrade this issue to "confirmed bug."

Aaronontheweb on 12 Sep 2018

This is also neither a Windows or Linux-specific issue. Appears to be both runtimes.

Aaronontheweb on 12 Sep 2018

I saw this issue in K8s, I put together a repo with some net/akka versions that have will not form a cluster happily. https://gitlab.com/ian.horton.vibe/akka-cluster-issue/

ianhorton on 22 Sep 2018

Thanks for sending this.. I'm trying to imagine what change we could have introduced in v1.3.9 that could have caused this, but only on Linux

Sent from my iPhone

On Sep 22, 2018, at 12:35 PM, Ian Horton notifications@github.com wrote:

I saw this issue in K8s, I put together a repo with some net/akka versions that have will not form a cluster happily. https://gitlab.com/ian.horton.vibe/akka-cluster-issue/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Aaronontheweb on 22 Sep 2018

I'm convinced that part of the solution to this problem relies in solving https://github.com/akkadotnet/akka.net/issues/3573 - we've had issues in the past with .NET Core on Linux demonstrating different behavior than it does on Windows and outside of a few built-in specs we have inside the Akka.Remote.Tests and Akka.Cluster.Tests libraries, we don't do much in the way of automated End2End testing on Linux. This stems largely from a decision we made last year to punt on trying to get the MNTR to run on POSIX systems.

It's time to revisit that and fix it - only way we can catch these issues on Linux. Assuming that we're going to get identical behavior on Windows and Linux from the same runtime and dependencies is clearly not good enough.

Aaronontheweb on 22 Sep 2018

Just wondering if there have been any new developments in this area? I need to do some work in Akka.NET vs .net core, so should I wait or should I roll back to .net core 2.0 runtime?

mmisztal1980 on 26 Sep 2018

Safest course of action would be to do a rollback until we get to the bottom of what's going on. I'm fighting for every spare minute I have here to work on this but I'm not in a position to offer a fix-by date.

Aaronontheweb on 27 Sep 2018

👍1

FYI: .NET Core 2.0 reaches end-of-life on 10/1.
https://www.microsoft.com/net/platform/support-policy

Just saying...
;)

pshrosbree on 28 Sep 2018

Guess it's a good thing we target .NET Standard then :p

Aaronontheweb on 28 Sep 2018

Not on Linux its not.
😉

Regards,
Peter

From: Aaron Stannard notifications@github.com
Sent: Thursday, September 27, 2018 3:27 PM
To: akkadotnet/akka.net
Cc: Peter Shrosbree; Comment
Subject: Re: [akkadotnet/akka.net] DotNetty Remote Transport Issues with .NET Core 2.1 (#3506)

Guess it's a good thing we target .NET Standard then :p

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/akkadotnet/akka.net/issues/3506#issuecomment-425262532, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AA5KcP1McIIamHB_-G-r7cIBAYYKJz3Cks5ufVDagaJpZM4UmTHi.

pshrosbree on 28 Sep 2018

Deleted my previous comment - turns out that was definitely not the latest sources. It was the master branch from about a year ago 👍...

Aaronontheweb on 9 Oct 2018

👍1

As fate would have it... Same issue with DotNetty using the v1.3.8 sources, using Ubuntu 16.04 via the Windows sub-system.

Unhandled Exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (The type initializer for 'DotNetty.Transport.Channels.DefaultChannelId' threw an exception.))))) ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (The type initializer for 'DotNetty.Transport.Channels.DefaultChannelId' threw an exception.)))) ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (The type initializer for 'DotNetty.Transport.Channels.DefaultChannelId' threw an exception.))) ---> System.AggregateException: One or more errors occurred. (One or more errors occurred. (The type initializer for 'DotNetty.Transport.Channels.DefaultChannelId' threw an exception.)) ---> System.AggregateException: One or more errors occurred. (The type initializer for 'DotNetty.Transport.Channels.DefaultChannelId' threw an exception.) ---> System.TypeInitializationException: The type initializer for 'DotNetty.Transport.Channels.DefaultChannelId' threw an exception. ---> System.Net.NetworkInformation.NetworkInformationException: An error was encountered while querying information from the operating system. ---> System.AggregateException: One or more errors occurred. (Could not find a part of the path '/proc/sys/net/ipv4/conf/eth0/forwarding'.) (Could not find a part of the path '/proc/s

Aaronontheweb on 9 Oct 2018

Going to give that another full rebuild locally here though on the off-chance that something was cached from an earlier build.

Aaronontheweb on 9 Oct 2018

Investigating this issue in more detail by spinning up a WebCrawler cluster in docker-compose with Akka.NET v1.3.9 installed.

Cluster formed just fine with .NET Core 1.1 Lighthouse (Akka.NET v1.3.4) and .NET Core 2.0 Web / Tracker / Crawler nodes (v1.3.9.)

Cluster formed just fine with .NET Core 2.1 Lighthouse (Akka.NET v1.3.9) and .NET Core 2.0 Web / Tracker / Crawler nodes (v1.3.9.)

Aaronontheweb on 10 Oct 2018

I'll be updating those projects tonight with some repro instructions you can use, but so far it looks like my Docker environment on Linux (using whatever the default OS is on the .NET Core images, same as @ianhorton's K8s example.)

I'm going to try to get your K8s example running too, but I had some issues with MiniKube on my office computer. I'll try at home on my Ryzen machine tonight.

Aaronontheweb on 10 Oct 2018

Cluster formed just fine with .NET Core 2.1 Lighthouse (Akka.NET v1.3.9) and .NET Core 2.0 Web / Tracker / Crawler nodes (v1.3.9.)

As far as I recall, my sample uses the 2.1 version of the web project.

mmisztal1980 on 10 Oct 2018

@mmisztal1980 I'll try it with that next, but I need to update the docker-compose files since the 2.1 build tools added some new build errors that get thrown if you only copy over some project files but not others

Aaronontheweb on 10 Oct 2018

👍1

I'll be updating those projects tonight with some repro instructions you can use, but so far it looks like my Docker environment on Linux (using whatever the default OS is on the .NET Core images, same as @ianhorton's K8s example.)

I'm going to try to get your K8s example running too, but I had some issues with MiniKube on my office computer. I'll try at home on my Ryzen machine tonight.

minikube version 0.28.0 is the latest version I can get working consistently. I generally find deleting the folder at \Users\.minikube is necessary as well...

ianhorton on 10 Oct 2018

Comment from the DotNetty project lead:

@Aaronontheweb, you guys may want to update rather soon. In .net core 2.1 Socket ConnectAsync now sometimes completes synchronously and that exposed a bug in connect handling in dotnetty.

Aaronontheweb on 24 Oct 2018

Yeah, this sure smells like the problem:

https://github.com/Azure/DotNetty/pull/413

Aaronontheweb on 24 Oct 2018

❤1

Doesn't mean I don't want to still finish investigating the issue on the MNTR, but sounds a lot like what's been reported.

Aaronontheweb on 24 Oct 2018

This too https://github.com/Azure/DotNetty/issues/422

Aaronontheweb on 24 Oct 2018

I'll be updating those projects tonight with some repro instructions you can use, but so far it looks like my Docker environment on Linux (using whatever the default OS is on the .NET Core images, same as @ianhorton's K8s example.)
I'm going to try to get your K8s example running too, but I had some issues with MiniKube on my office computer. I'll try at home on my Ryzen machine tonight.

minikube version 0.28.0 is the latest version I can get working consistently. I generally find deleting the folder at \Users.minikube is necessary as well...

@ianhorton @Aaronontheweb you guys might wish to try out Kubernetes support that comes along with Docker for Windows. IMHO it's a more stable experience than the minikube one

mmisztal1980 on 24 Oct 2018

👍1

This helped me figure out how to finally get K8s running on the new Docker for Windows releases: https://devkimchi.com/2018/06/05/running-kubernetes-on-wsl/

Aaronontheweb on 1 Nov 2018

@ianhorton I was able to get your reproduction samples up and running on K8s using Docker for Windows and yep, I can reproduce the problem locally using the Windows Subsystem for Linux. This is very helpful. Thank you!

Aaronontheweb on 1 Nov 2018

👍3

Received this error message when re-running this sample on the latest DotNetty (v0.6.0):

led to associate with tcp://cluster-system@cluster-seed-netcore21-akka139-1.cluster-seed-netcore21-akka139:2550) ---> Akka.Remote.Transport.InvalidAssociationException: Failed to associate with tcp://cluster-system@cluster-seed-netcore21-akka139-1.cluster-seed-netcore21-akka139:2550 ---> DotNetty.Transport.Channels.ConnectException: LogError connecting to 10.1.0.6:2550 ---> System.Net.Sockets.SocketException: Transport endpoint is not connected
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.Shutdown(SocketShutdown how)
   at DotNetty.Transport.Channels.Sockets.TcpSocketChannel.DoClose()
   at DotNetty.Transport.Channels.Sockets.TcpSocketChannel.DoConnect(EndPoint remoteAddress, EndPoint localAddress)
   at DotNetty.Transport.Channels.Sockets.AbstractSocketChannel.AbstractSocketUnsafe.ConnectAsync(EndPoint remoteAddress, EndPoint localAddress)
   --- End of inner exception stack trace ---
   at DotNetty.Transport.Bootstrapping.Bootstrap.DoResolveAndConnectAsync(EndPoint remoteAddress, EndPoint localAddress)
   at Akka.Remote.Transport.DotNetty.TcpTransport.AssociateInternal(Address remoteAddress)
   --- End of inner exception stack trace ---
   at Akka.Remote.Transport.DotNetty.TcpTransport.AssociateInternal(Address remoteAddress)
   at Akka.Remote.Transport.DotNetty.DotNettyTransport.Associate(Address remoteAddress)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Akka.Remote.Transport.ProtocolStateActor.<>c.<InitializeFSM>b__11_55(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---

So Failed to associate with tcp://cluster-system@cluster-seed-netcore21-akka139-1.cluster-seed-netcore21-akka139:2550 ---> DotNetty.Transport.Channels.ConnectException: LogError connecting to 10.1.0.6:2550 ---> System.Net.Sockets.SocketException: Transport endpoint is not connected is a new one for me.

Error message when using DotNetty v0.4.8:

AssociationError [akka.tcp://ClusterSystem@cluster-seed-netcore21-akka138-0.cluster-seed-netcore21-akka138:2552] -> akka.tcp://cluster-system@cluster-seed-netcore21-akka138-1.cluster-seed-netcore21-akka138:2550: Error [Association failed with akka.tcp://cluster-system@cluster-seed-netcore21-akka138-1.cluster-seed-netcore21-akka138:2550] []
[WARNING][11/06/2018 09:52:25][Thread 0007][remoting] Tried to associate with unreachable remote address [akka.tcp://cluster-system@cluster-seed-netcore21-akka138-1.cluster-seed-netcore21-akka138:2550]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Association failed with akka.tcp://cluster-system@cluster-seed-netcore21-akka138-1.cluster-seed-netcore21-akka138:2550] Caused by: [System.AggregateException: One or more errors occurred. (Connection refused tcp://cluster-system@cluster-seed-netcore21-akka138-1.cluster-seed-netcore21-akka138:2550) ---> Akka.Remote.Transport.InvalidAssociationException: Connection refused tcp://cluster-system@cluster-seed-netcore21-akka138-1.cluster-seed-netcore21-akka138:2550
   at Akka.Remote.Transport.DotNetty.TcpTransport.AssociateInternal(Address remoteAddress)
   at Akka.Remote.Transport.DotNetty.DotNettyTransport.Associate(Address remoteAddress)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Akka.Remote.Transport.ProtocolStateActor.<>c.<InitializeFSM>b__11_54(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)

So it's worth taking a look at the DotNetty source to see what all has changed here - appears to be a low-level sockets issue still.

Aaronontheweb on 6 Nov 2018

👍1

My theory on this error is that we have a concurrency bug happening somewhere inside the DotNetty stack.

This error message:

System.Net.Sockets.SocketException: Transport endpoint is not connected

Seems to indicate to me that we're trying to use a Socket object before the connection has actually had an opportunity to complete successfully. Whatever they did to fix the "synchronous socket connect" issue after DotNetty v0.4.8 may have introduced this problem.

Aaronontheweb on 7 Nov 2018

Previous statement made based on the content of the error message and some inferences I've made based on the PRs I looked over on the DotNetty repo.

I'll pull down the DotNetty test suite and see if I can take a crack at it there.

Aaronontheweb on 7 Nov 2018

So I think this was the change that caused the error message to become different between DotNetty v0.4.8 and v0.6.0 https://github.com/Azure/DotNetty/pull/413

If I were to translate that SocketException into something we understand in the Windows universe, based on Windows Socket Error codes, looks to me like the underlying socket would return a WSAENOTCONN _10057_.

The v0.6.0 upgrade should eliminate this issue for Windows users, but it definitely hasn't fixed the issue at all for Linux users.

Therefore, I have to wonder if this code inside DotNetty:

        protected override bool DoConnect(EndPoint remoteAddress, EndPoint localAddress)
        {
            if (localAddress != null)
            {
                this.Socket.Bind(localAddress);
            }

            bool success = false;
            try
            {
                var eventPayload = new SocketChannelAsyncOperation(this, false);
                eventPayload.RemoteEndPoint = remoteAddress;
                bool connected = !this.Socket.ConnectAsync(eventPayload);
                if (connected)
                {
                    this.DoFinishConnect(eventPayload);
                }
                success = true;
                return connected;
            }
            finally
            {
                if (!success)
                {
                    this.DoClose();
                }
            }
        }

I have to wonder if the this.Socket.ConnectAsync call returns different values depending on whether or not you're running on Windows or on Linux. That would explain why my attempts to re-run @ianhorton's reproduction sample still didn't result in a cluster forming on .NET Core 2.1.

Looks like I'm going to need to take a look at the actual socket implementation in .NET Core 2.1 and try to understand wtf is going on there.

Aaronontheweb on 13 Nov 2018

❤1

Created a real sample to try to reproduce this using DotNetty only: https://github.com/Aaronontheweb/DotNettyRepro

I can get it to work on .NET Core 2.1 and .NET Core 2.0, so I'm going to need to take a look at our remoting code and see if there's something we need to change in the architecture of the DotNettyTransport itself.