Akka.net: Eliminate or fix racy unit tests

[Aaronontheweb]

Version: Akka.NET v1.3.13 or later.

The time has come, once again, to clean up the mess that impedes our work on Akka.NET as a result of some racy tests.

• [x] [DistributedPubSubMessageSerializerSpec.Can_serialize_SendToOneSubscriber](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56858&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - .NET Core Windows
• [x] [DistributedPubSubMessageSerializerSpec.Can_serialize_Status](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56858&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - .NET Core Windows
• [ ] [AkkaProtocolSpec.ProtocolStateActor_must_in_inbound_mode_accept_payload_after_Associate_PDU_received](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56858&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - .NET Core Windows
• [x] [FlowDelaySpec.A_Delay_must_pass_elements_with_delay_through_normally_in_backpressured_mode ](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56858&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - .NET Core Windows
• [x] [BackoffSupervisorSpec.BackoffSupervisor_must_stop_restarting_the_child_if_final_stop_message_received_using_BackOff_OnStop](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [BackoffSupervisorSpec.BackoffSupervisor_must_stop_restarting_the_child_after_reaching_maxNrOfRetries_limit_using_BackOff_OnFailure](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [RemoteConnectionSpecs.RemoteConnection_should_send_and_decode_messages](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [TailChoppingSpec.Tail_chopping_group_router_must_throw_exception_if_no_result_will_arrive_within_the_given_time](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [LazySinkSpec.A_LazySink_must_cancel_upstream_when_internal_sink_is_cancelled ](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [LazySinkSpec.A_LazySink_must_fail_gracefully_when_upstream_failed](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [ ] [SnapshotDirectoryFailureSpec.LocalSnapshotStore_configured_with_a_failing_directory_name_should_throw_an_exception_at_startup](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [FlowSelectAsyncUnorderedSpec.A_Flow_with_SelectAsyncUnordered_must_not_run_more_futures_than_configured](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56859&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [RemotingSpec.Remoting_must_lookup_actors_across_node_boundaries](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56861&tab=buildResultsDiv&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - Windows .NET Core
• [x] [PersistentActorSpec.PersistentActor_should_be_able_to_persist_events_that_happen_during_recovery](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56862&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [x] [FlowDelaySpec.A_Delay_must_properly_delay_according_to_buffer_size](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56871&buildTypeId=AkkaNet_AkkaNetWindowsUnitTests&guest=1) - .NET Framework
• [ ] [AkkaProtocolSpec.ProtocolStateActor_must_in_inbound_mode_accept_payload_after_Associate_PDU_received](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56880&tab=buildResultsDiv&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - .NET Core Windows
• [x] [ObservableSinkSpec.An_ObservableSink_must_allow_the_same_observer_to_be_subscribed_only_once](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56969&tab=buildResultsDiv&buildTypeId=AkkaNet_AkkaNetLinuxNetCoreUnitTests&guest=1) - .NET Core Linux
• [x] [RemotingSpec.Bug_884_Remoting_must_support_reply_to_Routee](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=57012&tab=buildResultsDiv&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - .NET Core Windows
• [x] [StageActorRefSpec.A_Graph_stage_ActorRef_must_ignore_and_log_warnings_for_PoisonPill_and_Kill_messages](http://petabridge-ci.cloudapp.net/viewLog.html?buildId=56990&tab=buildResultsDiv&buildTypeId=AkkaNet_AkkaNetWindowsUnitTestsNetCore&guest=1) - .NET Core Windows
• [x] [Akka.Persistence.Tests.AtLeastOnceDeliverySpec.AtLeastOnceDelivery_must_limit_the_number_of_messages_redelivered_at_once](https://dev.azure.com/dotnet/Akka.NET/_build/results?buildId=17518&view=ms.vss-test-web.build-test-results-tab&runId=1041238&resultId=100731&paneView=debug) - .NET Core Windows

Try to fix them, but failing that - mark them as Skipped inside XUnit.

[Aaronontheweb]

Important detail about why Akka.Remote.Tests.RemotingSpec.Remoting_must_lookup_actors_across_node_boundaries fails occasionally

[DEBUG][6/4/2019 3:04:54 PM][Thread 0137][LocalActorRefProvider(akka://RemotingSpec-101)] Resolve of path sequence [/temp/k] failed

Looks like a racy issue with looking up temporary actors when the actor on the other side of the network sets up a reply - might need to take a look at the VirtualPathContainer if a round-trip I/O operation is able to complete before the recipient FutureActorRef is able to register with it.

[Aaronontheweb]

Relevant logs: https://dev.azure.com/dotnet/Akka.NET/_build/results?buildId=17879&view=ms.vss-test-web.build-test-results-tab&runId=1043496&paneView=attachments&resultId=100886

[Aaronontheweb]

Looks like a racy issue with looking up temporary actors when the actor on the other side of the network sets up a reply - might need to take a look at the VirtualPathContainer if a round-trip I/O operation is able to complete before the recipient FutureActorRef is able to register with it.

Spent a lot of time looking into this over the past two days while before I decided to just skip the racy remoting specs. We're not doing anything terribly fancy inside the VirtualPathContainer - all of the concurrent state is managed using a ConcurrentDictionary and that's about it. I don't see how the state could be unsafe inside of it, so scratch that idea. The failure to resolve the path is real and I should note that I'm having an easier time reproducing it with the Debugger attached on my local machine - meaning that I think something needs to be running slightly slower than usual in order for the issue to appear. I don't know what kind of hardware Azure DevOps is using to run the CI pipelines, but it might be busier than what we were using before on our dedicated D2V2 instances.

[Aaronontheweb]

cc @IgorFedchenko @nagytech - so here's the list of the tests with the highest flip rates: https://dev.azure.com/dotnet/Akka.NET/_test/analytics?definitionId=84&contextType=build

I'd recommend adding a comment to here and say which test you're working on. Some tests might just be crappy / bad tests that need to be deleted or disabled. Others might reveal a real underlying problem with the infrastructure and take some time to debug. Others might be simple fixes (i.e. using AwaitAssert instead of a simple assertion, for testing a volatile condition)

[IgorFedchenko]

Working on
Akka.Cluster.Sharding.Tests.CoordinatedShutdownShardingSpec.Sharding_and_CoordinatedShutdown_must_run_successfully
Akka.Cluster.Sharding.Tests.DisabledInactiveEntityPassivationSpec.Passivation_of_inactive_entities_must_not_passivate_when_passivation_is_disabled
Akka.Tests.IO.TcpIntegrationSpec.Should_fail_writing_when_buffer_is_filled

[Aaronontheweb]

Looking good now - had several big PRs run the full test suite without any racy issues. Going to mark this as closed for the time being.