Grpc-dotnet: Getting "System.InvalidOperationException: Reading is already in progress" With BiDirectional Streaming

[thefringeninja]

What version of gRPC and what language are you using?

2.24.0 and C#

What operating system (Linux, Windows,...) and version?

Linux Mint 19.2

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET Core SDK (reflecting any global.json):
 Version:   3.1.100-preview3-014645
 Commit:    b32d27f4b3

Runtime Environment:
 OS Name:     linuxmint
 OS Version:  19.2
 OS Platform: Linux
 RID:         linuxmint.19.2-x64
 Base Path:   /snap/dotnet-sdk/53/sdk/3.1.100-preview3-014645/

Host (useful for support):
  Version: 3.1.0-preview3.19553.2
  Commit:  13f35c3d86

.NET Core SDKs installed:
  3.1.100-preview3-014645 [/snap/dotnet-sdk/53/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.0-preview3.19555.2 [/snap/dotnet-sdk/53/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.0-preview3.19553.2 [/snap/dotnet-sdk/53/shared/Microsoft.NETCore.App]

What did you do?

This is a follow up of #624 and aspnet/AspNetCore#15415

First off I wanna say thanks for getting that change into Kestrel, means I don't need to stand up a server which is great. However I'm having what I believe is another regression. If I use the Kestrel TestServer, and the server throws an RpcException during the call, instead of getting the RpcException on the client side I get an InvalidOperationException: Reading is already in progress." from DefaultPipeReader.TryRead. Unfortunately I cannot create a small repro, but you can see an example failing test here: https://github.com/thefringeninja/EventStore/blob/grpc/src/EventStore.Grpc.PersistentSubscriptions.Tests/PersistentSubscriptions/connect_to_existing_persistent_subscription_without_permissions.cs

What did you expect to see?

An AccessDeniedException.

What did you see instead?

System.InvalidOperationException: Reading is already in progress.
  at at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_AlreadyReading()
  at at System.IO.Pipelines.Pipe.TryRead(ReadResult& result)
  at at System.IO.Pipelines.Pipe.DefaultPipeReader.TryRead(ReadResult& result)
  at at Microsoft.AspNetCore.TestHost.HttpContextBuilder.CompleteRequestAsync()
  at at Microsoft.AspNetCore.TestHost.HttpContextBuilder.<>c__DisplayClass23_0.<<SendAsync>g__RunRequestAsync|0>d.MoveNext()
  at at Microsoft.AspNetCore.TestHost.ClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  at EventStore.Grpc.EventStoreGrpcFixture.ResponseVersionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/jwow/dev/EventStore/src/EventStore.Grpc.Tests.Common/EventStoreGrpcFixture.cs:86

[JamesNK]

The problem is you're reading the request in the background and you finish the gRPC method with a pending read. You really shouldn't do that. Kestrel will handle that problem to the best of its ability, but TestServer is more aggressive about telling you that you've done something wrong.

This method needs a try/finally that will cancel the background read going on here: Task.Run(() => requestStream.ForEachAsync(HandleAckNack));, and then await the task to ensure it finishes.

btw I did find another bug in TestServer that I've fixed here - https://github.com/aspnet/AspNetCore/pull/17158. You may run into it.

[thefringeninja]

@JamesNK can you elaborate please? There's something I'm not grokking here.

[JamesNK]

public void Task DoThing(IAsyncStreamReader<RaceMessage> requestStream, IServerStreamWriter<RaceMessage> responseStream, ServerCallContext context)
{
    var readTask = requestStream.MoveNext(); // waiting for a message form the client

    // exit server method
}

[JamesNK]

Improved the error message to make this situation easier for devs to understand with https://github.com/aspnet/AspNetCore/pull/17164

Closing.