Grpc-dotnet: .NET Core 3 fails accepting Java client calls

[thefolenangel]

Issue description

We are transitioning our micro-services from .NET Core 2.1 with the old gRPC dotnet nuget, to .NET Core 3.0 with the inbuilt support for gRPC in Kestrel.

One of our clients is Android app.
When tried to connect from the Android client:

Failed... : 
io.grpc.StatusRuntimeException: INTERNAL: Protocol error
Rst Stream
    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235)
    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
    at io.grpc.examples.helloworld.GreeterGrpc$GreeterBlockingStub.sayHello(GreeterGrpc.java:228)
    at io.grpc.helloworldexample.HelloworldActivity$GrpcTask.doInBackground(HelloworldActivity.java:90)
    at io.grpc.helloworldexample.HelloworldActivity$GrpcTask.doInBackground(HelloworldActivity.java:72)
    at android.os.AsyncTask$2.call(AsyncTask.java:333)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:764)

Set up is Development, it's over HTTP, no TLS has been enabled.

Someone reported the same issue on stackoverflow for Kotlin

I want to point out that we tried the same Stubs with different clients like .NET Core 3.0/ Swift and even BlazorRPC and they worked without an issue.

General version information

gRPC:
Java uses : 1.23.0
.NET uses: 2.23.2
Windows 10
Docker for Win 19.03.2

Images used for building the docker final image:
mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim AS base
mcr.microsoft.com/dotnet/core/sdk:3.0-buster AS build

How to reproduce:

1. Download .NET Greeter example: https://github.com/grpc/grpc-dotnet/tree/master/examples/Greeter
2. Download Java Greeter for Android: https://github.com/grpc/grpc-java/tree/master/examples/android/helloworld
3. Modify proto Greeter file in .NET example by adding these extra options before the package Name:

syntax = "proto3";

option java_multiple_files = true;
option java_package = "io.grpc.examples.helloworld";
option java_outer_classname = "HelloWorldProto";
option objc_class_prefix = "HLW";

package Greet;

1. Make sure Java client Uses the modified proto to generate Stubs.
2. Run server in Docker
3. Get port of container and connect via Local network using IPv4 Address
4. Call server from Client

[ivan-penchev]

Hi,

We have the same issue, I reported it here:
https://github.com/grpc/grpc-java/issues/6202

There is an repo for reproduction here:
https://github.com/ivan-penchev/gRPC-Java-dotnetcore3/tree/master

Also this is client long for package sniff:
https://github.com/ivan-penchev/gRPC-Java-dotnetcore3/blob/master/client-packet-sniff-full

[JamesNK]

I think this is your problem:

https://docs.microsoft.com/en-us/aspnet/core/grpc/troubleshoot?view=aspnetcore-3.0#mismatch-between-client-and-service-ssltls-configuration

The gRPC template and samples use Transport Layer Security (TLS) to secure gRPC services by default. gRPC clients need to use a secure connection to call secured gRPC services successfully.

Either use a secure connection from the client, or disable TLS on the server.

[ivan-penchev]

Interesting suggestion for a solution? Because I could swear I have done it. Maybe I have missed something ?
How do you disable TLS on the server explicitly?
Currently Java builds the channel like this:

  ManagedChannelBuilder managedChannelBuilder = ManagedChannelBuilder
                    .forAddress(mUrl, mPort)
                    .keepAliveTime(3, TimeUnit.MINUTES)
                    .keepAliveTimeout(30, TimeUnit.SECONDS).
                    .usePlaintext()
                    .build();

On the server side I use the appsettings.json

{
  "Kestrel": {
    "Endpoints": {
      "Grpc": {
        "Url": "http://*:13377",
        "Protocols": "Http2"
      }
    }
  }
}

As you can notice there is no "Certificate" path provided.
@JamesNK thank you for the assistance.

[JamesNK]

Edit: Actually, looking at your appsettings.json you might already not have TLS on that endpoint. I'm not incredibly familiar with setting up endpoints via appsettings.json.

---

ASP.NET Core automatically provides a HTTPS endpoint using a development certificate.

By default:

• Port 5000 = http = no TLS
• Port 5001 = https = TLS

Connect to port 5000.

More info: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-3.0#endpoint-configuration

[ivan-penchev]

Thanks James I checked the link, and even though I don't see much difference in regards of configuring an endpoing via json config, tomorrow I am definitely going to try to not use a custom port, but rather the default port 5000 and see if that fixes it.

However shouldn't this behavior be rather flexible rather than dependable on default ports?

Is it possible to CC someone that is a bit more familiar with endpoint configuration, just for sanity check, since the configuration above (which also included an http1.1 on a different port) worked without a problem on .NET Core preview7? (I checked the release notes from that preview to GA and there seems to not be any breaking changes in this area).

Edit:
Just fyi we have not enabled

app.UseHsts();
app.UseHttpsRedirection();

[JamesNK]

Custom ports should work. Although some OSes block certain port ranges.

I will try your appsetting.json when I have time.

One thing you could try is configuring the endpoint in Program.cs:

.ConfigureKestrel((context, options) =>
{
    options.ListenAnyIP(50052, listenOptions =>
    {
        listenOptions.Protocols = HttpProtocols.Http2;
    });
})

Uses port 50052 which I think is relatively safe to use.

[ivan-penchev]

Hi @JamesNK , I read in details "Make it easier to configure Kestrel endpoints from config" issue and based on that I do not think my config was incorrect.
Never the less I tried your solution. In my Program.cs:

...
 .ConfigureKestrel((context, kestrelOptions) =>
                {
                    kestrelOptions.ListenAnyIP(50052, listenOptions =>
                    {
                        listenOptions.Protocols = HttpProtocols.Http2;
                    });
                    kestrelOptions.ListenAnyIP(50051, listenOptions =>
                    {
                        listenOptions.Protocols = HttpProtocols.Http1;
                    });
                })
...

And the problem still persist. I am able to access all the grpc endpoints, just not using Java client.

I have updated the initial repo , where I could reproduce the problem with the official Greeter examples for Java and .NET , with your suggestion. But I also left my initial, appsettings.json, configuration just commented it out

Now if you run the solution using docker-compose, you would have 3 endpoints (All set to not use TLS):

1. http://localhost/testgrpc (Webapi that has the .NET Greeter Client you can use for teting)[Client]
   --this call uses unencrypted communication between Client and Server
2. http://localhost:50051/test (Just an normal HTTP 1.1 endpoint on the server, for sanity check)[Server]
3. http://localhost:50052/ (Greeter server service running on HTTP2)[Server]

However if you run the Android solution and try to connect to the server[http://localhost:50052/] (that works fine with .NET Client) you would get the same exception, that was in the original post.

I would also like to point out that, there are problems with people using .NET Client, trying to connect to Java Server.

Is it possible that there is some sort of misalignment, between Java and .NET?

Edit:
Maybe it would help also a bit, if there is a reference in the issue how we build the Android Client:

//host and port variables are takes from the input boxes.
channel = ManagedChannelBuilder.forAddress(host, port).usePlaintext().build();
GreeterGrpc.GreeterBlockingStub stub = GreeterGrpc.newBlockingStub(channel);

[JamesNK]

Could the issue be that the Android client doesn't support plaintext (aka non-TLS) HTTP/2 calls?

[ivan-penchev]

Hi @JamesNK short answer should be NO, because internally we develop on plaintext for 1.5 years, however we used: https://github.com/grpc/grpc/tree/master/src/csharp (For server side)

NB!: The grpc guys forked OkHTTPClient and did some internal changes for the purpose of gRPC.

Btw if I plug this: https://github.com/grpc/grpc/tree/master/examples/csharp/Helloworld
It works flawlessly. Also if I test the new .NET Client on the old .NET Server (from the example above) it also works.

Today I am going to do everything securely and inform you what would the result of that be.

[JamesNK]

Ok. When I have time I will see if I can get Android setup on my computer and test myself.

[ivan-penchev]

@JamesNK Over SSL it works fine.

[JamesNK]

I've managed to recreate this. I suspect the issue is in Kestrel's implementation of HTTP/2. Something about the HTTP/2 request from the Android client is causing Kestrel to reject it.

Kestrel issue: https://github.com/aspnet/AspNetCore/issues/14745

[jtattermusch]

CC @jtattermusch

[JamesNK]

Closing as fixed in Java client. Kestrel discussion: https://github.com/aspnet/AspNetCore/issues/14745