Grpc-dotnet: JSON/REST transcoding

[cwe1ss]

Hi, first, thanks for all the work you put into this library! I'm really looking forward to using it!!

Are there any plans to support the REST-like HTTP/JSON transcoding (https://cloud.google.com/endpoints/docs/grpc/transcoding ) via the http service annotations?

Having that together with OpenAPI/Swagger is a HUGE help during development when you can just call the API via JSON objects.

---

2020/01/14 UPDATE: https://github.com/grpc/grpc-dotnet/issues/167#issuecomment-574005197

---

2020/04/24 UPDATE:

• Introducing gRPC HTTP API blog post
• Microsoft.AspNetCore.Grpc.HttpApi is added to NuGet.org

---

2020/12/04 UPDATE:

• Create JSON Web APIs from gRPC is on docs.microsoft.com
• Microsoft.AspNetCore.Grpc.Swagger is added to NuGet.org

[JamesNK]

No plans to do it for 3.0. Beyond 3.0, it depends on how many people ask for it.

[JunTaoLuo]

We're keeping this issue open to solicit feedback and see how much demand there exists for this feature.

[jtattermusch]

FTR, regardless of whether this gets implemented or not, there are multiple proxies that currently offer this functionality:
https://github.com/grpc-ecosystem/grpc-gateway
https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/grpc_json_transcoder_filter

[govis]

We need HTTP/JSON to gRPC transcoding so that same plumbing/service can be used to support REST and gRPC clients

[raz-canva]

This would be super helpful to be provided out of the box. As opposed to setting up grpc gateway

[Jonathan34]

Also interested. We use grpc gateway in golang but need the same functionality in dotnet as we are a big .net consumer.

[bobrot]

Also interested in this. While there are 3rd party tools that can do this, a simpler implementation is always better.

[Jonathan34]

if i had to start again, I'd probably use an envoy proxy instead similar to that: https://blog.envoyproxy.io/envoy-and-grpc-web-a-fresh-new-alternative-to-rest-6504ce7eb880
so at least I can have the same solution and transcoding for all the languages

[VladimirLuzhin]

That would be very cool since would allow smooth transition from JSON API to GRPC

[Bob9098-source]

We're ru

[JamesNK]

We have an experimental project that allows JSON/REST with gRPC here: https://github.com/aspnet/AspLabs/tree/master/src/GrpcHttpApi

The README.md at the link has details about how to use it and its status as a project.

[ElderJames]

Looking forward to this feature.

[dashkan]

Great job!
I think this is a must have for dotnet core framework and tooling. Why proxy gRPC calls when you don't have to. Much more elegant IMO.

[TomDeVree]

This is very helpful. Should it be possible to use Swagger (for example Swashbuckle) with this setup? For now it seems it cannot find any operations.

[JamesNK]

There is currently no way to generate Swagger. I think a Swashbuckle extension would need to be created that lets Swashbuckle understand gRPC endpoints and protobuf schemas.

[MedAnd]

Thoughts on support for Azure Functions?

cc @jeffhollan

[Otto404]

impressive. I hope to find these functionalities in production soon

[kollerdroid]

Great project. I think GrpcHttpApi should be part of dotnet 5.
Waiting for AuthContext implementation... ;-)

[Dchoksi-brierley]

The gRPC HTTP API is a great solution - almost every .net application that is adding on gRPC will benefit in maintaining existing external REST interfaces via this solution.
We tried it within our team and it works seamlessly on .net core 3.0. However our application is being implemented using .net core 3.1 - any plans on moving the gRPC HTTP API forward?

[JamesNK]

Does it not work on 3.1?

[alejandrobonillamondragon]

Hi there,
I followed the instructions from
https://www.nuget.org/packages/Microsoft.AspNetCore.Grpc.HttpApi... Installed the pre-release version of Microsoft.AspNetCore.Grpc.HttpApi and Microsoft.AspNetCore.Grpc.Swagger (0.1.0-alpha.20254.1) from NuGet official library. However, when opening swagger UI it says "No operations defined in spec!" and the endpoints are not reachable through HTTP (getting 404)... I'm using .NET Core 3.1, I think a similar issue was already reported here: https://github.com/grpc/grpc-dotnet/issues/745. Do you know if Grpc.Swagger works for 3.1?

This is the Startup.cs

using GrpcGateway.Server;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.OpenApi.Models;

namespace GrpcGateway
{
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}

    public IConfiguration Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddGrpcHttpApi();

        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "gRPC HTTP API Example", Version = "v1" });
        });
        services.AddGrpcSwagger();

        //services.AddControllers();
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseHttpsRedirection();
        app.UseDefaultFiles();
        app.UseStaticFiles();

        app.UseSwagger();
        app.UseSwaggerUI(c =>
        {
            c.SwaggerEndpoint("/swagger/v1/swagger.json", "gRPC HTTP API Example V1");
        });

        app.UseRouting();

        //app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapGrpcService<GreeterService>();
        });
    }
}

}

[Dchoksi-brierley]

Thanks for the details Alejandro.. James - please let us know if you have any info for us or need any details about the issue. Thanks so much for responding so quickly - much appreciated!

[JamesNK]

It should work in 3.0. I haven't got time to debug this, but feel free to copy the source code for those packages to your machine and step through the code to find out why you get that message.

[Dchoksi-brierley]

Thanks again for responding back.
We'll debug into the source code - fingers crossed. We were not sure if there were known issues with 3.1 ("Beyond 3.0, it depends on how many people ask for it")... In any case, we'll share what we find - in case it helps any one else.

[JamesNK]

Works fine in 3.1 - https://github.com/aspnet/AspLabs/tree/master/src/GrpcHttpApi/sample

[Dchoksi-brierley]

Many thanks!!
We were able to make progress - adding a reference to Grpc.AspNetCore.Server and Grpc.Tools made the difference. Those two packages were not required to generate the server stubs in the latest version of Grpc.AspNetCore but looks like they were needed for Microsoft.AspNetCore.Grpc.HttpApi.

[alejandrobonillamondragon]

It works in 3.1. In case this helps someone else. In my case the issue was in the proto files, the Proto folder needs to be placed in the solution's root folder and create a service reference in the project using the main .proto file (the one who imports annotations.proto) marking it to generate the server class. I was placing those files in the project folder and generating the server stub directly from it (without the service reference), in that way it doesn't work.

[kollerdroid]

Has Enum type handling already implemented?
I have a field which is completely missing from JSON result.

[JamesNK]

It should work.

Can you post your proto file?

[kollerdroid]

Sure. Thank You.

Using enum, bookDefinition/status is missing:

{
  "id": "1",
  "isbn": "98765",
  "author": "Author",
  "description": "Description",
  "isActive": true,
  "bookGroups": [
    "1",
    "2"
  ],
  "bookDefinition": {
    "id": "1",
    "content": "A longer text...",
    "version": 1
  },
  "readerPermissions": [
    "15",
    "23"
  ],
  "availableVersions": [
    1
  ]
}

Using int32 or string, bookDefinition/status exists:

{
  "id": "1",
  "isbn": "98765",
  "author": "Author",
  "description": "Description",
  "isActive": true,
  "bookGroups": [
    "1",
    "2"
  ],
  "bookDefinition": {
    "id": "1",
    "content": "A longer text...",
    "version": 1,
    "status": 2
  },
  "readerPermissions": [
    "15",
    "23"
  ],
  "availableVersions": [
    1
  ]
}

[crawdell]

Very interested in GrpcHttpApi. Along with OpenAPI / Swagger it would provide us the best overall solution without having to piece it together with other tools.

[meareindel]

Interested too. It would be very conveniently to have OpenAPI + REST on top of gRPC at least for dev/test, but also for external API consumers that cannot use gRPC for some reasons.

[dmytro-pryvedeniuk]

No plans to do it for 3.0. Beyond 3.0, it depends on how many people ask for it.

Guys, could you please share what has influence on your decision? How do you count these people and what can we do to change it? I believe if you make it GA you can expect higher interest than now when it's "very experimental".

[JamesNK]

Guys, could you please share what has influence on your decision?

There is a lot of work to resolve outstanding technical problems, write documentation, security reviews, measure performance, stress test, compatibility tests with grpc-gateway, etc. That needs to be done before it can be endorsed it as a real product. Then there is ongoing work to maintain the product like answering issues, fixing bugs, etc.

How do you count these people and what can we do to change it?

Likes on this issue, NuGet package downloads, community PRs, customers using the experimental in apps.

[KyleGobel]

This is great! It's very nice to use something like this when we're transferring services over to gRPC, allows us to get things going without having to change clients right away too.

I have been wrestling with trying to get some of the routing working as I want it.

Here's a google example of what currently doesn't work (unless I set it up completely wrong)

// Lists books in a shelf.
rpc ListBooks(ListBooksRequest) returns (ListBooksResponse) {
  // List method maps to HTTP GET.
  option (google.api.http) = {
    // The `parent` captures the parent resource name, such as "shelves/shelf1".
    get: "/v1/{parent=shelves/*}/books"
  };
}

When attempting to hit something like /v1/shelves/non-fiction/books the route is unrecognized.

I think this is because the asp.net routing template language doesn't really work like that?

Can you point me in the direction of what I would need to alter or change to get this different type of routing to work?

Was thinking right now I might have to make my own middleware router to get this to work, but thought maybe there's some way it could work in conjunction with the endpoints routing. Any pointers appreciated!

[JamesNK]

This style of route isn't supported: "/v1/{parent=shelves/*}/books"

You can use instead: "/v1/shelves/{parent}/books"

To be honest I don't really understand how the first syntax works and why you'd want to use it.

[KyleGobel]

I couldn't tell you the pros or cons on google's way vs asp.net way. Though in the example case above (/v1/shelves/non-fiction/books), parent would be bound to shelves/non-fiction, in the way you provided, it would be only non-fiction.

I thought since this was using the google HttpRule option, it would follow the syntax they laid out with it (https://cloud.google.com/endpoints/docs/grpc-service-config/reference/rpc/google.api#google.api.HttpRule)

Google as far as I can tell does use this in their public api's though e.g https://github.com/googleapis/googleapis/blob/master/google/firestore/v1/firestore.proto

I don't know if it's widely used outside of google, or if it's supported by the other transcoders out there, just thought I'd bring it to attention and see if you had any ideas on how to make it work.

[viktor-nikolaev]

Hi!
Thanks for your job!
We're interesting in grpc http api, as we're going to use grpc for our internal communications and rest api for external calls

[qkss]

Thanks for your job!
It's a very good idea

[oceanmist-github]

Thanks for all the good work! I would love to use this in my upcoming project where we are planning to implement Backend for Frontends that provide RESTful API curating access to the backend services to be implemented using gRPC.

[adchang]

+1 Thanks @JamesNK !

Great feature! Love to see this go GA! Quick and lightweight way to expose HTTP1.1 without involving envoy/gRPC-gateway.

2 feature requests:

1. Package maintains own annotations and http proto. I think most users of this library won't be calling protoc directly anyway.
2. Make route annotations in different file a la gRPC-gateway to keep proto file clean

[jsantanders]

Looking forward to this feature.

[ADustyOldMuffin]

I'll also toss in my hat and say I would personally love this feature. I've been using the experimental package and loving it.

[adityavaish]

I would be more than interested to have this out of alpha and to full release

[subclassed]

Very interested in this.

[JamesNK]

2. Make route annotations in different file a la gRPC-gateway to keep proto file clean

How does that work? Can you point me towards an example?

[ernestoalejo]

How does that work? Can you point me towards an example?

I think he is referring to this way of configuring transcoding through a YAML file:
https://cloud.google.com/endpoints/docs/grpc/transcoding#configuring_transcoding_in_yaml
https://cloud.google.com/endpoints/docs/grpc/grpc-service-config

On the other hand we have been using this library for a production project and the two main pain points for us (already mentioned before) were:

1. Having to copy manually the annotations.proto and http.proto files.
2. Changing the standard way of describing routes because the format was not supported. Routes like /v3/{name=events/*}:cancel[1] or /v1/{settings.name=users/*/settings}[2]

[1] https://cloud.google.com/apis/design/custom_methods#http_mapping
[2] https://cloud.google.com/apis/design/design_patterns#singleton_resources

[kollerdroid]

Swagger has been stopped working:

services.AddGrpcSwagger();  // throws:

Exception has occurred: CLR/System.TypeLoadException
An exception of type 'System.TypeLoadException' occurred in My.WebApi.dll but was not handled in user code: 'Could not load type 'Swashbuckle.AspNetCore.SwaggerGen.IDataContractResolver' from assembly 'Swashbuckle.AspNetCore.SwaggerGen, Version=5.6.3.0, Culture=neutral, PublicKeyToken=d84d99fb0135530a'.'
   at Microsoft.Extensions.DependencyInjection.GrpcSwaggerServiceExtensions.AddGrpcSwagger(IServiceCollection services)
   at My.WebApi.Startup.ConfigureServices(IServiceCollection services) in /home/myuser/proj/src/My.WebApi/Startup.cs:line xy
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()

Swashbuckle version is 5.6.3.
Is there any workaround to eliminate this?

[JamesNK]

Use Swashbuckle 5.3.1. They've made breaking changes in version 5.6.3 of their library.

[kharabasz]

Thank you for your work on this feature and I hope it becomes part of the official support for Grpc. Our use case is oriented around developer experience (api discovery, testing, debugging) - we have many services hosted under a handful of different frameworks and having Swagger as a standard starting point for each service is very important to us.

[mconradiesa]

We are in the process of redoing our internal webapi - it would be great to have this feature so services could make use of GRPC while our web projects can use the Web API without having to write everything twice.

[MarekPokornyOva]

@mconradiesa, maybe you could try to utilize https://github.com/MarekPokornyOva/MpSoft.AspNet.Grpc.MvcApi until official package gets released.