Ent: proposal: Protobuf + gRPC <> Ent interop

Created on 28 Feb 2021 · 4Comments · Source: ent/ent

Allow for interoperability between ent and protobuf/gRPC:

Generation of .proto files from an ent schema
Generation of a gRPC service definition for a basic CRUD server using the generated protobuf messages
Generate a go implementation of the gRPC service
Generation of ent schema from .proto files

[X] I have searched the issues of this repository and believe that this is not a duplicate.

Status

This is a WIP. I've started working on the implementation of .proto file generation and looking for feedback. Will update the design on the other parts soon.

Summary 💡

Implement a new cli tool entproto under ent/contrib that takes an ent schema directory as an input and outputs .proto files. In a new package entproto add new annotations to provide necessary mappings between ent and proto (defining stable field numbers, for example).
Add to entproto the capability to output a gRPC service definition with basic Add, Delete, Get, Put, and List methods.
Create a tool (TBD whether code gen or a runtime library) to generate a golang gRPC
Implement a protoc plugin protoc-gen-ent (example) that outputs .go files with ent.Schemas representing the input proto files.

Motivation 🔦

Many organizations use protobuf as their IDL. gRPC is becoming increasingly popular as an RPC transport due to its efficiency and code-generation capabilities. Data access in the protobuf/gRPC world is currently the wild west. Allowing programmatic interoperability between ent and proto/gRPC would open up many possibilities for both ecosystems.

Design

.proto file generation

By default, entproto will skip all schemas, unless they explicitly opt-in for proto file generation:

type User struct {
    ent.Schema
}

func (User) Annotations() []schema.Annotation {
    return []schema.Annotation{entproto.Message()}
}

By default the proto package name for the generated files will be entpb but it can be specified using a functional option:

func (MessageWithPackageName) Annotations() []schema.Annotation {
    return []schema.Annotation{entproto.Message(
        entproto.PackageName("io.entgo.apps.todo"),
    )}
}

Field Annotations

All fields must be annotated with entproto.Field to specify their proto field numbers

// Fields of the User.
func (User) Fields() []ent.Field {
    return []ent.Field{
        field.String("name").Annotations(entproto.Field(2)),
    }
}

The ID field is added to the generated message as well, in the above example it is implicitly defined, but entproto will respect explicitly defined ID fields as well. The above schema would translate to:

message User {
  int32 id = 1;
  string name = 2
}

Field type mappings:

Validations:

Field number 1 is reserved for the ID field
No duplication of field numbers (this is illegal protobuf)
Only supported ent field types are used

Edges

Edges are annotated in the same way as fields: using entproto.Field annotation to specify the field number for the generated field. Unique relations are mapped to normal fields, non-unique relations are mapped to repeated fields.
For example:

func (BlogPost) Edges() []ent.Edge {
    return []ent.Edge{
        edge.To("author", User.Type).Unique().Annotations(entproto.Field(4)),
        edge.From("categories", Category.Type).Ref("blog_posts").Annotations(entproto.Field(5)),
    }
}

func (BlogPost) Fields() []ent.Field {
    return []ent.Field{
        field.String("title").Annotations(entproto.Field(2)),
        field.String("body").Annotations(entproto.Field(3)),
    }
}

Is transformed to:

message BlogPost {
  int32 id = 1;
  string title = 2;
  string body = 3;
  User author = 4;
  repeated Category categories = 5;
}

Validation:
Cyclic dependencies are not supported in protobuf - so back references can only be supported if both messages are output to the same proto package. (In the above example, BlogPost, User and Category must be output to the same proto package).

gRPC service definition

To facilitate simple bootstrapping of application servers from the proto schema, entproto should support generation of a CRUD gRPC service definition from each ent.Schema. To request the generation of a gRPC service definition, a developer would annotate the schema with:

func (BlogPost) Annotations() []schema.Annotation {
    return []schema.Annotation{
        entproto.Message(
            entproto.GenerateService(),
        ),
    }
}

The proto generation creates a service named BlogPostService with 4 methods:

service BlogPostService {
  rpc Create ( CreateBlogPostRequest ) returns ( BlogPost );

  rpc Get ( GetBlogPostRequest ) returns ( BlogPost );

  rpc Update ( UpdateBlogPostRequest ) returns ( BlogPost );

  rpc Remove ( RemoveBlogPostRequest ) returns ( google.protobuf.Empty );
}

and generates the supporting input/output message types:

message CreateBlogPostRequest {
  BlogPost blog_post = 1;
}

message GetBlogPostRequest {
  int32 id = 1;
}

message UpdateBlogPostRequest {
  BlogPost blog_post = 1;
}

message DeleteBlogPostRequest {
  int32 id = 1;
}

The type of the id field for Get and Delete requests is determined from the gen.Type.ID field type.