Mattermost-server: Plugin framework: add gRPC support

[jwilander]

If you're interested please comment here and come join our "Contributors" community channel on our daily build server, where you can discuss questions with community members and the Mattermost core team. For technical advice or questions, please join our "Developers" community channel.

New contributors please see our Developer's Guide.

---

__This ticket has multiple parts and is a large undertaking. A fairly advanced understanding of the plugins system is recommended, as is familiarity with both protocol buffers and gRPC.__

Adding gRPC as an alternative to net/rpc for facilitating communication between the Mattermost server and plugins would make it feasible to write the server-side of plugins in any programming language that supports gRPC (which is most major languages).

The following tickets describe the work needed to add gRPC support:

1. Define protocol buffer services and messages
2. Create a tool to generate protocol buffer files
3. Add Makefile target to generate protocol buffer Go code
4. Generate glue code and hook up gRPC to the plugin system
5. Add protocol buffer and glue code generation to make pluginapi
6. Create test gRPC plugin in Go
7. Create test gRPC plugin in a non-Go language

If you have any questions or would like feedback, feel free to ask in the Developer Toolkit channel on our Mattermost community instance.

[cored]

@jwilander Can I try to tackle this one?

[jasonblais]

@cored Certainly! Let us know if you have any questions 👍

[sandeepsukhani]

Hi,

I am trying to understand requirements for this, might take this up.

Can someone please explain this from MM-11597:
Must maintain backwards compatibility, possibly by reading the current .proto files
Does this mean backwards compatible by preserving old versions of APIs/Hooks with the latest modified version or preserving old APIs/Hooks with newly added APIs?

[jwilander]

Yes, so essentially it needs to preserve old APIs/Hooks with newly added APIs. If I add a new API for GetSomeThing() the tool should generate all the existing APIs/Hooks in a non-breaking way so that if a plugin is using an older version of the protocol buffer the old APIs still work with a newer MM version (they just don't get the new GetSomeThing() until they upgrade their plugin)

If we modify an existing API then it's OK to break

That make sense?

[cored]

@jwilander wouldn't that imply that who needs to regenerate the protobuf stub would be the client?

[jwilander]

Not sure if I follow, both the plugin and the MM server will need the protocol buffer definitions. The plugin for whatever language it's using will need to generate the code using the protocol buffer files but I'm assuming that's deterministic, so as long as the protocol buffers our tool generates maintain backwards compatibility.

And I hesitate away from using "client" and "server" here since the plugin will be both. A gRPC client for the API but also a gRPC server for the Hooks

[jwilander]

@cored have you had a chance to work on this at all? Let me know if I should open it up for others to work on

[cored]

@jwilander Sorry haven't started on this just yet. I was planning to outline an strategy today to divide what I need to work on and start on that for the weekend.

[jwilander]

Awesome, sounds good. Let us know if you have any questions

[cored]

@jwilander Sure thing. Probably will join the channel to discuss further.

[cored]

@jwilander Hi, I'm so sorry, I didn't make any progress in all this time about this ticket. I've been too busy, I mean I can take a tackle at it again but at least in two more weeks. Don't know if you want to reassign it. Sorry for the delay on giving an update I thought I could've make it.

[jwilander]

No worries @cored ! I can throw the "Up for Grabs" label back on it but feel free to take another stab at it if you get some time

[cored]

Thanks, will do that.

On Thu, 29 Nov 2018 at 19:12, Joram Wilander [e-mail]
wrote:

No worries @cored https://github.com/cored ! I can throw the "Up for
Grabs" label back on it but feel free to take another stab at it if you get
some time

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-server/issues/9486#issuecomment-443041992,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAtx9B2atkSs2irRaIuDPdlwjebHuf1ks5u0HfsgaJpZM4W9mkD
.

--
Rafael George

[dv29]

I'd like to take a stab at this

[jwilander]

Awesome @dv29! If you have questions or want to discuss anything feel free to ask in the Developer Toolkit channel

[dv29]

This will take a little longe then I expected, just so everyone is aware of it, have started with creating the .proto files and working on the tool to auto generate them, will create a WIP asap.

[dv29]

Hi everyone, Just to let you know I am still working on this. Will take a little longer as somethings have come up, I'll try to finish it in couple of weeks

[jasonblais]

Sounds good, thanks for the follow-up @dv29!

[dv29]

Hi everyone. Keeping everyone posted about the progress, last couple of weeks have been hectic and couldn't get much done however next will be a little relaxed and I'll try to get the WIP PR ready

[hanzei]

Hey @dv29,

Just want to check out how your work is going? Do you have any questions? Looking forward to your PR!

[dv29]

@hanzei sorry I was held up with some house work. I'm still working on it, no questions so far

[crspeller]

Might still accept a well done PR for this that keeps things simple, but closing because of lack of internal interest in the feature.

[dv29]

I'll keep working at it and create a Pr when finished :( sorry for the delay

[jasonblais]

Thanks @dv29! I'll keep this help wanted issue open while you're working on it. How's the progress been, any questions so far?

[jasonblais]

Hey @dv29 - just touching base on this particular help wanted ticket.

We've done some exploration on our end and realize supporting gRPC will be more work than originally anticipated, requiring additional long-term maintenance and support. Depending on the approach, supporting gRPC may in fact be fairly complex and require an additional RPC layer to the existing plugin framework.

I'd be interested to hear how far you've gotten with the work, and what approach you've taken. Given the complexity, I would hate for you to do all the work, submit it for review, only to not be able to get it merged.

Let me know if you have any questions.

Tagging @lieut-data as well from the Toolkit team.

[dv29]

@jasonblais It is fairly complex and I am fairly new to it. I am trying to get a simple gRPC service running. I have tried some trial and error and I have really not figured out an approach that would scale properly TBH.

The amount of work I have done is minimal as I am still struggling to find time.
To that point if anyone else would like to take a dig at this, I'd like to help them out.

[jasonblais]

Thanks @dv29, sounds fair. Really appreciate your help here, agree this is fairly complex. We can pause on the work for now and revisit at a later time.

I'm curious if the plugin framework is of interest to you? There may be other smaller-sized tickets that aren't as time-consuming. Let me know :)