Gitea: Plugin system for obscure/non-universal features

[ethantkoenig]

I propose adding a plugin system to gitea, which would allow gitea users and developers to add new features while keeping the core gitea codebase small and maintainable. Please read my proposal, and let me know what you think.

Motivation

As gitea grows, there will be requests for more and more features. Many features will be relatively peripheral; while necessary/useful to some users, they will not be used by other users. An example of such a feature is #2211. I'm afraid that gitea will eventually become a bloated collection of incoherent features. Even if these features can be enabled/disabled in a config file, they still make the codebase more complicated and harder to maintain.

Meanwhile, many users of gitea may want to perform small-scale changes and customization to their personal forks. These could range from logo changes to small features that are for their specific use-case (and thus wouldn't make sense in upstream gitea). It would be nice if such users could make these changes without facing merge conflicts when they pull from upstream.

I believe plugins could solve both of these problems.

Specifics (I'm open to suggestions and discussion)

Plugins would be in-process (included during compilation). Plugins would interact with "core" gitea in the following ways:

• Plugins can update templates (this would happen during start-up). This allows plugins to add new UI elements
• Plugins can register new routes
• Plugins can install listeners/callbacks that are triggered when various events happened (e.g. new content is pushed to a repo, which could be useful to a plugin for full-text repo search)
• Plugins can add new tables to the database. Unfornuately, there doesn't seem to be a clean way for plugins to alter existing tables, since they corresponding struct in "core" gitea would need to be updated.

Unanswered Questions

1. What if two plugins rely on different version of the same dependency?

I don't have an answer right now, but hope to investigate possible solutions soon. The answer also depends on whether we end up switching to golang/dep (#2218)

[sapk]

We could use the plugin method introduce in Go1.8 : https://golang.org/pkg/plugin/
~Since they are compilated binary (that are loaded by main program), they can have there own deps like any programs.~

[sapk]

Futhermore, I propose :

• Plugins can add command line command (like: import github/bitbucket/gogs)

This would permit to add script from community wihout bloating main prog.

[lafriks]

@sapk Go plugins are supported only on Linux

[sapk]

@lafriks I have to correct what I have said on compiled plugin and vendored deps : https://github.com/golang/go/issues/20481

This should be fix but not yet ready.

[sapk]

Windows and macos plugin build are an objectif but not yet planned and no one on it. :disappointed:
I would still suggest it and explain that it is only possible for linux host in the wait of golang improvement.
Maybe there are better solutions ?

[andreynering]

Gogs cross-reference: https://github.com/gogits/gogs/issues/2438

First, I completely agree the we should not bloat Gitea with more and more small features.

About the plugin system, it's also hard to implement. Go has interpreters for other languages we could look:

• JavaScript: https://github.com/robertkrimen/otto
• Lua: https://github.com/yuin/gopher-lua
• Lua: https://github.com/Shopify/go-lua
• Bash: https://github.com/mvdan/sh (this one may not be suitable for the plugin system, but for allowing running bash commands that will run on Windows, too)

We should look for an approach that is either easy to maintain in the Gitea core and also for plugin developers.

[sapk]

https://github.com/hashicorp/go-plugin suggested on gogs issue is also a good solution. It support some few good feature like negotiation for compatibility between main and plugins (and is planning "plugin fetching"). And plugin could be develop in various languages just need gRPC interface.

[sapk]

gRPC permit support for plugin triggerring event and not wait to be called.

[jonasfranz]

I think that we only need a better filtering of features and we have to know what we want to implement and what not. But I disagree with you statement that #2211 is only used be a "small minority of users" because it makes gitea interesting for small businesses that need such features. And my apprehension is that the features will not be such good integrated or have a good quality if they come via plugin.
I agree that there should be a simple possibility to add small additions like #2221 but I do not think that it would be a good option for (in relation) big features like #2211.

[lunny]

I agree with @JonasFranzDEV enhancement of issues will let small company remove their redmine and such others.

[ethantkoenig]

@JonasFranzDEV @lunny Apologies if I came across as unfairly criticizing the timetracking feature. I didn't mean to say that the feature is useless, but only that it is peripheral to the main purpose of gitea, and that large subset of gitea users do not need it. I've updated the wording in my description accordingly.

I'm not sure "we just need to do a better job deciding what to implement" is a viable solution. Who decides what is worth implementing? What if user X needs feature Y, but it is decided that feature Y will not be implemented? User X's only option is to implement feature Y on his/her own fork, which is problematic as I described in the description.

Finally, I disagree with the "too big for a plugin" argument. The fact that the timetracking feature is large means it will add even more overhead and complexity to the codebase, which I believe further underscores the need for moving it to a plugin.

[lafriks]

@ethantkoenig plugins are good thing to have but also come with additional problems like it is harder to change core functionality/design/structure as such changes can and most probably will break plugins and is harder to test

[sapk]

@lafriks It also give us room for change because with too much function integrate in core we lock the structure of code and giving a standard interface (with versioning) for plugin will allow function to evolve at there own rythm. If a user want to keep a func implemented by a plugin that does not support the last version of core, it become the choice of the user to stay with an older version in the wait of an update of the plugin (or contribute to the update).

I think we all agree to had plugin, we just need to decide what type and how to limit bruden for gitea and plugins developer by starting a roadmap and defining interface and what a plugin can do.

[lafriks]

I agree that there could be plugins for webhooks/ci integration but for extending core functionality and UI it will be too much work to implement that correctly, especially to support multiple platforms etc. Also if plugins starts to be created that adds UI functionality we will start to get bug reports that will be hard to track down if bugs are only reproducible with plugin or combination of plugins enabled.

I think that we should continue to keep Gitea easy to install, run and packed everything is single binary. To make plugins right it would mean to rewrite gitea core to allow that and most probably it would be quite big task with no direct benefit.

[tboerger]

I just want to mention that we already had https://github.com/go-gitea/gitea/issues/661 ;)

[tboerger]

Now my two cents for this topic. I wish we had an grpc based plugin system that is comparable to packer by hashicorp. Of course it's a lot of work, but we can even simplify the core and extract more parts into plugins. To avoid problems with plugin versions we have to apply versioned apis.

When we built the basics we can even offer some kind of marketplace where everybody can offer his stuff.

[jonasfranz]

@tboerger Will the plugins be compatible to newer / older "versioned apis" if gitea updates and the plugin not?
Which parts of the core should be extracted into plugins?

[tboerger]

@tboerger Will the plugins be compatible to newer / older "versioned apis" if gitea updates and the plugin not?

That's the whole thing of versioned APIs, older plugins should work with newer core versions, they just can not use newer core API parts.

Which parts of the core should be extracted into plugins?

No fixed list, maybe wikis, timetracking or other future additions like pages and so on. Keeping the core minimalistic and extracting many things into plugins should bring in some additional overhead, but every feature/plugin can be developed and published independent of the core as long as the core already provides the API for it.

[jonasfranz]

I think that a concept like drone's plugin system might be a good idea. But we need a better API since we do not only provide "build steps". So my idea is that every plugin is started as docker container and connected to the plugin API via gRPC like @sapk pointed out (https://github.com/go-gitea/gitea/issues/2222#issuecomment-318653276). There is no installation process for the admin like JDK installation or other dependencies. But perhaps is this concept a little bit to complex.

[daviian]

Since we don't depend on Go 1.7 anymore, at least I thought so we could use the introducted go plugin architecture from 1.8.

A blogpost as an example: https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9

[lafriks]

@daviian we can not use built-in golang plugin architecture as it is not supported on Windows and other gitea build targers

[lunny]

@daviian we should declare v1.3 only support go1.8+ on README or something place.

[tboerger]

I would use some plugin system like packer, that's also grpc based and works also on Windows.

[sapk]

I could try to make a test with https://github.com/hashicorp/go-plugin. Any idea of a plugin that could be implemented ?
I was thinking of adding a file interpreter like markdown.

For the plugin architecture, I was thinking of allowing plugin to declare file handling based on regex/ file extension, declare url to be able to responds, declare event to listen for, be able to override some defined function.

[lavvy]

@sapk There are a ton of plugins here http://gitbucket-plugins.github.io/

[techknowlogick]

Another plugin system we could consider is the one that Caddy uses: https://github.com/mholt/caddy/wiki/Extending-Caddy

For the webserver for the docs site it is already using Caddy and its plugin system: https://github.com/go-gitea/gitea/blob/master/docs/Dockerfile#L8

[tboerger]

A compile time plugin system is not something I really like

[stevegt]

Trying to summarize this thread so far -- let me know what I'm missing. I see two viable alternatives for a plugin system:

• A single-process internal plugin system similar to Caddy's, where everything runs in one process and each module registers its presence at startup. We could optionally refactor some or all of existing Gitea to make "everything a plugin". In addition to enabling unforseen features while keeping core code small, it enables less risk in refactoring major subsystems, since it encourages the use of internal API between modules.
• A microservices architecture, where plugins each run in their own process and use the swagger API to interact with gitea and its db, which keeps gitea monolithic and its bugs relatively well-defined. This also means that, for instance, any plugin would have to maintain its own db for any custom tables or columns, and each plugin might best be run in its own container. We'd provide HTTP callback hooks so gitea can trigger plugins to do their jobs, and plugins would have to have some way of modifying Gitea's HTML.

Variations on these two themes might include runtime versus compile-time linking of the single-process system, or some RPC protocol other than HTTP for the multi-process system -- I'd describe a POSIX IPC plugin system as a variation of the second option, for instance. It would also be possible, though probably not desirable, to break existing Gitea up into microservices as well -- just tossing that out there as a strawman so it can get knocked down for performance reasons. ;-)

But in general, do these two pretty much cover all of the alternatives discussed so far? (I can't imagine any other category being computationally possible, but am willing to be surprised.) ;-)

If nobody can think of anything else, then we may want to split these two alternatives into two different issues, and see where they lead. Right now it's hard to tell which way the consensus is going, I think because we're actually talking about two completely different definitions of what a "plugin" even is.

Thoughts?

[jonasfranz]

I've tested the built-in plugin system of golang and discovered that it might fit very well in our use case. Plugins will be using the already existing database functions in models without having a seperate plugin api interface.

Plugins are going to be ~39MiB big since they include the gitea sources. This is a problem in my mind.
Any ideas on this?

[tboerger]

The builtin plugin system doesn't support windows, or has that changed?

[Fardinak]

Plugin support has indeed improved since 1.8, but Windows is still unsupported at this time and it doesn't seem like there's any active effort towards making it happen. However the community seems keen on accepting a pull request from someone who knows actually how to implement it…

Plugin support in v1.13

Currently plugins are only supported on Linux and macOS.

Plugin support in master

Currently plugins are only supported on Linux, FreeBSD, and macOS.

And for reference, here's the issue regarding Windows support: golang/go#19282

Personally I'm mostly in favor of golang plugins or some sort of scripting. RPC just seems like too much trouble to manage. But in any case, I'm looking forward to plugin support.

[zeripath]

Oops, didn't mean to close this.

One thing I think would help would be a more extensible markdown renderer. I understand Hugo has moved to the goldmark renderer.

[tboerger]

I still think it would make sense to implement the plugin api based on grpc and use the hashicorp plugin lib, this is already heavily battle tested.

[lunny]

https://github.com/hashicorp/go-plugin allow only an executable binary plugin and running on the local network. So the plugin have be run on a sandbox(i.e. docker) to avoid it read/write filesystem/database directly.

[sapk]

@lunny I also think that the grpc solution is good because it allow local and remote. If plugin dev choose to acces database and file directly when run locally it is their concern. Maybe later we could choose to isolate (chroot, container, ...) local plugin but I think it would be extra first since it would be platform dependant.

[lunny]

@sapk I don't think that's their concern. I think the security should be considered when we design the plugin system.
Once you choose to enable plugin system, gitea could require you installed docker or k8s. All plugins should declare their permissions(read disk/visit network and other etc.)

[tboerger]

I don't get your concerns... That doesn't make sense to me.

[lunny]

@tboerger Plugins will run on your gitea server machine, it can do anything that's what my concerns. Could you explain why did you think that's not a problem?

[tboerger]

They can do whatever they implement... That's the use case for plugins 😂

[zeripath]

Depending on the level of plugin integration, there is a slight issue to do with plugins being run whilst there is an ongoing transaction with the db - but we'd just have to be explicit about when this happens. I would argue that level of integration would be too much to begin with.

If a server owner wants to blow themselves up by having a plugin that erases their entire db that's their call. We should however, try to proffer reasonable interfaces to the plugins to allow them to behave in a safe manner - so for example, offer sensible sanitisation services, perhaps offer some kind of limited js framework or even just best practices, or make it explicit which plugins extra js on the server - I dunno - I think the idea needs some concrete fleshing out to see what would need extra thought.

One thing that would make very tight plugin integration possible would be to move to a (configurable) dependency injection framework (There would be numerous benefits from this - including making it easier to test gitea properly). Go Wire is preferred by the go developers but I can't see how to make that make sense in a very configurable / plugin dominated system - something like a java Spring framework configurable file makes for very configurable system albeit with the costs of domain specific language. Again I dunno - just thoughts.

[tboerger]

Dependency injection doesn't exist in go... How to use wire you can see within drone.

Define some hook endpoints via interfaces, there drone could also give some advice. Start with first parts that can be extended and increase the hooks step by step

[Fardinak]

I think @lunny's concerns are valid, but that problem also exists with Go plugins or most other methods. I don't find requiring docker to be a viable option for a user that needs only a couple of plugins with their instance either. Also since OS support is clearly important, we need to remember that Docker still requires running a resource intensive VM on macOS.

Running plugins in a safe sandbox would be ideal. I consider WASI to be the ultimate solution, but unfortunately it just doesn't seem to be there yet. The closest option would be one of the interpreters mentioned by @andreynering:

About the plugin system, it's also hard to implement. Go has interpreters for other languages we could look:

* JavaScript: https://github.com/robertkrimen/otto
* Lua: https://github.com/yuin/gopher-lua
* Lua: https://github.com/Shopify/go-lua
* Bash: https://github.com/mvdan/sh (this one may not be suitable for the plugin system, but for allowing running bash commands that will run on Windows, too)

I must say, otto looks kinda promising.

Otherwise, it seems that GRPC is the most solid solution currently available. Paired with supervisord and it's RPC API (or maybe a golang implementation of it), they should provide a robust plugin solution for Gitea.

[andreynering]

Yaegi (a Go interpreter written in Go) is also a thing nowadays.

[Fardinak]

Oh that looks really interesting. How feasible would it be to create a limited subset of the stdlib? Although I don't know how that would affect the rest of the libraries, it could for example help disable the plugins from accessing the filesystem, or limit them to only the interface provided by Gitea.

[tboerger]

With all the concerns you should continue with the mono binary. I will unsubscribe from this issue because I currently don't believe this will lead to something useful.

[Fardinak]

Well it seems like the GRPC option makes sense to most. Let me know if I can be of help in any way. We're currently using Go+GRPC on a company project so I have some experience with it.