Mattermost-server: Plugin: Auto-update plugins from marketplace

[lieut-data]

The Plugin Marketplace launched in v5.16, initially featuring only Mattermost plugins, but with plans to eventually include support for community plugins. While on-demand updating of plugins is scheduled for v5.18, there are no immediate plans to add support for auto-upgrading plugins in the marketplace directly into the product.

Of course, auto-updating plugins would still be a useful feature! It's applicable to development and testing environments, and any customer looking to stay "on the cutting edge".

Design a mattermost-plugin-marketplace-addon plugin that periodically queries the configured Marketplace URL (PluginSettings.MarketplaceUrl) for newer plugins and installs newer versions of plugins already installed. The plugin should include a bot (plugin-marketplace) that notifies all system administrators via a direct message when a plugin is successfully updated or fails to update. After review, expect this plugin to be promoted to the mattermost organization.

The choice of plugin name here is intentional, as this plugin could provide a testbed for prototyping new marketplace functionality before it's officially integrated into the core product.

Part of completing this effort will include:

• exposing the Marketplace API for programmatic consumption by the plugin (see https://github.com/mattermost/mattermost-marketplace/pull/6 for a closed WIP PR that will need to be completed)
• depending on https://mattermost.atlassian.net/browse/MM-16700

Be prepared to steward the above changes, along with tackling the necessary architecture for the plugin.

---

If you're interested please comment here and come join our "Contributors" community channel on our daily build server, where you can discuss questions with community members and the Mattermost core team. For technical advice or questions, please join our "Developers" community channel.

New contributors please see our Developer's Guide.

JIRA: https://mattermost.atlassian.net/browse/MM-16700

[ilgooz]

I'm very excited to take this one but probably won't be able to start immediately, I might get very busy next week to handle non-Mattermost related stuff. I should be able to start in 1 or 2 weeks, if it's OK can you assign it to me?

[RajatVaryani]

Even I am interested in this ticket. I presume the work here is big enough to be done by 2 people. If @ilgooz is assigned and if he's comfortable working with me then I would love to work in it as well.

[lieut-data]

@ilgooz & @RajatVaryani, you're both very welcome to spearhead a group effort. I'm happy to schedule half an hour to chat with you both about the vision here and help answer any questions. I'll tentatively assign you both? Let me know what you think :)

[ilgooz]

@lieut-data lets schedule a call for this Monday? Any other day also can work with me.

@RajatVaryani I'm happy to collaborate! Can you also share your availability please?

[RajatVaryani]

Sure. I am available from 7am PST to 10am PST and 7pm to 9pm PST.

[ilgooz]

@lieut-data we had a quick chat with @RajatVaryani about how to collaborate. We split the work in two parts. I'll be taking the part about checking versions and updating the plugins and Rajat will handle the notification part.

We're thinking about creating two packages as updater and notifier -think them like DAO layer- and gluing them together in the main package with the MattermostPlugin API. These two packages should accept plugin.API interface to interact with Mattermost.

We may also create a 3rd package (app/service) to glue updater and notifier together instead of doing this in main and only use this package in main. This way separation of logic would be cleaner but we would like to take the call with you before giving any decision about final design of the plugin.

But the overall plugin's structure should be similar to this one: https://github.com/ilgooz/mattermost-plugin-topdf (see package main only uses package topdf and package topdf glues other smaller packages together.)

We'll be working in this repo: https://github.com/ilgooz/mattermost-plugin-marketplace-addon

--

We'll be using new InstallPluginFromUrl() API, tho we may not need to use a marketplace client since Plugin's API already provides all necessary functions(haven't check deep but it seems ok) like GetPlugins().

A side note about InstallPluginFromUrl() to discuss it in our call, I think using existent InstallPlugin() API will give us more flexibility considering if we need to download plugins from an authenticated source like HTTP's Basic Auth or even a Git host which might be secured with an ssh key.

We can have all of this logic in this plugin in a separated package and once everything is stable, we can move it to built-in Plugin Helper API if needed.

--

btw, I'm in Istanbul timezone but I can be flexible on that

[lieut-data]

@ilgooz & @RajatVaryani, your description of the proposed code layout makes a lot of sense to me: nicely written up! A few things that might be good to pin down:

• what configuration options does the plugin provide? (thinking something to enable/disable auto-update, a frequency for which to check for updates, and perhaps notification options, including posting changes to an arbitrary channel)
• how does plugin signing play into all of this

I'm going to be out of the office tomorrow and Tuesday, and partly out of the office the rest of the week. If it still makes sense to meet later this week, I'd be happy to meet and help.

[ilgooz]

configuration options

I really like your suggestions of these 3 config/feature. we should have them:

• enable/disable auto-update
• frequency to check for updates
• notification options, including posting changes to an arbitrary channel

--

We may add more to above but seems enough to me right now but I have one more feature idea about having an interactive bot to take credentials.

Some plugins might not be available to public and secured with:

• known HTTP authentication methods (Barer token, basic auth, digest auth).
• custom HTTP authentication methods with custom headers, query string or request body.
• private Git releases protected with SSH keys.

If one or more plugins protected like this and if we expect that different private plugins might be protected with different methods we may have to support plugin basis configurations for users to set their configs.

To make this configurability possible we can create a bot that asks for credentials of each privately hosted plugin via an arbitrary channel (channel determined by the user in plugin configs). User can put an SSH key, set HTTP auth tokens or define custom headers there depending on the authentication method followed for that individual plugin.

Bot should ask credentials of each private plugin in separate interactive posts.

To determine if a plugin private or public, auto-update plugin can try to download it first like a public plugin and if we get an authentication error from the server, we can ask for credentials via this bot and try to download again with credentials.

We should save credentials for every plugin to use in later updates as long as credentials are not invalidated by the server otherwise we'll ask for credentials again.

We should also store this credentials in one big JSON object and put it to auto-update plugin's configuration page so admins can make a backup for their credentials or manually edit.

Backups can be provided by the bot too. It can just create a downloadable JSON file and post to channel when requested.

We can also introduce another config to auto-update plugin to download this JSON object from a private remote server in case of admin wants manage creds by its own and not use the credentials bot for that.

Supporting private plugins is a really cool feature but I think we don't need it in the initial version of auto-update plugin, let's have this idea growing and implement it once it's mature enough. I also would like to get more feedbacks about the way of collecting credentials and everything.

synchronization in HA mode

In High Availability mode, auto-update plugin will have multiple instances running. But for now, there is no any built-in synchronization primitives made available to Plugin API to avoid race conditions between instances.

For example, we would like to avoid updating the same plugin for multiple times in a row, sending duplicated notifications to admins or asking credentials for the same plugin multiple times.

In order to avoid these, and even load balance the work between instances which might be needed if a server has 100s of plugins, we need to have synchronization primitives and an MQ available to Plugin API.

A note: Load balancing here cannot be handled by Mattermost server because we're not using for example a regular hook that might triggered from any instance of plugin by the server. Here, plugin instance itself is making the polling internally and has to load balance the work by itself to other instances.

To make it more obvious what I'm talking about, let me give you a real world workflow:

Let's say that we have a distributed lock and an MQ API available to plugins via Plugin Helper API and we have 3 instances of auto-update plugin running.

Let's say that auto-update plugin tries to update plugins in every 5 minutes by doing pooling. In order to avoid multiple instances checking for new versions:

• Try to obtain a lock (will result with 1 instance obtaining and other 2 waiting),
• When obtained by an instance, wait for 5mins,
• Do the checking for all installed plugins.
• Create and send update messages to MQ for each plugin that needs to be updated.
• Release the lock.
• Repeat.

This way, only 1 instance will be able to obtain a lock and start checking and others will be sleeping. After check is completed, code will release the lock and this life-cycle will continue.

After we know which plugins should be updated by the above workflow:

• An update request will be created and sent to MQ's _update_ channel in the above workflow.
• This cannel is listened by all instances when they are first started. _update_ channel load balances every messages with round-robin algorithm. Once an instance receives an update request, it updates the plugin and sends necessary notifications to admin. If plugin is private, it'll ask for credentials instead, via the bot. After creds are taken, any instance might update the plugin because this time update request for that plugin will be triggered by bot hook (I don't know the bot API yet but probably by _MessageHasBeenUpdated()_) and this one load balanced by the Mattermost server.

--

Since we don't have a synchronization mechanism and an MQ for now, we still need to figure out how to avoid duplications talked above that possibly will occur in the HA mode.

We can implement a very basic distributed lock inside the auto-update plugin by using _KVCompareAndSet()_ & _KVSetWithExpiry()_ to avoid multiple instances polling and updating the plugins.

We just want 1 instance to do all the work and others to sleep forever.

We also don't care about load balancing the code related to actually updating the plugins between instances at this point.

What do you think about these?

--

@lieut-data btw, I'm not aware of what is "plugin signing", just heard of it. Can you please send me a link to read about?

[RajatVaryani]

@ilgooz You can please go through for plugin signing https://docs.google.com/document/d/1qABE7VEx4k_ZAeh6Ydn4pGbu6BQfZt65x68i2s65MOQ/edit#

[lieut-data]

Supporting private plugins is a really cool feature but I think we don't need it in the initial version of auto-update plugin, let's have this idea growing and implement it once it's mature enough.

Agree that it's not something we should support out of the gate. I'm also not sure it would belong in the "plugin-marketplace-addons", per se, since accessing a private repository to initiate a download would have to be a core feature of the marketplace itself (otherwise the "Download" button wouldn't work in the product), and at that point, the server would have to manage the credentials.

But for now, there is no any built-in synchronization primitives made available to Plugin API to avoid race conditions between instances.
We can implement a very basic distributed lock inside the auto-update plugin by using KVCompareAndSet() & KVSetWithExpiry() to avoid multiple instances polling and updating the plugins.

I actually think the idea of a truly distributed lock is overkill. Distributed locks solve two problems in my mind:
1) Coordinating events between asynchronous actors who lack a central data store facilitating same.
2) Potentially boosting performance of highly concurrent, asynchronous actors by avoiding a global lock.

The first doesn't apply, since all Mattermost servers have just such a central store in the form of the database. And as you note above, with KVCompareAndSet and KVSetWithExpiry -- and as of https://github.com/mattermost/mattermost-server/pull/11818, a unified KVSetWithOptions -- we can coordinate events among actors. We're even formalizing this model of "run on a single node" in the form of another helper as of https://github.com/mattermost/mattermost-server/pull/12969.

The second also doesn't apply in this case, since we won't be checking the marketplace for updates very often, and it's fine for the plugin instance on a particular server to hold the lock for an extended duration while it does so.

That being said, I do agree that distributed locks are a neat solution to real technical problems, and would love to work on an effort where they were required.

[ilgooz]

Moving the discussion related to locks to a chat thread: https://community.mattermost.com/core/pl/bb376sjsdbym8kj7nz7zrcos7r

[ilgooz]

Agree that it's not something we should support out of the gate. I'm also not sure it would belong in the "plugin-marketplace-addons", per se, since accessing a private repository to initiate a download would have to be a core feature of the marketplace itself (otherwise the "Download" button wouldn't work in the product), and at that point, the server would have to manage the credentials.

Thanks Jesse. If the direction is to keep plugin related logic at the core, it makes sense to have credentials related logic there too. I thought, you may want to remove more logic out from core in future as possible and recreate them as plugins. Including the actions that happens when clicking to the "Download" button.

But this is certainly something that can be done in the long term and requires lots of thinking before doing. I have a close interest and past work about Microservices and extendible APIs and breaking monolithic apps to parts like this always excites me.

Going back to the topic, I'm giving up about the idea related to supporting private plugins for now and also, probably going to use this new _RunOnSingleNode()_ API for synchronization.

As a side note: we can introduce a new built-in UpdatePlugin(id string) (updated bool, err error) API to plugins which should also handle credentials. Not having this today doesn't block our work on the plugin but just an idea to consider for future.

I read the proposal about plugin signing. Looks pretty straightforward. I think we can just use this package for verification as pointed out in the proposal. I think there are no worries about this one.

[lieut-data]

@ilgooz, sounds good!

With regards to plugin signing, my expectation is that the server is still doing the checking -- i.e. using the built-in logic being added for v5.18 -- but we just need to coordinate on how to provide the server with both the plugin bundle and the plugin signature. I /think/ the APIs right now aren't setup for that, but I stand to be corrected. @ali-farooq0 / @iomodo, can you provide direction here?

[ilgooz]

This bug https://github.com/mattermost/mattermost-server/issues/13040 is blocking us. We're not able to implement this plugin without solving the bug first or we need to think about some workarounds.

_edit: the bug is fixed and should be merged to master soon._

[ilgooz]

We have a working prototype of _marketplace-addon_ now at: https://github.com/ilgooz/mattermost-plugin-marketplace-addon/pull/3

I’ve created the structure, implemented the main, updater and other dependency packages and made necessary connections between the logic. Also, initialized the notifier package with its public facing APIs for @RajatVaryani you to do the actual implementation.

Plugin is now working but we need latest versions of _mattermost-server_ and _mattermost-dlock_ pushed to the remote with the latest changes for testers to test the plugin.

I have the changes on my local and created a PR to _mattermost-server_ to reflect the changes there.

I'll be updating _mattermost-dlock_ repo too. _-edit updated._

Initial version of the auto-update feature is ready and working and @RajatVaryani will be implementing the notifier package to send update notifications to admins and user configured arbitrary channel.

Initial version has no plugin signature verification feature but it'll be implemented later when the actual plugin signing feature is finalized in the Mattermost Server's side.

Features needed in the notifier package (/cc @lieut-data, @RajatVaryani):

1- Notify admins on every update notification. Use notifications channel to receive notifications.
2- Send a copy of notifications to the optional user defined channel. (plugins.json, channel-name)
3- Notification channel is automatically closed when updater is stopped also means when plugin is deactivated. notifier is reinitialized with notifier.New() once plugin is reactivated.

Please see the Notification type to consume in the notifier.

@lieut-data, can you please share your further ideas about what kind of notifications we should send to admins and to the notification channel?

[ilgooz]

Anyone, since I have no time complete the remaining work, if you're interested working on this plugin, please use the initiative that we have started to complete the rest. In order to add your changes, please fork the following repo, add your changes on top and kindly request a review from @lieut-data.

Repo: https://github.com/ilgooz/mattermost-plugin-marketplace-addon

Plugin is ready and working. But notifier part has to be implemented. Except the notifier, you can start with solving remaining change requests at: https://github.com/ilgooz/mattermost-plugin-marketplace-addon/pull/3

@RajatVaryani Please free feel to fork and do your needed changes.

@mattermost please feel free clone and push the repo under @mattermost org if you'd like to further maintain this plugin.

[lieut-data]

Thanks again, @ilgooz :)

[lieut-data]

Going to close this ticket at present given a shift to other priorities.