Pocketmine-mp: Composer-based plugin manager framework ("zygote")

[SOF3]

Motivation

The current plugin loading system has lots of problems:

• Difficult to include libraries
• Non-standard NIH class loading system, mixed with composer autoloader
• Lack of centralized plugin manager and installer

Proposal

PocketMine only distributes an installer, which manages PHP versions, PocketMine updates and plugins.

The zygote tool

zygote is a native executable that manages PocketMine versions with the following usage:

zygote [-z <zygote directory>] [-d <data directory>] <subcommand> [<versions file>]

Parameters

• : The directory to use for installing executables, default to ./.zygote
• : The directory to use as PocketMine directory, default to ./data
• : The file specifying required versions, default to ./versions.txt

Subcommands

zygote init

Initialize the versions file if it is not already present.

zygote install

Install required tools in ./.zygote, running init if required; also auto-updates tools.

A dummy composer project is created in ./zygote/stage, with a composer.json generated from the versions.txt.

zygote run

Run the server, running install if not yet staged.

zygote run also triggers a zygote install --dry-run and a self-update of zygote, unless the --no--update option is set.

The versions.txt file

Each line specifies a dependency in one of the following types:

• pocketmine <version>: the PocketMine semantic version, e.g. pocketmine 4. Also accepts a local path in place of the version. (If the local path is version-like, use ./)
• minecraft <version>: the client version to support, e.g. minecraft 1.6.200
• <plugin name> <version>: installs an official plugin with the given semantic version, e.g. mineflow 1.3. This is equivalent to pmmp-plugins/<plugin name> <version>.
• <packagist name> <version>: installs an unofficial plugin with the given semantic version, e.g. aieuo/mineflow 1.3
• dev <local path>: installs a plugin in a local directory, relative to parent of versions.txt (not cwd).

Note that versions are automatically prepended with a ^ if no comparators like >, ~, =, etc. are found.

Restructuring PocketMine

The plugin loader logic is removed from PocketMine. Instead, the pocketmine\server function accepts a $plugins array that specifies the plugin descriptions, as specified in the next section.

Platform requirement checking shall have been performed by composer, so they may be removed from the startup script.

PocketMine is released on Packagist as pocketmine/pocketmine-mp. The protocol-dependent network module is split into a separate library.

Writing plugins

A plugin is distributed as a composer library with autoload.files loading a file like the following:

<?php
pocketmine\plugin([
  "name" => "plugin name",
  "version" => "version",
  "main" => fully\qualified\main\ClassName::class,
]);

The api, extensions and mcpe-protocool attributes are removed since it is solved by composer dependencies. Other fields are consistent with current plugin.yml format.

Containerization support

Container images can be built by running zygote install during build and running zygote run during execution.

Old proposal

Proposal

The PocketMine installation is reformed into this structure:

• bin/php/ (containing PHP binaries)
• bin/composer.phar
• zygote.php
• [OPTIONAL] start.cmd or start.sh (no need for start.ps1)
• data/ (data directory)
• stage/ (automated install directory)
• versions.txt

zygote.php

start.cmd/start.sh is just a simple alias to execute php zygote.php. No looping logic is required.
When started without any arguments, zygote.php does the following:

versions.txt

zygote.php loads versions.txt, which contains:

• The desired PocketMine version (specify dev to load from a local clone of pmmp/PocketMine-MP.git)
• The desired plugins, optionally with vendor name, optionally with plugin version

For example:

## PocketMine version
# Latest stable, auto update
PocketMine stable
# Fixed at 3.6.1
PocketMine 3.6.1
# Local installation
PocketMine local ../pmmp-src (a git clone, `.git` required)

## Officially approved plugins (from Poggit or other sources)
# Latest approved version
BlockSniper stable
# Fixed at 3.2.3
BlockSniper 3.2.3

## Plugins from GitHub published on Packagist
# dev-master
SOF3/EvalPlugin
# specific tag
SOF3/EvalPlugin 1.2.3

## Local development plugins
# All directories in ./plugins are automatically included
# Otherwise, relative paths
local ../PurePerms
# Otherwise, absolute paths
local C:/Users/sofe/PurePerms
local /home/sofe/PurePerms
local ~/PurePerms

## Glob patterns
local ../common-plugins/*

composer install

Generate a stage/composer.json based on the dependencies from above, and run composer install.
To improve performance, don't run composer install if filemtime(stage/composer.json) > filemtime(versions.txt)

The magic behind: official platform (e.g. Poggit) submits a modified version of the plugin onto Packagist, including the appropriate composer.json and preserves backward compatibility

Check for zygote.php updates

Otherwise, users will never be motivated to update this script
Rate limit this check to prevent slowing down frequent restarts

Self invocation

Execute $PHP_BINARY __FILE__ --start $args, where $args is the list of plugin directories scanned

zygote.php --start

When invoked with --start, zygote.php should do the job of PocketMine.php
In particular, it is responsible for invoking new pocketmine\Server and creating stdio—logger adapters
In addition, zygote.php shall instantiate plugin classes and pass them as objects to Server::__construct

When --start exits with code 42 (a magic number), or if a command line option was passed, the original zygote process should restart it.

Problems

Management of zygote.php

This file tends to be enormous. Should we make it a phar? In that case, how do we manage the development workflow?

Preprocessing

Naive Packagist distribution prohibits preprocessing to be performed. Is it possible to only publish preprocessed code to Packagist?

Custom plugin loaders

This issue increases the sophistication of loading plugins in custom formats.
However, custom plugin loaders are less necessary since we have composer installer. New plugin frameworks can easily be created by extending the PluginBase class. With #2043, this is not difficult.
After all, all the mess with DevTools and PharPluginLoader are unnecessary when all plugins are in composer folders.
The only non-plugin.yml plugin loader used extensively is ScriptPluginLoader. Can we work out a simple alternative? This needs more discussion.

[dktapps]

While this looks somewhat reasonable in words, I can say this is probably never going to happen due to the amount of work it will require.

I'm not a fan of doing a tremendous amount of reworking to achieve simple goals. 2 out of 3 of the problems listed can be resolved with relatively little changes. As far as a plugin manager goes, we already have tools like Sheep for this and I don't see the benefit of reinventing _that_ that justifies the work it's going to require.

[dktapps]

In addition, as I wrote on discord, composer is simply not designed for this job. You can't have installed modules which depend on the project itself, and this issue makes no apparent attempt to address this problem.

[dktapps]

(this doesn't even begin to dive into how confusing composer would be for the average non-technical user...)

[SOF3]

In addition, as I wrote on discord, composer is simply not designed for this job. You can't have installed modules which depend on the project itself, and this issue makes no apparent attempt to address this problem.

As I wrote on Discord, the dependency graph is:

local -> pmmp/pocketmine-mp
local -> sof3/evalplugin
local -> blockhorizons/blocksniper
sof3/evalplugin -> pmmp/pocketmine-mp
blockhorizons/blocksniper -> pmmp/pocketmine-mp

There is no cyclic dependency at all.

(this doesn't even begin to dive into how confusing composer would be for the average non-technical user...)

If you read the issue carefully, users don't have to know about composer at all. All they need to do is to type a plugin name in versions.txt.

[SOF3]

Two problems that I forgot to mention:

Too new plugins

If the user is running an outdated PocketMine version, it might be difficult to locate the correct branch to fetch plugins from.

Performance of startup

As we all know, loading from source is much slower than loading from phar. There are two excuses:

• it is possible to JIT-compile a phar from stage/ directory, but this is a self-defeating argument.
• most users already ran from source anyway (this is just an excuse)

[dktapps]

this is just an excuse

it's also false in recent months

[SOF3]

I disagree on that number 3 can be resolved with simple solutions. Developing a plugin repository/manager system is indeed complex, and doing it together with composer library resolution is one of the solutions I'm looking at.

[SOF3]

Another advantage/disadvantage: Editing source becomes much harder. You have to either clone it in a separate directory or perform very dangerous and unstable in-vendor editing, both of which are not easy for non-professionals. However, non-professionals should not be editing the source anyway, so I would say this is more an advantage than a disadvantage.

[nathfreder]

[SOF3]

Details of plugin searching:

• Zygote shall glob-scan $stage/vendor/*/*/plugin.yml and copy them into a list at $stage/plugins.list
• For each local path in versions.txt $path,
  
  
  
  • Zygote shall check if $path/plugin.yml exists, and append it into $stage/plugins.list
  
  
  • Zygote shall check if $path/vendor/autoload.php exists, and append it into $stage/composer.json .autoload.files
  
  
• For each local path in versions.txt $path, Zygote shall glob-scan $path/vendor/*/*/plugin.yml and include them.

Users shall receive a big warning (or even be disallowed) if more than one module simultaneously use local, because this may lead to conflict between multiple vendor directories.

[SOF3]

Closed in favour of #2811

[SOF3]

I noticed that this approach could also be used to handle multiple API channels.

For example, if we split PocketMine's network module into a separate library pocketmine/network, plugins that currently need to declare mcpe-protocol could just declare dependency on a specific version for pocketmine/network, while plugins that don't use the network module do not declare this dependency.

While splitting PocketMine into a library sounds like tedious work, it ends up that we only need to remove the plugin loader logic from the core and accept an array of plugin description in the main function. It seems to be less complex than I expected.

Regarding the management of zygote.php, I would alternatively propose that zygote be written in a native executable language such as Go or Rust, such that it is easier to self-update and manage the PHP binaries in the same tool. I am not sure if this should be within the context of the pmmp/PocketMine-MP repo, but for sure this is useful for version management in the long run.

Regarding running from source, it appears that the first stackoverflow answer here is misleading; it is possible to use path as the repository type, and it looks really simple to use.

I propose to reinitiate research on this approach of plugin loading.

[SOF3]

This depgraph illustrates my expected design.
Note that zygote here is a dynamically generated project, not the standard tool distribution.

[SOF3]

An alternative design is to have pocketmine/mcpe as a proxy library that includes the corresponding protocol version, so

• zygote depends on pocketmine/mcpe $CLIENT_VERSION
• pocketmine/mcpe depends on pocketmine/mcpe-protocol $PROTOCOL_VERSION
• plugins depend on pocketmine/mcpe-protocol $PROTOCOL_VERSION

This allows multiple compatible client versions to have the same protocol.

In that case it is possible that we specify the full client version 1.16.200 as the pocketmine/mcpe version, since it does not need to change no matter how pocketmine updates -- the pocketmine/mcpe-protocol version remains in the form protocol.minor.patch, where minor and patch are defined by ourselves while protocol is the client version number, but pocketmine/mcpe 1.16.200 only depends on the ^$protocol version, which can update automatically.

[SOF3]

I have updated the details in the proposal to define behaviour more precisely.