Multiqc: Module Repository Ideas

[ewels]

Introduction

As MultiQC grows, so does the number of supported tools. Currently, each new module is added to the main MultiQC distribution and goes out to every user. This isn't sustainable - at some point there will hopefully be hundreds of modules and most users will only need a handful.

So - we need some way to manage active modules. We also need to consider how to host modules - core distribution versus niche.

Model 1 - Core Release

1. Bundle all modules with the core package, as done currently.
2. Build a new command line wizard that writes config files for users that builds a config file that populates config.exclude_modules
3. Add new command line option to print list of active and inactive modules
4. Add logging messages that encourage use of the wizard

Thoughts:

• _Pro:_ Relatively little work
• _Pro:_ Keeps code centralised and together (easy to find modules)
• _Con:_ Newly written modules will be added in new releases of MultiQC and automatically enabled. This is both good and bad, but probably mostly bad.
• _Con:_ The main MultiQC repository could get huge and difficult to manage

Model 2 - GitHub organisation

1. Make a MultiQC organisation on GitHub (optional)
2. Make GitHub repos in this organisation for each module
3. Ship MultiQC program with few / no modules.
4. _Do_ ship the search configuration for all core modules. If a file is found for a missing module, prompt the user to download the module.
5. Rebuild the MultiQC log search function so that it does all modules in one sweep using core code.
6. Build features to download new modules (also with option to name desired modules).
   
   
   
   • Could also list available modules / notify about new ones?
   
   

Thoughts:

• _Pro:_ Less code to ship and maintain with the core program
• _Pro:_ Code still relatively centralised and vetted by @ewels
• _Pro:_ Organisation approach allows easy way to expand MultiQC "team" of maintainers
• _Pro:_ Users don't have to do much, will be prompted when they're missing modules.
• _Con:_ More reliant on internet connection
• _Con:_ Requires extra download step before use - less fluid
• _Con:_ List of active modules will only ever grow, no removal mechanism.
• _Con:_ May be tricky for modules with strange search patterns. Probably can overcome though.
• _Con:_ Need to work out how this will fit into existing setuptools installation system.

Model 3 - Distributed modules

1. Build some kind of module registration system. Either a file within MultiQC or using a website.
2. Modules can be hosted anywhere. Add support for GitHub / BitBucket etc.
3. Build code to install new modules on command line. _eg._ multiqc install ewels/multiqc-fastqc

Thoughts:

• _Pro:_ More level playing field for modules. No classification of core / niche.
• _Pro:_ Expands compatability beyond GitHub.
• _Con:_ Requires some kind of repository listing to be built and maintained.
• _Con:_ Module building outside of core maintainer ecosystem.
• _Con:_ Other cons from Model 3 apply.

Summary

My gut feeling is that Model 2 is probably the best way to go, with an organisation fork for every core module. I like that users will be prompted in Model 2 and don't have to research available modules themselves. This keeps the simplicity of always just running multiqc . for any analysis directory.

These don't have to be mutually exclusive - I can also add in support to install from everywhere as mentioned in Model 3 but without being reliant on a listing. Module names can be preferred from the central repo, _eg._ multiqc install fastqc will first look for an organisation repository called fastqc. Other syntaxes can then be handled in a hierarchical manner - eg. user/modname will assume GitHub, then full paths can be specified etc. This is modelled on how NextFlow pulls pipelines (see here and here).

Does anyone have any suggestions that fall outside of the above? I will add in comments below for some kind of voting but feel free to add comments.

[ewels]

Give this comment a 👍 if you like Model 1

[ewels]

Give this comment a 👍 if you like Model 2

[ewels]

Give this comment a 👍 if you like Model 3

[ewels]

Ping @lpantano @dcdanko @dakl @moonso @robinandeer @guillermo-carrasco @avilella

[lpantano]

Good discussion. I like model 2, and model 3, although maybe need much work. I like the 3 point of model 3 though as an easy installation of modules.

[lpantano]

I think for model 2, if all repos are inside one github repo, it would be easy to remove modules if we can assign a developer to each module. It can be done as BioConductor, after 2 years, developers need to ping if the package wants to be there. Something like, last commit was 2 years ago, cannot install. This way, if somebody complain we can port to other developer or whatever.

[dakl]

My vote is for model 2.

This is related to the comparison between module vs plugin. What is included in the core functionality of multiqc (modules) and what should be project/group specific and mostly not relevant to other people (plugins). With model 2, the distinction still applies since I assume that only modules would be added in the multiqc org.

Re removal mechanism, there is one. If modules are defunct and outdated, they can be removed by admins (Mostly by @ewels I guess) if no-one commits to updating that (mainly the responsibility of the original developer). Edit: By "removed", I think I mean tagged as defunct so that fresh installations of multiqc won't see them but older installation still can (to play nice with versions of modules, see below).

In model 2, how would module versioning work? I think it would be good to have a mechanism for a user to lock down to a specific version of the modules they use, i.e. allow the modules.include files to have a pattern like

fastqc==0.1.1
samblaster==@1234abcd
bwa

or similar. I use multiqc in a production environment an want to provision my prod server with specific version of the tools I use. (maybe there's a better way of doing this...)

@ewels Great work with multiqc!

[robinandeer]

I was planning something simliar for my tool Chanjo. I went with Setuptools entry_points which I guess is the closest to an official Python plugin system. It would fit into _model 3_.

• _PRO_ you get a solution for versioning modules
• _PRO_ you can quite easily integrate functionality like plugin-specific CLI options/subcommands
• _PRO_ modules need to explicitly pick which version(s) of MultiQC they support
• _CON_ it requires module authors to learn quite an advance Python topic and publish on PyPI, I wish I didn't feel this was such a big hurdle..

Like you say @ewels, I agree that it's nice to ship a standard set of modules/have an official listing of vettet plugins _and_ have the non-official ones be distributed. Perhaps entry_points could be used for modules that require advanced functionality?

Also, take a look at homebrew maybe, they have an official repo with many modules and then you can "tap" un-official ones. Works really well.

At this point I think I would start with Model 2 if not just to separate modules from the core code.

[ewels]

Thanks for the comments everyone!

@lpantano - I agree that having the module centralised in one organisation allows sharing maintainer responsibilities and lessens the chance of modules "going stale". Excellent point, and definitely a _Pro_.

@dakl - Yes you're right, in fact old modules could be labelled defunct by simply removing their search patterns from core MultiQC (people can add them back in their own config files). I more meant that users will only ever add modules and could still end up with a long list that is slow to run. But if I refactor the file searches then maybe this won't be too much of a problem as we'll always be searching for all core modules anyway, whether they're installed or not.

Versioning should definitely be possible and was in the back of my mind when I wrote the above. See the nextflow docs for a nice example of how to handle this. Should be possible to specify tags (release versions) or exact commits as you suggest, or to leave it blank and take the latest version. But instead of using an -r flag this info would presumably be added to the user's MultiQC config file if desired. _eg:_

mqc_module_versions:
  fastqc: 0.1.1
  samblaster: @1234abcd
  # Omitting bwa here would take latest version

@robinandeer - MultiQC already supports Setuptools entry_points as you describe. It was you who helped me write this code ;) It will always support niche add on modules in this way. _eg._ see the NGI plugin. The biggest _Con_ with leaving this as it is (with no core distributed modules) is the difficulty of _discovering_ modules.

Homebrew is an interesting idea, not something I'd considered. It does add an extra dependency to running MultiQC though. Is it not OSX only?

Model 2 is chosen!

You could probably tell by the way I wrote this issue, but I was kind of gunning for Model 2 myself. It was rewritten a number of times as I wrote the issue and I kept thinking of extensions to the idea. Kind of a GitHub issue based version of rubber duck debugging I guess.

Glad everyone else seems to agree with me! This isn't going to happen any time _very_ soon, certainly not before v0.6 release, but I will try to get the ball rolling with the new organisation.

Thank you all for your input!

[robinandeer]

@ewels: I meant that homebrew has an interesting idea how to deal with modules. Not adding it as a dependency :)

[dakl]

I don't think homebrew has support for versions though.