Multiqc: Module request: fastp

[AMChalkie]

Not a bug exactly, but I am using fastp instead of afterqc and notice it is not supported, even though it is supposed to be and implementation of the same thing (not sure if/how the output file has changed in the reimplementation).

So: multiqc doesn't recognise fastp files (that are created to be the same as afterqc).

Also a useful addition would be:

• listing an invalid module choice in alphabetical order.

Thanks for the tool, it is great!

[ewels]

Hi @AMChalkie,

You can see the example data from AfterQC that were used for the MultiQC module here: https://github.com/ewels/MultiQC_TestData/tree/master/data/modules/afterqc

MultiQC finds the output files from AfterQC using the following search rules: https://github.com/ewels/MultiQC/blob/master/multiqc/utils/search_patterns.yaml#L7-L9

If the output is identical and still matches these search patterns, it _should_ work. If you can identify the differences then we should be able to modify the module to work with both. If you're able to upload some example outputs from fastp to the MultiQC_TestData above in a comparable folder structure, that would be a great start 👍

Regarding the alphabetical order thing, you mean this error message?

Error: Invalid value for "-m" / "--module": invalid choice: fastp. (choose from rna_seqc, theta2,
 peddy, bismark, preseq, fastqc, sargasso, sortmerna, cutadapt, deeptools, goleft_indexcov,
 bamtools, qorts, kallisto, busco, samtools, clusterflow, picard, disambiguate, rseqc, quast, interop,
 bbmap, gatk, hicup, bowtie2, qualimap, conpair, vcftools, featureCounts, star, adapterRemoval,
 flexbar, bcftools, supernova, rsem, salmon, fastq_screen, methylQA, htseq, hicexplorer, snpeff,
 verifybamid, bowtie1, trimmomatic, jellyfish, homer, bcl2fastq, prokka, slamdunk, leehom,
 biobloomtools, hisat2, afterqc, macs2, skewer, tophat, custom_content, samblaster)

Yes, that's probably a good idea! It was probably easy enough to read when I originally wrote that log message with about 5 modules, but times have changed since then 😉

Phil

[ewels]

Ok, module names now alphabetically sorted 👍

I couldn't customise the output as much as I'd have liked to (eg. have a bullet pointed vertical list) as the message is generated by the click library that handles the command line option parsing, but at least now that it's sorted it should be a little easier to skim through. Thanks for the suggestion!

Phil

[AMChalkie]

Great with the alphabetic sort!

For the parsing, it looks like the .json from fastp doesn't contain the search string

afterqc:
    fn: '*.json'
    contents: 'allow_mismatch_in_poly'

more example_fastp_report.json | grep allow_mismatch_in_poly
[achalk@bioinfo1 fastp] 2018-01-24 10:19:39$ 

A better search might be for the tool name in the .json file.

more example_fastp_report.json | grep fastp
        "command": "fastp -i FCHJ7K5ALXX_L5_CHKPEI85217030079_1.fq.gz -I FCHJ7K5ALXX_L5_CHKPEI85217030079_2.fq.gz -o fastp._1.fq.gz -O fastp._2.fq.gz -h fastp._report.html --json fastp._report.json -5 -3 --overrepresentation_analysis "

In fact, maybe thats the case for a few of these? (And the requirement that in a .json file the command contains that information (and version number - but thats scope creep).

I've included a .json output here from fastp: example_fastp_report.json.gz

Cheers
/Alistair

[ewels]

Thanks for this. Unfortunately from a very quick scan the structure and content of this JSON file looks very different to the AfterQC one. It may be more trouble than it's worth to get the AfterQC module to support the output from FastP, easier instead to just make a new MultiQC module for FastP. Or at least, it'll need a fair bit of thought.

[mblue9]

It would be great to have MultiQC support for fastp!! as there's now a Galaxy version of fastp https://github.com/galaxyproject/tools-iuc/pull/1763
(hello @AMChalkie 👋 fancy meeting you here :smile:)

[srynobio]

+1 for fastp support

[florianduclot]

I started working on creating a module for fastp:
https://github.com/florianduclot/MultiQC/tree/add_fastp_module

This is my first attempt at creating a module so it only populate a few data in the _general statistics table_, but wanted to share it for those who would feel interested in improving and building upon this. I'll try to upload this to the best of my time and abilities in the meantime.
Of note, I used the _afterqc_ module as starting point, which explains the commented portions of the code.

[ewels]

Hi @florianduclot - this is great! Looking forward to seeing the PR when it's ready.. Shout if I can be of any help (here or on gitter)

Phil

[florianduclot]

Just wanted to give a quick update:
I finally figured out a way to parse the required data, and I thus have populated the general statistics table with a few new metrics.
Next steps would be:

• [ ] Add all important plots. There are a lot of possibilities (http://opengene.org/fastp/fastp.html) but a few are more important than others.

• [x] Cleanup code as much as possible.

• [x] Fix formatting for some metrics in the general statistics table.

• [x] Add test data to MultiQC_TestData repository

• [x] Parse only required keys of JSON report.

• [x] Add fastp module to documentation

• [x] Parse whole JSON report and write data for multiqc_data output

[ewels]

Parse only required keys of JSON report.

Note that although I encourage only putting the most relevant information into the HTML reports, I often end up parsing all available information so that it can be written into the multiqc_data files. This can be helpful for people doing downstream analysis and also means that the data will be passed on to other downstream applications (such as MegaQC).

Phil

[florianduclot]

I just submitted a corresponding PR: #782.

[timedreamer]

Hi,

I tried MultiQC v1.6.dev0 on fastp outputs, but no analysis could be found. Command is multiqc .. Any ideas? Or I'm using an older version? I think this is the newest version on conda. Below is the MultiQC output.

Thanks!

[INFO   ]         multiqc : This is MultiQC v1.6.dev0
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching '.'
Searching 6 files..  [####################################]  100%
[WARNING]         multiqc : No analysis results found. Cleaning up..
[INFO   ]         multiqc : MultiQC complete

[florianduclot]

Hi @timedreamer , sorry you're having an issue...

Do you see any mention of the fastp module in the multiqc_data/multiqc.log log? Either way, this log may bring more information to explain what's happening.

I believe the fastp module started being available in 1.6dev0 (v1.6.dev0 (3935c8c)).

[timedreamer]

Is this directory in where I run multiqc . ? I can't find a directory called multiqc_data or the file multiqc_log. Does it get cleaned?

Sorry if this question is obvious.

[florianduclot]

By default I believe it is in the working directory from which you ran multiQC, yes. If you don't see it, it's probably because no report was generated (which would be the case if no other reports were found).

In this case, maybe running the multiqc command with the -v (verbose) flag and pasting the output here may help.

[timedreamer]

You are right. I think the fastp module is missing in my version MultiQC v1.6.dev0. Below is the partial verbose output (Some dir info was deleted).

Thanks.

[2018-07-23 12:42:21,204] multiqc                                            [DEBUG  ]  No MultiQC config found: anaconda3/lib/python3.6/site-packages/multiqc_config.yaml
[2018-07-23 12:42:21,204] multiqc                                            [DEBUG  ]  No MultiQC config found: /.multiqc_config.yaml
[2018-07-23 12:42:21,205] multiqc                                            [DEBUG  ]  No MultiQC config found: multiqc_config.yaml
[2018-07-23 12:42:21,205] multiqc                                            [DEBUG  ]  Command used: /anaconda3/bin/multiqc -v .
[2018-07-23 12:42:21,378] multiqc                                            [DEBUG  ]  Latest MultiQC version is v1.5
[2018-07-23 12:42:21,379] multiqc                                            [INFO   ]  This is MultiQC v1.6.dev0
[2018-07-23 12:42:21,379] multiqc                                            [DEBUG  ]  Command     : /anaconda3/bin/multiqc -v .
[2018-07-23 12:42:21,379] multiqc                                            [DEBUG  ]  Working dir : data/trim_fastq/metrics
[2018-07-23 12:42:21,379] multiqc                                            [INFO   ]  Template    : default
[2018-07-23 12:42:21,379] multiqc                                            [INFO   ]  Searching '.'
[2018-07-23 12:42:21,379] multiqc                                            [DEBUG  ]  Analysing modules: custom_content, conpair, peddy, methylQA, qualimap, preseq, quast, qorts, rna_seqc, rsem, rseqc, busco, goleft_indexcov, disambigu
ate, supernova, deeptools, sargasso, verifybamid, homer, macs2, theta2, snpeff, gatk, htseq, bcftools, featureCounts, dedup, picard, prokka, samblaster, samtools, bamtools, jellyfish, vcftools, longranger, bbmap, bismark, hicexplorer, hi
cup, salmon, kallisto, slamdunk, star, hisat2, tophat, bowtie2, bowtie1, kat, leehom, adapterRemoval, clipandmerge, cutadapt, flexbar, trimmomatic, skewer, sortmerna, biobloomtools, fastq_screen, afterqc, fastqc, clusterflow, bcl2fastq,
interop
[2018-07-23 12:42:21,380] multiqc                                            [DEBUG  ]  Using temporary directory for creating report: /state/partition1/job-7749962/tmpwv1ij2_b

[florianduclot]

Indeed, it looks like the bioconda version is behind. If you need to run the fastp module, I would install MultiQC manually, then:

git clone https://github.com/ewels/MultiQC.git
python setup.py install

[timedreamer]

Thank you.

[ewels]

Exactly, thanks both! I will do a release very soon which will include this module.

[dbu008]

Dear @ewels,

the bioconda multiqc still does not have the fastp?

I have multiqc and it does not see the fastp files.

Moreover, git clone https://github.com/ewels/MultiQC.git
python setup.py install

it does not work with my Mac Catalina.

Is there any other option to download multiqc working with fastp?

I really like your tool, however I do not want to exchange the fastp to afterqc, which is very slow.

I would be glad if you could answer what to do.

Best,
Dorota

[ewels]

Hi @dbu008,

Please make a new issue for this and fill in all of the fields in the issue template: https://github.com/ewels/MultiQC/issues/new?template=bug_report.md

Thanks,

Phil