MultiQC is not recognising fastp.json files

[tluquez]

Description of bug:
MultiQC is not recognising fastp.json files

MultiQC Error log:

[INFO   ]         multiqc : This is MultiQC v1.6.dev0
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching '.'
Searching 6 files..  [####################################]  100%
[WARNING]         multiqc : No analysis results found. Cleaning up..
[INFO   ]         multiqc : MultiQC complete

File that triggers the error:
To replicate the error, attached you will find the .html and .json output files from fastp in fastp_output.zip

MultiQC run details (please complete the following):

• Command used to run MultiQC: multiqc .
• MultiQC Version: MultiQC v1.6
• Operating System: Ubuntu 18.04 LTS (GNU/Linux 4.15.0-24-generic x86_64)
• Python Version: Python 3.6.5
• Method of MultiQC installation: pip 3

Additional context
I already know, by the issue #791 and others, that the fastp module has just been launched, and I think it is great to have it supported in MultiQC.

Basically, even though the attached files are present in the folder where multiqc is run, it does not recognises them. I was looking at the search_patterns.yaml and realise that it only recognises the search term "fastp.json", which is how my files are named, so I do not understand why is this happening.

[ewels]

Hi @TainVelasco-Luquez,

I suspect that this is because of an installation / update problem. If you pulled the latest changes via a cloned git repository, you'll need to re-run the python setup.py install command again because new modules have additional setuptools handles. Alternatively it could simply be that you don't quite have the latest version of the code? If you run in verbose (-v) or check multiqc_data/multiqc.log you will see a list of all of the modules that are running, and I guess fastp will be missing.

Anyway, when I run with your example output file, the fastp module runs and finds your file. However, it then crashes 😆

Module fastp raised an exception: Traceback (most recent call last):
  File "/Users/philewels/GitHub/MultiQC/scripts/multiqc", line 440, in multiqc
    output = mod()
  File "/Users/philewels/GitHub/MultiQC/multiqc/modules/fastp/fastp.py", line 43, in __init__
    self.parse_fastp_log(f)
  File "/Users/philewels/GitHub/MultiQC/multiqc/modules/fastp/fastp.py", line 199, in parse_fastp_log
    for k in parsed_json['adapter_cutting']:
KeyError: 'adapter_cutting'

This is due to the variation in fastp output files, and isn't really unexpected (I requested more test files on issue #791 for this reason). It looks like your data here is single ended, so is missing all of the stuff you'd expect. I'll look into this now.

Phil

[ewels]

Ok, I added a bunch of new code and your example now works for me :)

Thanks for the example file and feedback! Let me know if you run into any other problems. Hopefully if you pull the latest changes and rerun the installation then things should work ok 👍

Phil

[tluquez]

After a fresh install of the development versions of both fastp and MultiQC, everything goes smoothly.

Thank you very much for such a great tool and the quick response.

Tain.

:ok_hand: :thumbsup:

[GuoYu-Peng]

I have same problem now. My multiqc installed use pip in python 3.6.
multiqc: MultiQC v1.7

[mtinti]

Same in here,
MultiQC v1.7 and fastp with bioconda, pyton 3.7.3

[ewels]

Hi @MatthewP-Newbee and @mtinti,

Please can you make new GitHub issues about your problems and fill out the template / attach example fastp outputs so that I can replicate the issue. Without this I am unable to help.

Thanks,

Phil