Multiqc: Qualimap result parser is very slow with high depths (> 8000)

[lbeltrame]

As the subject says, the parser for the results made by Qualimap can be very low with results that have extremely high coverage (such as targeted data from circulating tumor DNA). The parser can take more than one hour running and still not complete processing.

To be precise, the hot path looks like to be _calculate_bases_within_thresholds in QM_BamQC. When this occurs, multiqc gets a constant 100% CPU.

I tested this with a Qualimap file from a sample around 12K X coverage.

[ewels]

Many thanks @lbeltrame - I had noticed that this code could be a little slow, but I've never run it on a report with depths as extreme as you describe here :) I will have a look at whether it can be conditionally disabled when coverage is very high.

If you have a set of Qualimap results which can trigger this which you're able to share, that would be helpful..

Phil

[ewels]

Hi @lbeltrame,

I've just pushed a change which should improve the run speed I hope. If you could install the development version and give it a try that would be brilliant:

pip install --upgrade --force-reinstall git+https://github.com/ewels/MultiQC.git

Note that if you have custom General Stats coverage values that are super high, then it will still be slow..

Phil

[ewels]

Closing this issue as I think it should be fixed. Feel free to reopen or comment if you have any feedback or further problems.

Phil

[ewels]

Hi @lbeltrame,

I've just come back to this as I realised that my previous fix actually broke one of the Qualimap plots. I've just had a sit down and done proper profiling (with help from @Galithil) and worked out where the issue is. I managed to refactor the code to remove a nested loop and the function now runs in 0.1s instead of 10.9s with my test dataset. This should make a huge difference for you with your extreme coverage levels hopefully.

Sorry that a proper fix took so long - let me know if you get to try this one out.

Phil

ps. Ping @vladsaveliev as you wrote this code in the first place - let me know if this change breaks anything else 😉

[vladsaveliev]

That's a very nice stuff, Phil.