Multiqc: BBMap Module

[c-guzman]

Was hoping we could get a module for a couple of the BBMap Suite tools. They're incredibly useful for splice aware and unsplice aware NGS data. (https://sourceforge.net/projects/bbmap/)

BBMap is a suite of NGS tools for NGS. I'm specifically intrested in modules for BBMap, and BBDuk.

BBMap: Short read aligner for DNA and RNA-seq data. Capable of handling arbitrarily large genomes with millions of scaffolds. Handles Illumina, PacBio, 454, and other reads; very high sensitivity and tolerant of errors and numerous large indels. Very fast.

BBDuk: Filters, trims, or masks reads with kmer matches to an artifact/contaminant file.

[ewels]

Hi @c-guzman,

Sounds great! I'm away for a while now (until mid-Feb) so won't be able to write anything until then, but I'd be happy to do so after that. Until then, if you could submit some example data to ewels/MultiQC_TestData then that would be really helpful - then I have something to work with when I get back. Pull requests to this repo for new modules are also welcome, I'll do whatever reviewing I can.

Phil

[boulund]

I've been meaning to write some MultiQC modules for the BBTools suite for my own use. Didn't realize there was an issue on this here already. Maybe we should discuss how to best identify the relevant output files, as none of the BBTools enforce any file naming scheme?
I'm essentially interested in presenting most of the statistics you can get out of e.g. BBMap or BBDuk, which for me means something like this (from the help output):

Histogram output parameters:                                 
bhist=<file>        Base composition histogram by position.  
qhist=<file>        Quality histogram by position.           
qchist=<file>       Count of bases with each quality value.  
aqhist=<file>       Histogram of average read quality.       
bqhist=<file>       Quality histogram designed for box plots.
lhist=<file>        Read length histogram.                   
gchist=<file>       Read GC content histogram.               

I could also be interested in summarizing these with MultiQC:

stats=<file>        Write statistics about which contamininants were detected.
refstats=<file>     Write statistics on a per-reference-file basis.           

I normally output these as compressed (gzipped) textfiles, but as BBTools doesn't enforce any naming scheme I think it can become somewhat annoying to auto-identify these output files (especially considering that the sample/filenames are not included anywhere in the output files). Perhaps it is easiest to recommend/enforce a naming scheme from the MultiQC module's side.

Writing parsers for these files is trivial, so the file naming issue is probably the only problem here.

Edit: oh, and also the statsfile output from BBMap.

[boulund]

Hi,

Just wanted to double-check that no one has been working on this in secret so I don't do any unnecessary work, if I were to start working on this some day this week?

[ewels]

Hi @boulund - sounds great! Yes I haven't made a start yet sorry, been busy with other things lately so haven't had much time to work on MultiQC (also plenty of other issues for me to work on!).

Let me know if I can be of any help.

Phil

[boulund]

That's great! My plan is to put all BBMap related stuff in a single BBMap module, even if it's sometimes a pre-processing tool (bbduk, bbmask, bbnorm, bbmerge), sometimes an aligner (bbmap), sometimes an assembler (tadpole), and sometimes a post-processing tool (reformat, stats). But I guess it doesn't really matter that much? If it does, we could split it into several different modules, but that feels a bit weird. Anyway, I'll start with what I have a need for right now; BBDuk and BBMap stats summaries.

[ewels]

Yeah - that would be my approach too, to keep them all together. See samtools, picard, rseqc and others for examples of where I've done this with other packages.

Irrespective of what they do, if they all come under the umbrella of BBMap then I think it makes sense to keep them together.

Phil

[epruesse]

I'd started something. Give me a few minutes to push the changes to my branch.

[boulund]

I also started something this afternoon. I'm probably about half-way through. I'll hold up until your changes are in, then I'll compare and throw away everything superfluous.

[epruesse]

Ok, here's my work from a while ago: epruesse/MultiQC

[epruesse]

It's on the add-bbmap branch.

[boulund]

Ok, cool! I'll give it a look and see if there's anything useful to be had from a merge between our two approaches. I probably won't have much more time this week however, so I'll see when I get to it. You should definitely keep working on it if you have time. Any feeling for how far you've gotten @epruesse ?

[epruesse]

Not very, I'm afraid. I'd mostly created all the hooks and thrown in the things to detect and parse most of the BBTools stats files. I didn't get around to building graphs yet.

[epruesse]

@ewels BBTools can create a large number of statistics, but doesn't have any default file names. So they need to be detected by content. I'd patched base_module to allow limiting the number of lines read into each file by passing max_lines to find_log_files so that things wouldn't get awfully slow.

It might be more efficient to encode that in search_patterns.yaml, so that down the line code can be added that checks each file only once to send it to the right module, rather than a growing number of modules trying to parse each and every file.

[boulund]

Looks like you took a totally different approach than I was intending to with regards to the file identification. I'm generally not a big fan of using content-based file identification for files like these; I think they look too generic for that, and it means there's a lot of unnecessary opening of files just to find which ones to actually look for (and for files like these, it essentially means reading the whole file most of the time because of filesystem read-ahead, which can potentially absolutely kill performance across a networked file system). My notion regarding the filenames of these files is that I think most people just call them something like <sample>.qchist.txt (but I noticed your reference files in the MultiQC data-repo aren't named like that ;)). I don't think it's really that bad trying to enforce some kind of default naming scheme from MultiQC's side (even if I know it's a bit against MultiQC philosophy). It's easy for end-users to override naming schemes anyway.

I'm slightly impressed by your read-all parser. I was taking a completely separate path there, splitting each type of output into separate modules, arguably unnecessarily duplicating some code snippets... Your's looks cleaner.

[epruesse]

Well, unix file identification (e.g. file) works by looking at the beginning of the file and checking for magic patterns. That works actually quite well, and fast enough. It'd be nice if BBTools would put a clear ID in the first line, but as it is, the first 6 lines so <1kb suffice. That's easy enough, and more robust than requiring specific filenames.

[epruesse]

Also, my experience is that people name things in all kinds of unexpected fashions, including the use of space or (ugh) UTF8 non-ascii characters...

[epruesse]

I've created a PR (#429) for ease of viewing changes. Should we work off of my code or yours, @boulund?

[boulund]

Yeah, content-based file identification is probably not a super-big issue, but I've already experienced severe slowness in the prokka module because of it and I think MultiQC would benefit from the best solution we can come up with regardless :).
I suggest we continue working on your code. I'll incorporate whatever ideas I had that you haven't already implemented any other solutions for, and fill out the docs (for some reason I started there while figuring out implementation ideas).

Regarding file in unix. You're right. I can't recall exactly, but maybe Python's file reading mechanisms read larger chunks, but I might be mistaken. We can always profile it later if it seems slow, no need to spend time on now anyways, so no worries :). Let's focus on getting something that gives us some summary plots/graphs/tables first and worry with other issues later.

[epruesse]

Re reading files with Python -- using the pythonic

import io  # (io.open is python3 open in both 2 & 3)
with io.open(fn) as f:
   for line in f:
      work_on(line)

works better than most "smart" solutions and best in almost all cases. It buffers 8 or 4kb by default (depending on OS), but lowering that to say 1kb (open(...,buffering=1024)) shouldn't yield much gain (depending on FS), in particular if we end up reading the file later.

Mostly, the time is spent with running find_log_files from each module, re-opening the files over and over. A callback solution or some kind of caching should give the best gains. That's MultiQC core however, so Phil @ewels needs to decide how/whether he wants to go about this.

[boulund]

Yes, I did a lot of extensive testing on how to best stream through files in Python a couple of years ago when researching IO performance in Python for a toy project. Though, granted, it was in Python 2.6, so things are probably better now. Sorry for slightly derailing the discussion!

I'll be away for the rest of the week, but I'll go through your code in detail and see if I can make any useful suggestions/additions as soon as I can. Glad to be working with someone else! @epruesse

Earlier today I was thinking about implementing some kind of option that defaults to only showing the richer variant of information in those cases where the same or overlapping info is presented in different output files (e.g. like average read quality vs the boxplot data of the same). Also, I'd love your input/ideas on how to best summarize some of these things, as I feel that just putting the raw data for all samples in graphs quickly becomes cumbersome to interpret.

[ewels]

Hi both,

I started the issue for the much needed file search refactoring way back in #202. It's now pretty much my top priority (in fact, it's the main reason that v1.0 hasn't been released yet). This conversation only gives me more reason to work on it! I'll try to find some time to do so in the near future..

The max_lines for content searching will definitely appear, as well as doing a single loop per run, rather than per module. The current implementation is the result of a slow evolution of this feature since the very start of MultiQC, so it's time it was improved!

Any suggestions for other new functionality is welcome! (pref. on that issue thread).

Phil

[boulund]

Hi @epruesse @ewels!

Now that summer is practically here and MultiQC finally got that update on the refactored file search mechanism, I'm thinking it's a good time to revisit the construction of a BBMap module. I'm guessing no additional work has been done on this since last we spoke?

I'm currently planning to put some work into it towards the end of July, if you have any good ideas on how to best implement plotting the data I'd be happy to hear them! Otherwise, I'll probably borrow inspiration from the FastQC module as it's somewhat similar.

Hope you have a great summer :)

[ewels]

Sounds great! No updates from my end, I'm leaving this module to you guys 😁 Happy to chip in with advice / review as required though!

[epruesse]

This module may also be a prime candidate for the "run multiple times" ticket. BBMap has the same output (options) for each of its tools, so there may be runs from (adapter) trimming, deduplication, contaminant filtering, mapping, etc.

[boulund]

I've done some more work on this today. There are now some very basic plots for some of the different output files from BBMap. Most of my work today was incorporating changes introduced from MultiQC v1.0 that weren't already handled in our previous code.
Next up is tweaking the parsing code for the remaining plot types, and see if I have time to spend on finding the most suitable ways to plot the different data.

[ewels]

Done and merged 👍