Multiqc: Module request: HiC-Pro

[rysterzhu]

• Name of tool:
  
  
  
  • HiC-Pro
  
  
• Tool description:
  
  
  
  • An optimized and flexible pipeline for Hi-C data processing
  
  
• Tool homepage:
  
  
  
  • https://github.com/nservant/HiC-Pro
  
  
• Complete log file output:
  
  
  
  • test.mpairstat.log
  
  
  • test_allValidPairs.mergestat.log
  
  
• Log filename pattern:
  
  
  
  • *stat
  
  
• Most interesting data for General Stats table:
  
  
  
  • total reads, Valid interaction, Valid interaction rmdup, Trans interaction, Cis interaction, Cis shortRange, Cis longRange, %Duplicates, %Valid
  
  
• Data suitable for MultiQC plot(s):
  
  
  
  • amazing Histogram
  
  

[nservant]

Hi guys,
I'm definitely interested in working with you on this.
There are actually more log files that can be useful to plug into MultiQC, all of them based on the same format, and displayed as barplots.
I will try to look into the MultiQC manual to see how it works to write new modules.
Please keep me posted if you move on your side.
Best wishes
Nicolas

[ewels]

Sounds good! I recommend starting with the MultiQC docs which describe how to write new modules: http://multiqc.info/docs/#writing-new-modules

I'll let you know if I start working on this (unlikely in the near future), otherwise assume that it's fine to go for it 👍

[nservant]

Hi Phils,
One question ; how can I disable the bowtie2 automatic log detection ?
Thanks

[ewels]

Aha, does HiC-Pro wrap around bowtie2? The same was true for Bismark. There's no clean and easy answer currently - I just added a small hack to the bowtie2 module to fix that one:
https://github.com/ewels/MultiQC/blob/4c4e83929de1c159c27257c7b483ff0e37eb39b0/multiqc/modules/bowtie2/bowtie2.py#L33-L38

It's probably easiest to just extend this. eg:

# Check that this isn't actually Bismark or HiC-Pro using bowtie 
logf = f['f'].read()
skip_strings = ['bisulfite', 'HiC-Pro']
if not any(s in logf for s in skip_strings):
    self.parse_bowtie2_logs(f) 
else:
    log.debug('Skipping "{}" as contained one of the following strings: {}'.format(f['fn'], ', '.join(skip_strings))) 

[nservant]

Yes, HiC-Pro is using bowtie2, but in a special way (two steps of mapping).
So I was thinking about skipping the bowtie2 'classical" output and making a new one.
Instead of modifying the bowtie2 module, I was wondering if it could be feasible to have a tag in the search_patterns.yaml saying to exclude a given list of log file (based on regexp) for instance ?
That way, it could be useful for any tools, not only bowtie2.

[ewels]

Yes, the same thought crossed my mind. There is already the option to add such strings to skip files, but it's not currently module specific. So if you add a string or a regexp, it will also be skipped for the HiC-Pro module.

[nservant]

ok. So far, I'll go ahead with the piece of code you provided to skip HiC-Pro in the bowtie2 module.
Let me know if you have any other idea to test
Thanks !

[nservant]

Btw, it works well. Thanks

[ewels]

One other option is to try to use some regex-fu with the contents_re config. For example:

bowtie2:
    contents: 'reads; of these:'
    contents_re: '^((?!bisulfite)(?!HiC-Pro).)*$'
    shared: true

This based on a quick play in regex101 - may not actually work in reality.

This will probably be quite slow though as it will now be checking all files for that regex. So it'd be better to add a new exclude flag which is only tested on matching files. I'll have a think about adding this now..

Phil

[ewels]

Ok - @nservant if you pull the latest changes then I've added a new feature for you in https://github.com/ewels/MultiQC/commit/847beb32397602ef2cfa29f1573d88d107d21616

I added the string Hi-C Pro as a criteria to ignore bowtie2 logs:
https://github.com/ewels/MultiQC/blob/847beb32397602ef2cfa29f1573d88d107d21616/multiqc/utils/search_patterns.yaml#L103-L108

Feel free to customise this if you need to exclude files using a different string / criteria.

Updated docs describing the new features are here: http://multiqc.info/docs/#step-1-find-log-files

Let me know if you run into any problems!

Phil

[nservant]

I've just pulled your last version.
Just one question, the string which is in the exclude_contents (here 'Hi-C Pro'), is a keyword that has to be within the file to exclude ?
The point here is that the bowtie log that I want to exclude does not have any special content. This is just a classical bowtie2 output.

250000 reads; of these:
  250000 (100.00%) were unpaired; of these:
    31456 (12.58%) aligned 0 times
    184567 (73.83%) aligned exactly 1 time
    33977 (13.59%) aligned >1 times
87.42% overall alignment rate

Can I for instance, exclude the file using regexp on its name (not on its content) ?
Thanks again

[ewels]

Sure - just use exclude_fn instead of exclude_content (described in the docs).

bowtie2:
    contents: 'reads; of these:'
    exclude_fn: '*_hicpro.txt'
    exclude_contents: 'bisulfite'
    shared: true

[nservant]

Great thanks.
Back to the exclude_contents.
I added the 'Hi-C Pro' keyword in the bowtie logs, but the file doesn't seems to be excluded.
Did I miss something ?

> less bowtie_SRR400264_01_R1_global_hg19.log
#Hi-C Pro
250000 reads; of these:
  250000 (100.00%) were unpaired; of these:
    30562 (12.22%) aligned 0 times
    185070 (74.03%) aligned exactly 1 time
    34368 (13.75%) aligned >1 times
87.78% overall alignment rate

[ewels]

Are you sure you pulled in the latest version and you're not overwriting the default config or anything? I just tried with your file and it's excluded correctly.

[INFO   ]         multiqc : Only using modules bowtie2
Searching 1 files..  [####################################]  100%
[WARNING]         multiqc : No analysis results found. Cleaning up..

(picks up the file fine if I remove the first line)

[nservant]

Humm ... yes I think so. I'll check again

[nservant]

sorry. my fault. It works well. Thanks

[nservant]

Hi @ewels ,
Just to let you know that I have now a HiC-Pro module for MultiQC that works well.
https://github.com/nservant/MultiQC
I just want to make additional tests before creating a pull request.
Best

[ewels]

Fantastic, thanks! Looking forward to seeing the PR 👍

[ewels]

Done in #785 and #811