Multiqc: Implement core "plotting" function to produce tables

[dcdanko]

Can MultiQC support adding other tables to the report?

Essentially I just want to display a filtered csv file which can be sorted by different columns.

Edit:

I think I can add this myself. If it's something you would like to add to core MultiQC I am more than happy to contribute.

[ewels]

Hi @dcdanko,

Is the .csv file from a standard bioinformatics tool? Generally the concept behind MultiQC is to display summary data, so I'm not super keen on rendering raw results within the reports, but it depends on what the data is.

If it's more niche, you can of course write your own module to do this see docs. You should be able to piggy-back on much of the user interface code used by the General Statistics table if you produce HTML with the correct markup. You can see a discussion about this with some example code on a previous pull request here.

Let me know if I can help in any way!

Phil

[dcdanko]

I'm not sure if it's niche or not really. I'm taking the output from a CGAT pipeline for analysing differential expression from microarrays, the data isn't raw but plotting alone won't quite cover the richness I'd want.

It shouldn't be hard for me to add this to my fork based on your examples. I suppose my question is more if MultiQC would want to support arbitrary data tables in the same way it currently supports Bar Charts and Line Graphs?

[ewels]

Ok - is the tool producing the output publicly available as a stand alone program? In other words, is anyone outside of CGAT likely to be running it? :)

When I originally put together the tabular stuff I intended to make it reusable for modules, but haven't needed it yet so never got to the refactoring required to make this work nicely. Again, it should be possible - the stuff for the general stats table is already partly in functions so it would be a case of moving this to somewhere where the base module can access it.

A few things confuse this a little - firstly, some of the table markup is done inside the template and the general stats code needs to run after all modules so it needs to be accessible outside of the base module. Both of these should be fixable though.

Please don't edit any of the core general stats functions just at the moment though, I'm currently working on a feature branch at the moment which involves this code, so it will result in a merge conflict.

Cheers,

Phil

[dcdanko]

Probably. I'm not a part of CGAT so I assume there are others besides me using their pipelines. I think I will try to add a few CGAT tools as MultiQC modules eventually.

My approach so far has been to add a plot_data_table() method to the BaseMultiqcModule. The function will take in data (in a similar format to general_stats_addcols()) and return html for a new table.

[ewels]

Ok, great. You touch on something that has been in the back of my mind for a while and needs addressing at some point soon. I can't keep adding new modules indefinitely as they will start to slow MultiQC down too much (each module runs each time). So we could perhaps have a 'bundle' of CGAT modules one day. Or perhaps a single module with multiple sections. Anyway, that's a tangent for now.

Your approach sounds great - in the above I was thinking of using the same code for both tables but I think that's probably overcomplicating things. Let me know how you get on!

Phil

[dcdanko]

Yes I noticed that on the website you had multiple tabs for different types of analysis?

The only thing I'm struggling with now is how to incorporate the html. Should I have a template like arbitrary_data_table.html or should the python code output absolutely everything?

[ewels]

Yes exactly, but the tabs only reflect different input log files. All of those modules were run on all of those reports. Needs some careful thought..!

Initially I tried to keep the HTML in the template, but it's getting increasingly difficult to maintain this. I think it's fair to get the Python to output all markup, this can always be customised with CSS by different templates.

I'll be really interested to see what you come up - I'm now thinking that I should refactor general stats to be a sort of specialist module that still uses the BaseMultiqcModule class. Then it can make use of your new plot_data_table() function and other core functionality.

Phil

[dcdanko]

Hopefully this works out!

It probably is obvious at this point but I think that MultiQC is a really important and useful project.

[ewels]

Great, thanks! It's really nice that so many people are finding it useful.. :)

[lpantano]

I would love to have some function that can create the HTML table!

[dcdanko]

So I have the code to produce an HTML table working but I'm not sure how to hook it into the base module.

Multiqc is correctly finding my testing file and is initializing my module but then quits (error below). How do I signal to MultiQC to keep running? Right now my module is generating html based on the input file and appending that to self.intro

[INFO   ]         multiqc : This is MultiQC v0.6.dev0
[INFO   ]         multiqc : Searching 'test-dir/'
test-dir/head-MFB-MFO.diff_exp.result # output from my module
[WARNING]         multiqc : No analysis results found. Cleaning up..
[INFO   ]         multiqc : MultiQC complete

[ewels]

Hi @dcdanko - is your code available to see somewhere? What log messages do you get in verbose mode (-v or look in multiqc_data/.multiqc_log).

You should only get the No analysis results found message if your module is returning a UserWarning (indicating no recognised files).

Phil

[dcdanko]

I have fixed this.

Current code is here: https://github.com/dcdanko/MultiQC/tree/microarray The relevant files are multiqc/utils/data_table.py and multiqc/modules/diff_exp/diff_exp.py. I've been developing as a module because it seems like there's a fair amount of interest in data tables.

It seems like MultiQC is trying to generate the report in a temp directory?

Here is the full verbose output:

$ multiqc -v test-dir/
[2016-04-19 10:42:52,086] multiqc                                            [INFO   ]  This is MultiQC v0.6.dev0
[2016-04-19 10:42:52,086] multiqc                                            [DEBUG  ]  Command     : /usr/local/bin/multiqc -v test-dir/
[2016-04-19 10:42:52,086] multiqc                                            [DEBUG  ]  Working dir : /home/ddanko/Dropbox/KIR/MultiQC
[2016-04-19 10:42:52,086] multiqc                                            [DEBUG  ]  Template    : default
[2016-04-19 10:42:52,090] multiqc                                            [DEBUG  ]  Loading config settings from: /home/ddanko/Dropbox/KIR/MultiQC/multiqc_config.yaml
[2016-04-19 10:42:52,090] multiqc                                            [DEBUG  ]  New config 'plots_flat_numseries': 100
[2016-04-19 10:42:52,090] multiqc                                            [DEBUG  ]  New config 'num_datasets_plot_limit': 50
[2016-04-19 10:42:52,090] multiqc                                            [DEBUG  ]  New config 'data_dir_name': multiqc_data
[2016-04-19 10:42:52,090] multiqc                                            [DEBUG  ]  New config 'plots_force_flat': False
[2016-04-19 10:42:52,090] multiqc                                            [DEBUG  ]  New config 'plots_force_interactive': False
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'title': None
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'report_readerrors': False
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'fn_ignore_files': ['.DS_Store']
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'prepend_dirs': False
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'log_filesize_limit': 1000000
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'no_version_check': True
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'make_data_dir': True
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  New config 'output_fn_name': multiqc_report.html
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  Added to filename clean extensions. Now looking for: ['.gz', '.fastq', '.fq', '.bam', '.sam', '.sra', '_tophat', '_star_aligned', '_fastqc', '.hicup', '.counts', '_counts', '.txt', '.gz', '.fastq', '.fq', '.bam', '.sam', '.sra', '_tophat', '_star_aligned', '_fastqc']
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  No MultiQC user config found: /home/ddanko/Dropbox/KIR/MultiQC/multiqc_config.yaml
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  Config error: 'dict' object has no attribute 'extend'
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  No MultiQC user config found: /home/ddanko/.multiqc_config.yaml
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  Config error: [Errno 2] No such file or directory: '/home/ddanko/.multiqc_config.yaml'
[2016-04-19 10:42:52,091] multiqc                                            [INFO   ]  Searching 'test-dir/'
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  Analysing modules: diff_exp, snpeff, qualimap, featureCounts, methylQA, rseqc, picard, preseq, samblaster, samtools, bismark, hicup, star, tophat, bowtie2, bowtie1, cutadapt, skewer, fastq_screen, fastqc
[2016-04-19 10:42:52,091] multiqc                                            [DEBUG  ]  Using temporary directory for creating report: /tmp/tmpR1yDKb
[2016-04-19 10:42:52,093] multiqc                                            [DEBUG  ]  Ignoring file as too large: MFB-MFO.diff_exp.result.dontmatchglob
test-dir//head-MFB-MFO.diff_exp.result
[2016-04-19 10:42:52,096] multiqc.modules.diff_exp.diff_exp                  [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,097] multiqc.modules.snpeff.snpeff                      [DEBUG  ]  Could not find any data in ('test-dir/',)
[2016-04-19 10:42:52,099] multiqc.modules.qualimap.qualimap                  [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,100] multiqc.modules.featureCounts.feature_counts       [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,102] multiqc.modules.methylQA.methylQA                  [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,104] multiqc.modules.rseqc.rseqc                        [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,106] multiqc.modules.picard.picard                      [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,108] multiqc.modules.preseq.preseq                      [DEBUG  ]  Could not find any preseq data in ('test-dir/',)
[2016-04-19 10:42:52,109] multiqc.modules.samblaster.samblaster              [DEBUG  ]  Could not find any data in ('test-dir/',)
[2016-04-19 10:42:52,111] multiqc.modules.samtools.samtools                  [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,112] multiqc.modules.bismark.bismark                    [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,114] multiqc.modules.hicup.hicup                        [DEBUG  ]  Could not find any HiCUP data in ('test-dir/',)
[2016-04-19 10:42:52,116] multiqc.modules.star.star                          [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,118] multiqc.modules.tophat.tophat                      [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,119] multiqc.modules.bowtie2.bowtie2                    [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,121] multiqc.modules.bowtie1.bowtie1                    [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,122] multiqc.modules.cutadapt.cutadapt                  [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,124] multiqc.modules.skewer.skewer                      [DEBUG  ]  Could not find any data in ('test-dir/',)
[2016-04-19 10:42:52,125] multiqc.modules.fastq_screen.fastq_screen          [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,127] multiqc.modules.fastqc.fastqc                      [DEBUG  ]  Could not find any reports in ('test-dir/',)
[2016-04-19 10:42:52,127] multiqc                                            [WARNING]  No analysis results found. Cleaning up..
[2016-04-19 10:42:52,127] multiqc                                            [INFO   ]  MultiQC complete

[dcdanko]

I have arbitrary data tables working here: https://github.com/dcdanko/MultiQC/tree/csv_tables

In the above repo/branch I have installed csv_tables as a module for demonstration and testing. Likely this doesn't make sense for a module as part of the larger project; instead it is just an example of a new type of plot.

[ewels]

Great! Glad you got it working. The code looks good, though I might strip it back a bit for use as a core plotting function - mainly removing the parsing functionality as I think this should be within the module really. My view of the data flow is that modules parse data however they want, then they produce HTML by passing data structures to core functions. Data structures are the most variable aspect of the input so I think it makes sense to keep this on the module specific level.

[dcdanko]

I've changed the data parsing to a static method.

I think parsing a csv file would be a common use case so I didn't want to remove the functionality completely.

[dcdanko]

I've updated my data table to use a sql backend. It will now support arbitrary queries which will be useful for filtering large data sets.

[ewels]

Hi @dcdanko,

Thanks for your work on this! I'm getting anxious about scope again though. When will you ever be parsing the amount of data that will need a sql backend? I'm keen that MultiQC remains a tool for reporting summary logs, not doing any data processing of its own. The way I envisage the plot functions is fairly simple - they should be given a dict of parsed data and they should return HTML to go into the report. Parsing functions, SQL backends and filtering large datasets are all fairly far beyond this.

What's the context for this new code - what is it that you're parsing exactly? Are there any example log files I can see to get a better idea of what you're working with?

Cheers,

Phil

[dcdanko]

Still CSV files. Data sets don't need to be huge for sql to be useful.

I haven't removed the non-sql data table though I would like to clean it up a bit if that's the plot you'd like to integrate.

That said my use case actually does involve parsing huge datasets. Rather than a quality control suite I'm building a set of interactive visualizations for my lab. Maybe it wasn't your goal but you've built a really flexible tool set.

[ewels]

Glad to hear it, that was the aim :) Ok great, in that case I might migrate the existing general stats table code into plots and refactor that a bit, then those tools will be there if you need them.

[ewels]

So getting all of the table functionality to work turned out to be quite a massive job (floating headers, toolbox features, sorting by highlighting, column customisation and so on). This code was pretty hardwired into the general stats table, as that was basically the first thing written for MultiQC.

Anyway, it's done! Along with a lot of refactoring of the plots code. I'm pretty happy with it, I think it's quite a bit clearer and will hopefully now be easier to maintain / extend plots generally. Thanks for the suggestion and motivation both!

For reference, you can now make tables in just the same way as bargraphs or other plots, as follows (from my test using bismark data):

table_html = plots.table.plot(self.bismark_data['alignment'], keys)
bargraph = plots.bargraph.plot(self.bismark_data['alignment'], keys, config)
return table_html + bargraph_html

Simple! I'll write up some docs now.

[lpantano]

that is so awesome!!!!! thanks!!!