Multiqc: ValueError with html as custom_content in config

[Maarten-vd-Sande]

Description of bug:
When I try to add a custom_content section with html to a config it seems like it tries to unpack the html into a sample and value kind-of-thing, while the html type does not actually contain this structure.

multiqc_config.yaml:

report_header_info:
    - Date: 'December 03, 2019'


custom_data:
    my_data_type:
        id: 'ID section'
        section_name: 'Section name'
        plot_type: 'html'
        data: 'html text here'

some_data.out:

# section_name: 'Custom data file'
# description: 'This output is described in the file header. Any MultiQC installation will understand it without prior configuration.'
# format: 'csv'
# plot_type: 'table'
# pconfig:
#    id: 'custom_bargraph_w_header'
#    ylab: 'Number of things'
Sample Name,some feature
GSM2219688_pass_1,42

Now run:
multiqc -c multiqc_config.yaml some_data.out

MultiQC Error log:

[INFO   ]         multiqc : This is MultiQC v1.8
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching   : /home/sande/Dropbox/Studie/PhD/snakemake-workflows/some_data.out
[ERROR  ]         multiqc : Oops! The 'custom_content' MultiQC module broke... 
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues 
  If possible, please include a log file that triggers the error - the last file found was:
    None
============================================================
Module custom_content raised an exception: Traceback (most recent call last):
  File "/home/sande/anaconda3/envs/snakemake-workflows/envs/multiqc/lib/python3.6/site-packages/multiqc/multiqc.py", line 546, in run
    output = mod()
  File "/home/sande/anaconda3/envs/snakemake-workflows/envs/multiqc/lib/python3.6/site-packages/multiqc/modules/custom_content/custom_content.py", line 64, in custom_module_classes
    cust_mods[c_id]['data'].update( f['data'] )
ValueError: need more than 1 value to unpack
============================================================
[WARNING]         multiqc : No analysis results found. Cleaning up..
[INFO   ]         multiqc : MultiQC complete

[ewels]

Hi @Maarten-vd-Sande,

Good spot - this seems to be something specific to the parsing code for config files. I think if you put the HTML in a separate _mqc.{something} file then it should work ok.

I'll look in to fixing this though, as it is a bug.

Phil

[Maarten-vd-Sande]

Thanks for the reply and solution. I'll leave the issue open for your administration.

[ewels]

Hi @Maarten-vd-Sande,

This is now fixed in v1.9dev - please do have a play and see if it works for you.

The problem was due to the fact that data usually has key:value pairs (typically one key for each sample). That makes sense for other plot types, but for HTML you want (and gave it) a big string instead. I've tweaked the code to tolerate this and it now seems to work fine.

Phil

[ewels]

Also note that I just added new functionality to make it easier to insert HTML into reports.

For example, you could now have:

myfile_mqc.html

<!--
id: 'ID section'
section_name: 'Section name'
-->
<p>html text here.</p>

[Maarten-vd-Sande]

I was just trying it out. I didn't manage to break it yet, so that's a good thing! One thing I was wondering if it is intended that the section_name appears twice. Fine by me either way.

I'm very excited inserting html into reports, which means I can put some "explanation" as how samples/metrics were derived.

[ewels]

Ah interesting - I was playing with that and I thought that I'd set it to hide that second description line if it wasn't set in the header config (or if there is no header config). I tweaked core MultiQC a little to make this happen - if you set description: False then I think it should explicitly stop printing that second line.

Note that you can override the description text for any module, this is usually how people add those kinds of descriptions. See the docs - you want module_order then set the info key. You can see this config file for an example of that in action.

Phil

[ewels]

ps. If you can send me your test file with the double headings I can double check.

[Maarten-vd-Sande]

You can reuse the files from the "original" issue. Maybe I misunderstood, but adding description: False just adds the text False behind the description -> "The section name False".

[ewels]

This works fine for me:

custom_data:
    my_data_type:
        id: 'ID section'
        section_name: 'Section name'
        description: False
        plot_type: 'html'
        data: 'html text here'

I set it to hide that second description line if it wasn't set in the header config (or if there is no header config).

This was only with *_mqc.html files with a header. I'll change the upstream behaviour though so it has similar function for any input type.

[Maarten-vd-Sande]

Okay my bad. This now works. I think I was using 7d5ecb20977585f675c4ccbcb695173869a3fff8 and not ae8e146dd2615c486de48bfb9f224c4b911c90f3.

Thanks for the fix :angel:

[ewels]

No problem! I've been iterating with it a bit.

Just updated it one last (?) time - False is no longer needed. If no description or link is provided then that line will be skipped (for any MultiQC section, not just Custom Content).

I also rediscovered the extra argument for the add_section() function and mirrored this across to Custom Content with the new section_extra key. That's another method that you can use to add custom HTML to a report section, above the plot.

Thanks for testing!

Phil