Multiqc: Hide empty rows in General Statistics table when opening html file

[pancheto]

Description of bug:
I've came across a possible solution for this issue described here #541, although it's not working properly right now. let me describe it:

• what I have
  starting from _fastq_ files and ending up with _bam_ and _vcf_ files, one ends up too with _fastq_ associated reports and also with _bam_ and _vcf_ associated reports. the first ones are listed in General Statistics table with suffixes (_L00\d_R\d_001 typically), and the last ones are listed without them. e.g., as you well said, a first sample_name line and subsequent sample_name lines per lane and direction.

• what I thought
  what happens if I hide all _fastq_ related columns from the _General Statistics_ table by default through the multiqc_config.yaml file? the data would be still there, but only the main sample lines with whole sample information would be visible, and the rest of columns would be accessible on demand through the _Configure Columns_ button.

• what I found out
  if you hide all _fastq_ related columns from the _General Statistics_ table by default the _fastq_ related empty rows keep being displayed by default, although single a touch on any checkbox in the _Configure Columns_ section updates these columns to be displayed only if a _fastq_ related column is selected, or to be hidden otherwise.

• what I request
  I know that this behaviour is not the same thing as merging _fastq_ data into main sample data as this issue started, which would be much more complicated indeed, but being able to hide already empty rows by default could be enough for most of us. also, being able to hide all columns from a section instead of having to write down all columns' ids in the configuration file would be a great plus.

• current example

table_columns_visible:
  fastp:
    pct_duplication: False
    after_filtering_q30_rate: False
    after_filtering_q30_bases: False
    after_filtering_gc_content: False
    pct_surviving: False
    pct_adapter: False
  FastQC:
    percent_duplicates: False
    percent_gc: False
    avg_sequence_length: False
    percent_fails: False
    total_sequences: False

• desired example

table_columns_visible:
  fastp: False
  FastQC: False

MultiQC run details (please complete the following):

• Command used to run MultiQC: multiqc -f --no-data-dir -c multiqc_config.yaml .
• MultiQC Version: MultiQC v1.6
• Operating System: linux-gnu
• Python Version: Python 2.7.11
• Method of MultiQC installation: manual

[ewels]

Hi @pancheto,

Many thanks for the nice suggestion: such a module-wide hiding of columns should certainly be feasible. The automatic hiding of empty rows / columns should happen already, I'll take a look into reproducing and fixing this behaviour.

Note that I have a work-in-progress branch for the new merging functionality that needs testing and finishing here: https://github.com/ewels/MultiQC/pull/576 - you're welcome to have a play around with that code if you would like.

I'm pretty busy travelling for the next few weeks but will get to this as soon as I can 👍

Phil

[matrulda]

I have the same problem: I'm hiding all FastQC related columns by default, but empty rows are displayed when opening the html report. If I check and uncheck a FastQC column using Configure Columns in the web UI the empty rows become hidden.

I'm using v1.7.

I'll try playing around with the code and see if I can find a solution.

[ewels]

Sorry, double-post from https://github.com/ewels/MultiQC/issues/541#issuecomment-553392089

---

Hi @pancheto,

It took me over a year, but I finally implemented your request! 🎉

This now works:

table_columns_visible:
  FastQC: False

Updated docs: https://multiqc.info/docs/#hiding-columns

Let me know if you find any problems, and thanks for the suggestion 👍

Phil

[ewels]

@matrulda - just replicated the bug you found. Looking into it now, pretty sure I know what's going on..

[ewels]

Ok, fixed in 045407a - should now behave as expected on page load.

Tricky bug, thanks for reporting! Let me know how you get on with the new code.

[jamigo]

Sorry, double-post from #541 (comment)

Hi @pancheto,

It took me over a year, but I finally implemented your request! 🎉

This now works:

table_columns_visible:
  FastQC: False

Updated docs: https://multiqc.info/docs/#hiding-columns

Let me know if you find any problems, and thanks for the suggestion 👍

Phil

it would be great to see this last option, but using MultiQC v1.8, although I can perfectly use this:

table_columns_visible:
    fastp:
        pct_duplication: False
        after_filtering_q30_rate: False
        after_filtering_q30_bases: False
        after_filtering_gc_content: False
        pct_surviving: False
        pct_adapter: False
    FastQC:
        percent_duplicates: False
        percent_gc: False
        avg_sequence_length: False
        percent_fails: False
        total_sequences: False

unfortunately this is not working:

table_columns_visible:
    fastp: False
    FastQC: False

in fact, I get the following error:

Traceback (most recent call last):
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/bin/multiqc", line 11, in <module>
    load_entry_point('multiqc==1.8', 'console_scripts', 'multiqc')()
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/easybuild-cesga/software/Compiler/gcccore/6.4.0/setuptools/40.4.3-python-3.7.0/lib/python3.7/site-packages/setuptools-40.4.3-py3.7.egg/pkg_resources/__init__.py", line 484, in load_entry_point
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/easybuild-cesga/software/Compiler/gcccore/6.4.0/setuptools/40.4.3-python-3.7.0/lib/python3.7/site-packages/setuptools-40.4.3-py3.7.egg/pkg_resources/__init__.py", line 2714, in load_entry_point
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/easybuild-cesga/software/Compiler/gcccore/6.4.0/setuptools/40.4.3-python-3.7.0/lib/python3.7/site-packages/setuptools-40.4.3-py3.7.egg/pkg_resources/__init__.py", line 2332, in load
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/easybuild-cesga/software/Compiler/gcccore/6.4.0/setuptools/40.4.3-python-3.7.0/lib/python3.7/site-packages/setuptools-40.4.3-py3.7.egg/pkg_resources/__init__.py", line 2338, in resolve
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/__main__.py", line 44, in <module>
    multiqc.run_cli(prog_name='multiqc')
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/multiqc.py", line 238, in run_cli
    kwargs=kwargs
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/multiqc.py", line 303, in run
    config.mqc_load_userconfig(config_file)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/utils/config.py", line 113, in mqc_load_userconfig
    mqc_load_config(p)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/utils/config.py", line 123, in mqc_load_config
    mqc_add_config(new_config, yaml_config)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/utils/config.py", line 179, in mqc_add_config
    update({c: v})
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/utils/config.py", line 210, in update
    return update_dict(globals(), u)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/utils/config.py", line 217, in update_dict
    d[key] = update_dict(d.get(key, {}), val)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/utils/config.py", line 217, in update_dict
    d[key] = update_dict(d.get(key, {}), val)
  File "/myhome/ngs/software/MultiQC-1.8/BUILD_FT/lib/python3.7/site-packages/multiqc-1.8-py3.7.egg/multiqc/utils/config.py", line 219, in update_dict
    d[key] = u[key]
TypeError: 'bool' object does not support item assignment

[ewels]

Hi @jamigo - please try to avoid posting at the end of old closed issues, as it's very tricky to keep track of these notifications. Instead it's better to make a new issue and refer to any other old relevant issues.

I'll move your comment into a new issue now 👍