Multiqc: MatPlotLib release candidate crashes MultiQC

[lpantano]

Description of bug:
This happened as I was trying the updated version of nf-core/smrnaseq, so maybe the issue is another but I figured you will know if it is something I did wrong quickly.

MultiQC Error log:

/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/utils/config.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  configs = yaml.load(f)
/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/utils/config.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  sp = yaml.load(f)
/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/utils/config.py:121: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  new_config = yaml.load(f)
[INFO   ]         multiqc : This is MultiQC v1.7
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching '.'
[INFO   ]         multiqc : Only using modules adapterRemoval, fastqc, mirtrace, custom_content
/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/modules/custom_content/custom_content.py:84: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  parsed_data = yaml.load(f['f'])
[INFO   ]  custom_content : nf-core-smrnaseq-summary: Found 1 sample (html)
[INFO   ]  custom_content : nf-core-smrnaseq-summary: Found 1937 samples (html)
[INFO   ]  custom_content : software_versions: Found 1 sample (html)
[INFO   ]  custom_content : software_versions: Found 522 samples (html)
[ERROR  ]         multiqc : Oops! The 'mirtrace' MultiQC module broke...
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues
  If possible, please include a log file that triggers the error - the last file found was:
    ./mirtrace/mirtrace/mirtrace-stats-mirna-complexity.tsv
============================================================
Module mirtrace raised an exception: Traceback (most recent call last):
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 170, in plot
    return get_template_mod().bargraph(plotdata, plotsamples, pconfig)
AttributeError: module 'multiqc.templates.default' has no attribute 'bargraph'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/bin/multiqc", line 440, in multiqc
    output = mod()
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 74, in __init__
    plot = self.mirtrace_qc_plot()
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 237, in mirtrace_qc_plot
    return bargraph.plot(self.summary_data, keys, config)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 181, in plot
    matplotlib_bargraph(plotdata, plotsamples, pconfig)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 486, in matplotlib_bargraph
    fig.savefig(img_buffer, format='png', bbox_inches='tight')
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/matplotlib/figure.py", line 2180, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 2060, in print_figure
    bbox_extra_artists=bbox_artists)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/matplotlib/figure.py", line 2361, in get_tightbbox
    artists = self.get_default_bbox_extra_artists()

File that triggers the error:

mirtrace-stats-mirna-complexity.tsv.gz

MultiQC run details (please complete the following):

• Command used to run MultiQC: [eg. multiqc .]
  multiqc . -f --config multiqc_config.yaml -m adapterRemoval -m fastqc -m mirtrace -m custom_content
• MultiQC Version: [eg. MultiQC v1.1]
  1.7
• Operating System: [eg. macOS Sierra 10.12.5]
  centos7
• Python Version: [eg. Python 2.7.12]
  python3.6
• Method of MultiQC installation: [eg. pip, conda, manual]
  conda

Additional context
Add any other context about the problem here.

[lpantano]

this is related to matplotlib > 3.0.3, I have to set up matplotlib to that version to avoid the error. See here https://github.com/nf-core/rnaseq/pull/214

[ewels]

[ggabernet]

Hi, just for the record I had the same error and fixing the matplotlib version to 3.0.3 solved the issue. Thanks for posting!

[ewels]

Short answer

This bug has _kind of_ been addressed in #928 where a new <3.1.0 pin for matplotlib was added in setup.py.

This is kind of a crappy solution, but should mean that MultiQC continues to work for most people for a bit. But the longer it stays there the more annoying it will be for people with conflicting python package dependencies.

Long answer

Ok, so looking into the code the AttributeError: module 'multiqc.templates.default' has no attribute 'bargraph' bit is a red herring - this exception is expected:

https://github.com/ewels/MultiQC/blob/cab5a612734efd84cdb3f775e52c36114317b140/multiqc/plots/bargraph.py#L169-L171

So the real problem is the second one:

Traceback (most recent call last):
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/bin/multiqc", line 440, in multiqc
    output = mod()
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 74, in __init__
    plot = self.mirtrace_qc_plot()
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/modules/mirtrace/mirtrace.py", line 237, in mirtrace_qc_plot
    return bargraph.plot(self.summary_data, keys, config)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 181, in plot
    matplotlib_bargraph(plotdata, plotsamples, pconfig)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/multiqc/plots/bargraph.py", line 486, in matplotlib_bargraph
    fig.savefig(img_buffer, format='png', bbox_inches='tight')
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/matplotlib/figure.py", line 2180, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 2060, in print_figure
    bbox_extra_artists=bbox_artists)
  File "/om2/user/lpantano/shared/conda3/envs/nf-core-smrnaseq-1.0dev/lib/python3.6/site-packages/matplotlib/figure.py", line 2361, in get_tightbbox
    artists = self.get_default_bbox_extra_artists()

The line of MultiQC code that triggers the error is when we try to save the matplotlib figure:

https://github.com/ewels/MultiQC/blob/cab5a612734efd84cdb3f775e52c36114317b140/multiqc/plots/bargraph.py#L486

To go any further, I need to be able to replicate the error myself and get more logging detail. However, I'm having problems accessing anaconda.org at the moment so can't install a test environment 🤦‍♂

To be continued... I will drop this issue from the v1.8 release milestone, but keep it as high priority to find a proper solution.

Phil

[ewels]

So I just looked in to this - whilst pip avoids pre-releases, setuptools does not - see https://github.com/pypa/setuptools/issues/855 - that is why we are having difficulties here. Unstable release candidates are installed and those are causing problems.

[ewels]

Solved by changing documentation and tests to use pip install . instead of python setup.py install (or pip install -e . instead of python setup.py develop).