Multiqc: Spectra/colormath issue

[rasmusagren]

Hi,

It seems the spectra/colormath issue is still a problem. I think it's one of your newer PRs on conda-forge or bioconda that breaks it. My CI tests used to run fine but now I get No module named 'spectra' from MultiQC 1.4 from bioconda. I suggest you remove the noarch setting for all the involved packages, it always seem to cause problems. I only get this issue on some systems, so it's probably sensitive to which builds of the various packages that were installed.

Cheers,
Rasmus

[ewels]

Sad times.. Could you please open an issue about this on https://github.com/bioconda/bioconda-recipes/ about this? These bioconda packaging errors are beyond my expertise now so this isn't something that I can fix by myself. Feel free to tag me there and link to this issue though. Maybe also worth asking on the bioconda gitter channel.

[rasmusagren]

I "solved" it by using MultiQC 1.3, and I'm afraid I don't have time to spend on it right now. But removing noarch never hurts and I have a feeling it will solve it. I think bioconda suggest people not to use it right now, but I don't know why.

[ewels]

We had to add noarch to fix other problems in the past (click incompatibility with python 3 and locale settings from memory). So I don't think that it was added lightly, and I think that removing it _will_ hurt unfortunately 😉

[ewels]

Wait, I can't find noarch on the MultiQC recipe now. I'm confused. I guess it must be on spectra then. Looks like this is why I added it: https://github.com/conda-forge/spectra-feedstock/pull/1#pullrequestreview-89443766

[ewels]

Calling @mbargull - conda build wizard! Any ideas what could be happening here?

[mbargull]

Calling @mbargull - conda build wizard!

:laughing: that's a funny one -- not necessarily true, but thanks nonetheless :wink:
@rasmusagren: I'd need some more information on that, e.g., conda info, conda config --show-sources, packages in environment, and the like. Or just a (a link to) a build log.

[rasmusagren]

Here is the build log that fails. It activates a conda environment and runs a bunch of tests. One fails due to MultiQC not being able to load spectra.

https://bitbucket.org/scilifelab-lts/lts-workflows-sm-scrnaseq/addon/pipelines/home#!/results/130

The conda environment is set up in a Docker image built on Docker Cloud. The build log for that image is attached below. I really hope now that this doesn't come from me screwing up!

docker_cloud_log.txt

[mbargull]

But removing noarch never hurts and I have a feeling it will solve it. I think bioconda suggest people not to use it right now, but I don't know why.

Some issues with noarch had to be ironed out since it was introduced. In Bioconda we still don't use it, but I hope to change that soon. Using the (reasonably) latest conda and conda-build should help a lot with that. (The only real issue I can currently think about is that we can't any noarch packages in constructor (the software that is used to create the Miniconda/Anaconda installers).)

[mbargull]

The conda environment is set up in a Docker image built on Docker Cloud.

Oh, that's just heaven for debugging stuff!

|[maba@mb830 ~]$ docker run --rm -it scilifelablts/lts-workflows-sm-scrnaseq-ci:latest /bin/bash
Unable to find image 'scilifelablts/lts-workflows-sm-scrnaseq-ci:latest' locally
latest: Pulling from scilifelablts/lts-workflows-sm-scrnaseq-ci
4f84e00c7348: Pull complete 
6d84121dba27: Pull complete 
3116a1a43400: Pull complete 
3b2ca9764399: Pull complete 
e182d2795ab9: Pull complete 
6208eca81e6c: Pull complete 
acf0a4310b4f: Pull complete 
5eb7908acff8: Pull complete 
aa0338ec039a: Pull complete 
43e507043112: Pull complete 
ae111fae3eea: Pull complete 
121f082ecbe2: Pull complete 
2011a7a4ec6b: Pull complete 
7b16367216da: Pull complete 
cfbe7647c924: Pull complete 
c9f0ca38385f: Pull complete 
b79ae26b72bf: Pull complete 
15d994bd38b9: Pull complete 
da14432140a3: Pull complete 
166cdc25bb03: Pull complete 
6214503fae50: Pull complete 
cac2782f6ce2: Pull complete 
4a6a735b5de2: Pull complete 
91a0ae58349d: Pull complete 
462d43e3d83b: Pull complete 
c7144e839e98: Pull complete 
8d03c0b10017: Pull complete 
Digest: sha256:9c60c84582a009a09392a5e09c72a8c871a809401c85297737d937ec79c2b5b0
Status: Downloaded newer image for scilifelablts/lts-workflows-sm-scrnaseq-ci:latest
Adding user 9001
Starting with UID : 9001
user@0dd3d202c5d7:/$ echo $PATH
/opt/miniconda3/envs/snakemake/bin:/opt/miniconda3/envs/snakemake/bin:/usr/local/bin:/opt/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/miniconda3/envs/py2.7/bin:/opt/miniconda3/envs/nextflow/bin
user@0dd3d202c5d7:/$ conda info
Current conda install:

               platform : linux-64
          conda version : 4.2.12
       conda is private : False
      conda-env version : 4.2.12
    conda-build version : not installed
         python version : 3.5.2.final.0
       requests version : 2.11.1
       root environment : /opt/miniconda3  (read only)
    default environment : /opt/miniconda3/envs/snakemake
       envs directories : /home/user/.conda/envs
                          /opt/miniconda3/envs
          package cache : /home/user/.conda/envs/.pkgs
                          /opt/miniconda3/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
           offline mode : False

=> you are using conda 4.2 which has had some problems with noarch: python packages, but is also quite outdated. The newest conda version is 4.4.11, though, from conda-forge you'll only get 4.3.34, but anything >=4.3 should fix your issue, I think.

[rasmusagren]

Wizard indeed, that solved it! Scary bug though, I would never have guessed that it would boil down to Conda version. Thanks a lot! Phil, sorry for scaring you 😄

[mbargull]

Great to hear it's working now!
But no wizardry involved :wink: We encountered the same issue at Bioconda some time ago: https://github.com/bioconda/bioconda-utils/pull/204.

[ewels]

Awesome stuff, thank you @mbargull! I would have never figured this one out..