Rnaseq: Issue running rnaseq v2.0 (DSL2) with test profile

[ggabernet]

Hi @drpatelh,

I've tested the test profile for the dev branch of rnaseq in our cluster. We use singularity there as we cannot use docker due to security issues. Here is the command I've used for the tests:

#!/bin/bash
module load devel/singularity/3.4.2
nextflow pull nf-core/rnaseq -r dev
nextflow run nf-core/rnaseq -r dev -profile test,cfc_dev --pull_docker_container

Nextflow version: nextflow version 20.10.0.5430

Here is the error I am getting:

Error executing process > 'RNASEQ:PREPARE_GENOME:GTF2BED (genome_gfp.gtf)'

Caused by:
  Process `RNASEQ:PREPARE_GENOME:GTF2BED (genome_gfp.gtf)` terminated with an error exit status (126)

Command executed:

  gtf2bed genome_gfp.gtf > genome_gfp.bed

Command exit status:
  126

Command output:
  (empty)

Command error:
  .command.sh: /home-link/qeaga01/.nextflow/assets/nf-core/rnaseq/bin/gtf2bed: /usr/bin/perl: bad interpreter: No such file or directory

Work dir:
  /sfs/7/workspace/ws/qeaga01-rnaseq_dsl2-0/work/3b/8f3201faa3890d078fb87be9c6f79c

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

[ggabernet]

Should the --pull_docker_container option be used for singularity or not by default?

[ggabernet]

For more info, when I don't use the option to pull docker containers I also get errors. Command executed:

#!/bin/bash
module load devel/singularity/3.4.2
nextflow pull nf-core/rnaseq -r dev
nextflow run nf-core/rnaseq -r dev -profile test,cfc_dev

Error:

Error executing process > 'RNASEQ:QUANTIFY_SALMON:UNTAR (salmon.tar.gz)'

Caused by:
  Process `RNASEQ:QUANTIFY_SALMON:UNTAR (salmon.tar.gz)` terminated with an error exit status (255)

Command executed:

  tar -xzvf  salmon.tar.gz
  echo $(tar --version 2>&1) | sed 's/^.*(GNU tar) //; s/ Copyright.*$//' > untar.version.txt

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   container creation failed: mount /proc/self/fd/16-><path>/rootfs error: can't mount image /proc/self/fd/16: kernel reported a bad superblock for squashfs image partition, possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted

Work dir:
  /sfs/7/workspace/ws/qeaga01-rnaseq_dsl2-0/work/e2/c89ea5f1034c37a60e1dede372b4c4

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

[FriederikeHanssen]

Ran on the same cluster as @ggabernet:

Nextflow version: 20.07.1
Singularity: 3.4.2

nextflow run nf-core/rnaseq -r dev -profile test,cfc_dev --pull_docker_container

Error:

Error executing process > 'RNASEQ:QUANTIFY_SALMON:SALMON_TX2GENE (genome_gfp.gtf)'
Caused by:
  Process `RNASEQ:QUANTIFY_SALMON:SALMON_TX2GENE (genome_gfp.gtf)` terminated with an error exit status (127)
Command executed:
  salmon_tx2gene.py \
      --gtf genome_gfp.gtf \
      --salmon salmon \
      --id gene_id \
      --extra gene_name \
      -o salmon_tx2gene.tsv
Command exit status:
  127
Command output:
  (empty)
Command error:
  .command.sh: line 3: salmon_tx2gene.py: command not found

Command:

nextflow run nf-core/rnaseq -r dev -profile test,cfc_dev

Error:

Error executing process > 'RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)'

Caused by:
  Process `RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)` terminated with an error exit status (255)

Command executed:

  gunzip --force  genes.gtf.gz
  echo $(gunzip --version 2>&1) | sed 's/^.*(gzip) //; s/ Copyright.*$//' > gunzip.version.txt

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   container creation failed: mount /proc/self/fd/16->/sfs/4/sfs7/software/bwhpc/common/devel/singularity/3.4.2/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/16: kernel reported a bad superblock for squashfs image partition, possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted

[drpatelh]

Ok. Try this guys:

## (1) Download an archive containing all of the containers for the pipeline
wget -L https://dl.dropboxusercontent.com/s/1os36hmd73u0ygf/nf-core-rnaseq_v2.0.tar.gz
wget -L https://dl.dropboxusercontent.com/s/7vxsiyhz1kch36k/nf-core-rnaseq_v2.0.tar.gz.md5

## (2) Check tar archive was downloaded ok
md5sum -c nf-core-rnaseq_v2.0.tar.gz.md5

## (3) Untar archive and move this somewhere where the containers can be re-used for future pipeline runs
tar -xvf nf-core-rnaseq_v2.0.tar.gz

## (4) Load any modules you require on your local infrastructure
module purge
module load devel/singularity/3.4.2

## (5) Export this variable so Nextflow finds the containers you downloaded
export NXF_SINGULARITY_CACHEDIR=/PATH/TO/UNTAR/ARCHIVE/nf-core-rnaseq_v2.0/

## (6) Update your local cached version of the pipeline pipeline if you haven't done that recently
nextflow pull nf-core/rnaseq

## (7) Remove any files left-over from previous runs of Nextflow in the same directory
rm -r results/ work/ .nextflow*

## (8) Run the test profile
nextflow run nf-core/rnaseq -profile test,cfc_dev -r 2.0

This will at least rule out any issues with pulling the containers.

[ggabernet]

Ok, we already set our singularity cachedir in our cluster config profile, so won't need that step, but will try the rest.

[apeltzer]

Same issue here - just to confirm this can be reproduced using Singularity 3.4.2, with Nextflow 20.07.1 - checking solution now

[drpatelh]

Could be related to this issue. Apparently it is fixed in Singularity v3.5.1.

[drpatelh]

Thanks guys! Seeing as we have removed the --pull_docker_parameter and direct download of Singularity images I will close this issue. If you are still having problems with version 2.0 then please open another issue or reach out on Slack and we can try and get the pipeline running for you!

[ggabernet]

Hi @drpatelh ,

just tested this again. Manually downloading the containers before running the pipeline works, but of course this is not an ideal setting. Without manually downloading the containers in advance, we still get errors, also with the release v2.0:

module purge
module load devel/singularity/3.4.2
nextflow pull nf-core/rnaseq -r 2.0
nextflow run nf-core/rnaseq -r 2.0 -profile test,cfc

Nextflow version: nextflow version 20.10.0.5430

Error executing process > 'RNASEQ:SUBREAD_FEATURECOUNTS (WT_R2)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name quay.io-biocontainers-subread-2.0.1--hed695b0_0.img.pulling.1605260349454 docker://quay.io/biocontainers/subread:2.0.1--hed695b0_0 > /dev/null
  status : 255
  message:
    [34mINFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Skipping fetch of repeat blob sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7
    Skipping fetch of repeat blob sha256:cfb1ba34637d96787f6e780192bc5f09c04fe0b40c4e1acdcbc88953bce25b5b
    Skipping fetch of repeat blob sha256:ace2d8a63dd59063206661355f22da9e1c75c56d6cae5116d91db7a65f640667
    Skipping fetch of repeat blob sha256:75c080ef15ebb28daeadc91eeda0ca95a893fe1f4e55845da3feb58069d42fdc
    Skipping fetch of repeat blob sha256:316957f8baafcefd4c9f72b7e3f3fafb7fcb3ca805dc69d242aef656e0736490
    Skipping fetch of repeat blob sha256:dbd31e1d863d4a41565c08ad1d419c3a3538abfdd4a7971de63b8c5c8a43caec
    Skipping fetch of repeat blob sha256:2f8531d5a6ece34772cf99a2d30e3057c636f1ad05cdec0c8009eab401e616a7
    Skipping fetch of repeat blob sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7
    Skipping fetch of repeat blob sha256:2e178fd72baf28bc587b5d274574f1a3d3e25ac802fdaf154af258a3c5891831
    Copying blob sha256:2912e793bf3d1505b5d6f84f00b3e1f7c93c9aca610730896f448a89e9736a3d

     0 B / 29.76 MiB [-------------------------------------------------------------]
     1.73 MiB / 29.76 MiB [===>----------------------------------------------------]
     20.33 MiB / 29.76 MiB [=====================================>-----------------]
     29.76 MiB / 29.76 MiB [====================================================] 0s
    Copying config sha256:99094e35636901d0eebf3e0bb9acfc83007a0748982d102232c661889dcd3ce3

     0 B / 2.81 KiB [--------------------------------------------------------------]
     2.81 KiB / 2.81 KiB [======================================================] 0s
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: while building SIF from layers: conveyor failed to get: no descriptor found for reference "ccee1f6ebb924fd0b3b6db646a51dabc905697aa25be132386dd490c2318286c"

[drpatelh]

Hi @ggabernet! Yep I suspected it would still break 😅 I closed this issue because it had been muddied by our failed attempts to directly download Singularity images and thought it would be a good idea to start fresh.

So you got the pipeline to work when you download the containers beforehand? and it fails when you try to download them directly via the pipeline? To me that suggests its either an issue pulling the containers from quay.io or maybe in Nextflow. However, given that Nextflow is failing whilst running:

singularity pull  --name quay.io-biocontainers-subread-2.0.1--hed695b0_0.img.pulling.1605260349454 docker://quay.io/biocontainers/subread:2.0.1--hed695b0_0 > /dev/null

I would possibly point the finger at download issues via quay.io. Not sure how we can follow this up to resolve the issue. Suggestions welcome though 🙂

Does the pipeline eventually complete after multiple -resume?

[ggabernet]

yes, even after resuming several times it seems to have a problem with the subread-2.0.1 container:

Error executing process > 'RNASEQ:SUBREAD_FEATURECOUNTS (RAP1_IAA_30M_R1)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name quay.io-biocontainers-subread-2.0.1--hed695b0_0.img.pulling.1605262146587 docker://quay.io/biocontainers/subread:2.0.1--hed695b0_0 > /dev/null
  status : 255
  message:
    [34mINFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Skipping fetch of repeat blob sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7
    Skipping fetch of repeat blob sha256:cfb1ba34637d96787f6e780192bc5f09c04fe0b40c4e1acdcbc88953bce25b5b
    Skipping fetch of repeat blob sha256:ace2d8a63dd59063206661355f22da9e1c75c56d6cae5116d91db7a65f640667
    Skipping fetch of repeat blob sha256:75c080ef15ebb28daeadc91eeda0ca95a893fe1f4e55845da3feb58069d42fdc
    Skipping fetch of repeat blob sha256:316957f8baafcefd4c9f72b7e3f3fafb7fcb3ca805dc69d242aef656e0736490
    Skipping fetch of repeat blob sha256:dbd31e1d863d4a41565c08ad1d419c3a3538abfdd4a7971de63b8c5c8a43caec
    Skipping fetch of repeat blob sha256:2f8531d5a6ece34772cf99a2d30e3057c636f1ad05cdec0c8009eab401e616a7
    Skipping fetch of repeat blob sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7
    Skipping fetch of repeat blob sha256:2e178fd72baf28bc587b5d274574f1a3d3e25ac802fdaf154af258a3c5891831
    Skipping fetch of repeat blob sha256:2912e793bf3d1505b5d6f84f00b3e1f7c93c9aca610730896f448a89e9736a3d
    Copying config sha256:99094e35636901d0eebf3e0bb9acfc83007a0748982d102232c661889dcd3ce3

     0 B / 2.81 KiB [--------------------------------------------------------------]
     2.81 KiB / 2.81 KiB [======================================================] 0s
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: while building SIF from layers: conveyor failed to get: no descriptor found for reference "ccee1f6ebb924fd0b3b6db646a51dabc905697aa25be132386dd490c2318286c"

[ggabernet]

After a couple of resumes, now it is a different one:

Error executing process > 'RNASEQ:STRINGTIE (RAP1_IAA_30M_R1)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name quay.io-biocontainers-stringtie-2.1.4--h7e0af3c_0.img.pulling.1605262365802 docker://quay.io/biocontainers/stringtie:2.1.4--h7e0af3c_0 > /dev/null
  status : 255
  message:
    [34mINFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Skipping fetch of repeat blob sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7
    Skipping fetch of repeat blob sha256:cfb1ba34637d96787f6e780192bc5f09c04fe0b40c4e1acdcbc88953bce25b5b
    Skipping fetch of repeat blob sha256:ace2d8a63dd59063206661355f22da9e1c75c56d6cae5116d91db7a65f640667
    Skipping fetch of repeat blob sha256:75c080ef15ebb28daeadc91eeda0ca95a893fe1f4e55845da3feb58069d42fdc
    Skipping fetch of repeat blob sha256:316957f8baafcefd4c9f72b7e3f3fafb7fcb3ca805dc69d242aef656e0736490
    Skipping fetch of repeat blob sha256:dbd31e1d863d4a41565c08ad1d419c3a3538abfdd4a7971de63b8c5c8a43caec
    Skipping fetch of repeat blob sha256:2f8531d5a6ece34772cf99a2d30e3057c636f1ad05cdec0c8009eab401e616a7
    Skipping fetch of repeat blob sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7
    Skipping fetch of repeat blob sha256:2e178fd72baf28bc587b5d274574f1a3d3e25ac802fdaf154af258a3c5891831
    Skipping fetch of repeat blob sha256:199f4f2209fe1fb72e01334848790fdd325ca26d3e201f9b3bded49610a46b63
    Copying config sha256:7afab82913894e12089bd109ff7b9966aeb49a6b6f11bd04dac2d493ec2b27b9

     0 B / 2.81 KiB [--------------------------------------------------------------]
     2.81 KiB / 2.81 KiB [======================================================] 0s
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: while building SIF from layers: conveyor failed to get: no descriptor found for reference "5c8644de00dd867c6560dfbe4f662f288ce479f618f53a2f5b0888d5274dbc9c"

So yes, this all points to download issues of the containers from quay.io

[ggabernet]

After investigating a bit the Singularity errors, it seems like the problem is when pulling the same container at the same time in two different processes.

This seems relevant: https://github.com/hpcng/singularity/issues/3634

And this too: https://github.com/hpcng/singularity/issues/5020

According to the last issue, this should be fixed in Singularity 3.6. I will test that and report back!

[drpatelh]

Ahhhh, but Nextflow should only be pulling that container once via the master job to avoid exactly that? 🤔

[drpatelh]

Think its still worth trying with a more recent version of Singularity though because it may well be an issue with the pull command 👍

[ggabernet]

There was also an issue in Nextflow about that: https://github.com/nextflow-io/nextflow/issues/1210

[apeltzer]

One of the suggested fixes was also to set --disable-cache as a Singularity option - you could try overwriting this in a custom config and see if this fixes the issue?

singularity.engineOptions = '--disable-cache' 

In the worst case, this will overwrite the image 1-2x until other processes afterwards will simply reuse the downloaded image I think 🤔 Not ideal, but maybe a solution until everyone has 3.6 ;-)

And yes, seems to be fine with Singularity >= 3.6

[drpatelh]

Nice! I will ask our HPC admins to install the latest version of Singularity! Did you manage to get it working @ggabernet ?

[ggabernet]

Unfortunately not, @drpatelh . After upgrading to singularity version 3.6.2-1.el7 still having issues pulling the containers:

Error executing process > 'RNASEQ:QUANTIFY_SALMON:SALMON_SE_TRANSCRIPT (salmon_tx2gene.tsv)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name quay.io-biocontainers-bioconductor-summarizedexperiment-1.18.1--r40_0.img.pulling.1605268292414 docker://quay.io/biocontainers/bioconductor-summarizedexperiment:1.18.1--r40_0 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500
    Copying blob sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6
    Copying blob sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88
    Copying blob sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7
    Copying blob sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053
    Copying blob sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c
    Copying blob sha256:f981c3bfe61f7355e034d40b620e60aefc6b272a8d0ac10fa9e1892bb6b17b56
    Copying config sha256:ff870dedc9d11d9622344d7a4ff0c0c25a890f2233a84926b6cb0e67f422500e
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: Error initializing source oci:/home-link/<username>/.singularity/cache/blob:70c154f9aee9152d9e03c474cd4b5e5eee5856cda5b62c46b10c4ae7932e763d: no descriptor found for reference "70c154f9aee9152d9e03c474cd4b5e5eee5856cda5b62c46b10c4ae7932e763d"

java.lang.IllegalStateException: java.lang.IllegalStateException: Failed to pull singularity image
  command: singularity pull  --name quay.io-biocontainers-bioconductor-summarizedexperiment-1.18.1--r40_0.img.pulling.1605268292414 docker://quay.io/biocontainers/bioconductor-summarizedexperiment:1.18.1--r40_0 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500
    Copying blob sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6
    Copying blob sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88
    Copying blob sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7
    Copying blob sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053
    Copying blob sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0
    Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
    Copying blob sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c
    Copying blob sha256:f981c3bfe61f7355e034d40b620e60aefc6b272a8d0ac10fa9e1892bb6b17b56
    Copying config sha256:ff870dedc9d11d9622344d7a4ff0c0c25a890f2233a84926b6cb0e67f422500e
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: Error initializing source oci:/home-link/qeaga01/.singularity/cache/blob:70c154f9aee9152d9e03c474cd4b5e5eee5856cda5b62c46b10c4ae7932e763d: no descriptor found for reference "70c154f9aee9152d9e03c474cd4b5e5eee5856cda5b62c46b10c4ae7932e763d"

[ggabernet]

Will investigate a bit further and let you know if I find a solution

[drpatelh]

😞 Thanks! Did you try the config setting that @apeltzer recommended too?

singularity.engineOptions = '--disable-cache' 

[ggabernet]

Just tested also this singularity.engineOptions = '--disable-cache' in a custom config. It does not seem to be recognized. I get this:

WARN: Tower request field `workflow.errorMessage` exceeds expected size | offending value: `Error for command "s
ingularity": unknown flag: --disable-cache

Options for singularity command:

  -d, --debug     print debugging information (highest verbosity)
  -h, --help      help for singularity
      --nocolor   print without color output (default False)
  -q, --quiet     suppress normal output
  -s, --silent    only print errors
  -v, --verbose   print additional information
      --version   version for singularity

Run 'singularity --help' for more detailed usage information.`, size: 492 (max: 255)
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Error executing process > 'RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet.csv)'

Caused by:
  Process `RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet.csv)` terminated with an error exit status (1)

Command executed:

  check_samplesheet.py samplesheet.csv samplesheet.valid.csv

Command exit status:
  1

Command output:
  (empty)

Command error:
  Error for command "singularity": unknown flag: --disable-cache

My custom config contains this for singularity:

singularity {
  enabled = true
  engineOptions = '--disable-cache'
}

@apeltzer is this supposed to go to this Nextflow custom config, or somewhere else in the Singularity configuration?

[apeltzer]

Not sure either - maybe @pditommaso knows ?

[pditommaso]

Nope, singularity.engineOptions is only for options you can specify immediately after singularity. Is --disable-cache and option for pull ?

Have you tried to run the pull command to check why is failing?

singularity pull  --name quay.io-biocontainers-bioconductor-summarizedexperiment-1.18.1--r40_0.img.pulling.1605268292414 docker://quay.io/biocontainers/bioconductor-summarizedexperiment:1.18.1--r40_0 

[drpatelh]

This should be fixed in https://github.com/nf-core/rnaseq/pull/531. @ggabernet @FriederikeHanssen would you mind re-trying to run the dev version of the pipeline once this is merged please? You will need to have Nextflow >=20.11.0-edge installed.

[drpatelh]

A number of users have reported that this is now working by using Nextflow >= 20.11.0-edge and the direct download of Singularity containers added to the pipeline in https://github.com/nf-core/rnaseq/pull/531. Is this ok to close now @ggabernet ?

[ggabernet]

Hi @drpatelh ,
yes I can confirm that the dev branch tests pass now on our cluster with Singularity and Nextflow >= 20.11.0-edge, thanks a lot for fixing this!