Rnaseq: problem with singularity

[mhebrard]

Hello,

I try this workflow on local machine using the singularity image.

Step 1

I first pull the image

singularity pull --name nfcore-rnaseq.img docker://nfcore/rnaseq

Minor issue
I notice on the documentation is

singularity pull --name nfcore-rnaseq-1.4.img docker://nfcore/rnaseq:1.4

But the tag 1.4 do not exist in the repository

Step 2

I run the workflow

 nextflow run nf-core/rnaseq --with-singularity nfcore-rnaseq.img --genome GRCm38 --reads 'data/*_R{1,2}*'

Nextflow v0.29.1
nfcore/rnaseq v1.5dev

Major issue

Command error:
  .command.sh: line 2: trim_galore: command not found

Seams like trimgalore is not found... is there an issue with the container ?

[ewels]

Hi @mhebrard,

Thanks for the issue. Apologies for the inconsistencies in the documentation - the pipeline has been ported over from https://github.com/SciLifeLab/NGI-RNAseq relatively recently and the docs need a major review. The nf-core version doesn't yet have a release, hence that tag doesn't exist.

Regarding the second problem, I just checked the container and it seems fine:

$ docker pull nfcore/rnaseq
Using default tag: latest
latest: Pulling from nfcore/rnaseq
c73ab1c6897b: Pull complete
4da7db3cf22b: Pull complete
2eec397d44ab: Pull complete
9c36f5b9109d: Pull complete
2ca921ae0a1d: Pull complete
f62d32b592f4: Pull complete
0cab7ab838e1: Pull complete
84b2897e9494: Pull complete
Digest: sha256:bbb73c43e675c946ca0ce24a08bed0e7e2895f928721c9f09f30cb4d94c42b2d
Status: Downloaded newer image for nfcore/rnaseq:latest

$ docker container run -it nfcore/rnaseq bash

(base) root@a7983f7f295d:/# trim_galore --version

                          Quality-/Adapter-/RRBS-Trimming
                               (powered by Cutadapt)
                                  version 0.4.4_dev

                             Last update: 24 03 2017

Also, running the test data seems to work fine:

$ cd nf-core/rnaseq/tests/

$ bash run_test.sh
Found existing test set, using ./test_data/ngi-rna_test_set
Attempting to use a reference genome from previous run in ./results/reference_genome
Starting nextflow... Command:
nextflow run ../main.nf -resume --aligner star --max_cpus 2 --max_memory '6.GB' --max_time '48.h' -with-docker --gtf ./test_data/ngi-rna_test_set/genes.gtf --bed12 ./test_data/ngi-rna_test_set/genes.bed --star_index results/reference_genome/star/ --singleEnd --reads "./test_data/ngi-rna_test_set/*.fastq.gz"
-----
N E X T F L O W  ~  version 0.30.1
Launching `../main.nf` [mighty_sanger] - revision: 6549eba824
WARN: It seems you never run this project before -- Option `-resume` is ignored
===================================
 nfcore/rnaseq  ~  version 1.5dev
===================================
Run Name       : mighty_sanger
Reads          : ./test_data/ngi-rna_test_set/*.fastq.gz
Data Type      : Single-End
Genome         : false
Strandedness   : None
Trim R1        : 0
Trim R2        : 0
Trim 3' R1     : 0
Trim 3' R2     : 0
Aligner        : STAR
STAR Index     : results/reference_genome/star/
GTF Annotation : ./test_data/ngi-rna_test_set/genes.gtf
BED Annotation : ./test_data/ngi-rna_test_set/genes.bed
Save Reference : No
Save Trimmed   : No
Save Intermeds : No
Max Memory     : 6.GB
Max CPUs       : 2
Max Time       : 48.h
Output dir     : ./results
Working dir    : /Users/ewels/GitHub/nf-core/rnaseq/tests/work
Container      : nfcore/rnaseq:latest
Current home   : /Users/ewels
Current user   : ewels
Current path   : /Users/ewels/GitHub/nf-core/rnaseq/tests
R libraries    : false
Script dir     : /Users/ewels/GitHub/nf-core/rnaseq
Config Profile : standard
=========================================
[warm up] executor > local
[82/b18781] Submitted process > trim_galore (SRR4238379_subsamp)
[59/a6d27d] Submitted process > workflow_summary_mqc
[33/5a86af] Submitted process > get_software_versions

Could you perhaps try deleting and pulling the singularity container again? Sometimes the pull goes wrong and weird stuff happens. You could also try opening a shell in the container and having a look to see if the command can be found.

Note that we have a bunch of updates to do to this pipeline to bring it up to spec for a first release on nf-core. Easier tests are the first thing, and a Singularity build script (and hosting on singularity hub) are a second.

Phil

[mhebrard]

I delete and pull again the image.

$ sudo singularity pull --name nfcore-rnaseq-1.4.img docker://nfcore/rnaseq
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/nfcore/rnaseq:latest
Cache folder set to /root/.singularity/docker
[8/8] |===================================| 100.0%
Importing: base Singularity environment
Exploding layer: sha256:c73ab1c6897bf5c11da3c95cab103e7ca8cf10a6d041eda2ff836f45a40e3d3b.tar.gz
Exploding layer: sha256:4da7db3cf22b4236dc563e571c5d899e086d907aa142e5b9a4a837603a503e48.tar.gz
Exploding layer: sha256:2eec397d44ab9b9b2aa619613a1c0fe6a2d2112c2feae2835e66e461507b6e33.tar.gz
Exploding layer: sha256:9c36f5b9109d83ea36bbc21bfc641a6ba8d9fb49e966747826dc8c0fa8e1eefe.tar.gz
Exploding layer: sha256:2ca921ae0a1dca973892ef9f8013035176f3fb53b90d897d8d69a35887d9578b.tar.gz
Exploding layer: sha256:f62d32b592f411fee4c0719bc9de1ef10cb2ca0bdac3236581ed08deb6596c64.tar.gz
Exploding layer: sha256:0cab7ab838e17a883ec4185e9547424db096ca04d888b16a49bdc28c923ac544.tar.gz
Exploding layer: sha256:84b2897e94941a2ad73ec6537e68e805e178c9b66af247bca4ef4382ed0b660f.tar.gz
Exploding layer: sha256:0ca33e5a88abd0be06e7f303809344267715b89cd619b4e1ec64daf28682a683.tar.gz
Building Singularity image...
Singularity container built: ./nfcore-rnaseq-1.4.img
Cleaning up...
Done. Container is at: ./nfcore-rnaseq-1.4.img

If I enter in the image, there is trim_galore, but no nextflow (as expected)

$ singularity shell nfcore-rnaseq-1.4.img
Singularity: Invoking an interactive shell within container...

Singularity nfcore-rnaseq-1.4.img:/> trim_galore --version

                          Quality-/Adapter-/RRBS-Trimming
                               (powered by Cutadapt)
                                  version 0.4.4_dev

                             Last update: 24 03 2017

Singularity nfcore-rnaseq-1.4.img:~/nf-core> nextflow -v
bash: nextflow: command not found

and outside of the image the pipeline do not run ...

$ nextflow run nf-core/rnaseq --with-singularity nfcore-rnaseq.img --genome GRCm38 --reads 'data/*_R{1,2}*'
N E X T F L O W  ~  version 0.29.1
Launching `nf-core/rnaseq` [tiny_cajal] - revision: 0c73bdeed6 [master]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [5e38b6a6bb]
===================================
 nfcore/rnaseq  ~  version 1.5dev
===================================
Run Name       : tiny_cajal
Reads          : data/*_R{1,2}*
Data Type      : Paired-End
Genome         : GRCm38
Strandedness   : None
Trim R1        : 0
Trim R2        : 0
Trim 3' R1     : 0
Trim 3' R2     : 0
Aligner        : STAR
STAR Index     : s3://ngi-igenomes/igenomes//Mus_musculus/Ensembl/GRCm38/Sequence/STARIndex/
GTF Annotation : s3://ngi-igenomes/igenomes//Mus_musculus/Ensembl/GRCm38/Annotation/Genes/genes.gtf
BED Annotation : s3://ngi-igenomes/igenomes//Mus_musculus/Ensembl/GRCm38/Annotation/Genes/genes.bed
Save Reference : No
Save Trimmed   : No
Save Intermeds : No
Max Memory     : 128 GB
Max CPUs       : 16
Max Time       : 10d
Output dir     : ./results
Working dir    : /home/vagrant/nf-core/work
Container      : nfcore/rnaseq:latest
Pipeline Release: master
Current home   : /home/vagrant
Current user   : vagrant
Current path   : /home/vagrant/nf-core
R libraries    : false
Script dir     : /home/vagrant/.nextflow/assets/nf-core/rnaseq
Config Profile : standard
=========================================
[warm up] executor > local
[b6/382415] Submitted process > fastqc (RMT983-CTTGTA_S6_L001)
[77/ad6f11] Submitted process > trim_galore (RMT983-CTTGTA_S6_L001)
[fd/3528bc] Submitted process > workflow_summary_mqc
[cd/e59080] Submitted process > get_software_versions
[b6/382415] NOTE: Process `fastqc (RMT983-CTTGTA_S6_L001)` terminated with an error exit status (127) -- Error is ignored
ERROR ~ Error executing process > 'trim_galore (RMT983-CTTGTA_S6_L001)'

Caused by:
  Process `trim_galore (RMT983-CTTGTA_S6_L001)` terminated with an error exit status (127)

Command executed:

  trim_galore --paired --fastqc --gzip     RMT983-CTTGTA_S6_L001_R1_001.fastq.gz RMT983-CTTGTA_S6_L001_R2_001.fastq.gz

Command exit status:
  127

Command output:
  (empty)

Command error:
  .command.sh: line 2: trim_galore: command not found
  .command.stub: line 98: 14216 Terminated              nxf_trace "$pid" .command.trace

Work dir:
  /home/vagrant/nf-core/work/77/ad6f11ad3131643e1a819d7a6648f0

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
[cd/e59080] NOTE: Process `get_software_versions` terminated with an error exit status (127) -- Error is ignored
[nfcore/rnaseq] Pipeline Complete
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.

I am not sure how to run the tests...
In addition, I do not have the reference genome yet...

[ewels]

Hi @mhebrard,

I suspect that the problem you're getting is the same as in #33. Steps to fix:

1. Update your copy of the pipeline - it's been updated on a daily basis currently as we work towards an initial release.
2. Add -profile singularity to your nextflow command
   
   
   
   • This hopefully won't be needed soon as I'm working on an update to remove the need for it.
   
   

Phil

[mhebrard]

Hi,
Yep the profile change something.

$ nextflow run nf-core/rnaseq --with-singularity nfcore-rnaseq.img --genome GRCm38 --reads 'data/*_R{1,2}*' -profile singularity
N E X T F L O W  ~  version 0.29.1
Launching `nf-core/rnaseq` [tender_gautier] - revision: 5e38b6a6bb [master]
WARN: Access to undefined parameter `genomes` -- Initialise it to a default value eg. `params.genomes = some_value`
ERROR ~ Cannot get property 'GRCm38' on null object

 -- Check script 'main.nf' at line: 85 or see '.nextflow.log' file for more details

Must be a minor typo in the profile "genomes" with s ...

Courage with the release !

[ewels]

Apologies, my fault - you need -profile standard,singularity. That should do it...

[mhebrard]

Oh I should have read more carefully. I could have guess that...

I updated nextflow to v0.30.2
I pulled nf-core/rnaseq on 20/06/18
and run the command

 sudo nextflow run nf-core/rnaseq --with-singularity nfcore-rnaseq.img -profile standard,singularity --genome GRCm38 --reads 'data/*_R{1,2}*'

First issue:
the option --with-singularity looks broken. I get the message

WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /home/vagrant/nf-core/work/singularity
Pulling Singularity image docker://nfcore/rnaseq:latest

Second issue:
I am worry the path for reference genome will failed as I am in local machine but get the path for s3

STAR Index     : s3://ngi-igenomes/igenomes//Mus_musculus/...

Third issue:
Then the first wave of process are launched, but failed...

Full log:

N E X T F L O W  ~  version 0.30.2
Launching `nf-core/rnaseq` [adoring_hamilton] - revision: 5e38b6a6bb [master]
WARN: Access to undefined parameter `readPaths` -- Initialise it to a default value eg. `params.readPaths = some_value`
=======================================================
                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~'
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

 nf-core/rnaseq : RNA-Seq Best Practice v1.5dev
=======================================================
Run Name       : adoring_hamilton
Reads          : data/*_R{1,2}*
Data Type      : Paired-End
Genome         : GRCm38
Strandedness   : None
Trim R1        : 0
Trim R2        : 0
Trim 3' R1     : 0
Trim 3' R2     : 0
Aligner        : STAR
STAR Index     : s3://ngi-igenomes/igenomes//Mus_musculus/Ensembl/GRCm38/Sequence/STARIndex/
GTF Annotation : s3://ngi-igenomes/igenomes//Mus_musculus/Ensembl/GRCm38/Annotation/Genes/genes.gtf
BED Annotation : s3://ngi-igenomes/igenomes//Mus_musculus/Ensembl/GRCm38/Annotation/Genes/genes.bed
Save Reference : No
Save Trimmed   : No
Save Intermeds : No
Max Memory     : 128 GB
Max CPUs       : 16
Max Time       : 10d
Output dir     : ./results
Working dir    : /home/vagrant/nf-core/work
Container      : nfcore/rnaseq:latest
Pipeline Release: master
Current home   : /home/vagrant
Current user   : root
Current path   : /home/vagrant/nf-core
Script dir     : /home/vagrant/.nextflow/assets/nf-core/rnaseq
Config Profile : standard,singularity
=========================================
[warm up] executor > local
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /home/vagrant/nf-core/work/singularity
Pulling Singularity image docker://nfcore/rnaseq:latest [cache /home/vagrant/nf-core/work/singularity/nfcore-rnaseq-latest.img]
[60/c2e221] Submitted process > workflow_summary_mqc
[81/85f569] Submitted process > get_software_versions
[18/f3a68d] Submitted process > fastqc (RMT983-CTTGTA_S6_L001)
[7f/106ff4] Submitted process > trim_galore (RMT983-CTTGTA_S6_L001)
[81/85f569] NOTE: Process `get_software_versions` terminated with an error exit status (127) -- Error is ignored
[18/f3a68d] NOTE: Process `fastqc (RMT983-CTTGTA_S6_L001)` terminated with an error exit status (127) -- Error is ignored
ERROR ~ Error executing process > 'trim_galore (RMT983-CTTGTA_S6_L001)'

Caused by:
  Process `trim_galore (RMT983-CTTGTA_S6_L001)` terminated with an error exit status (127)

Command executed:

  trim_galore --paired --fastqc --gzip     RMT983-CTTGTA_S6_L001_R1_001.fastq.gz RMT983-CTTGTA_S6_L001_R2_001.fastq.gz

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /home/vagrant/nf-core/work/7f/106ff4349ccaf517c4eaf4d6393205: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Work dir:
  /home/vagrant/nf-core/work/7f/106ff4349ccaf517c4eaf4d6393205

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

 -- Check '.nextflow.log' file for details
[nfcore/rnaseq] Pipeline Complete

Courage

[ewels]

the option --with-singularity looks broken. I get the message

This option uses one hyphen: -with-singularity. Because of this, it didn't use the local singularity image that you provided and instead downloaded it for you.

I am worry the path for reference genome will failed as I am in local machine but get the path for s3

When you specify --genome GRCm38 you're telling the pipeline to try to use reference indexes for alignment from that reference. The default paths are on s3, so that it can run without any local requirements. It's entirely possible to use a local copy of the required genome references though, see the documentation for instructions.

Then the first wave of process are launched, but failed...

Command error:
  /bin/bash: line 0: cd: /home/vagrant/nf-core/work/7f/106ff4349ccaf517c4eaf4d6393205: No such file or directory
  /bin/bash: .command.stub: No such file or directory

This one is tougher. File not found errors like this are usually because of problems with the way that Singularity is installed and configured. In short, the easiest way to do this is to enable overlays and hosted file systems. You can do this by editing the singularity.conf file:

# MOUNT HOSTFS: [BOOL]
# DEFAULT: no
# Probe for all mounted file systems that are mounted on the host, and bind
# those into the container?
mount hostfs = yes

# ENABLE OVERLAY: [yes/no/try]
# DEFAULT: try
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container.  If 'try' is chosen,
# overlayfs will be tried but if it is unavailable it will be silently ignored.
enable overlay = yes

If you're running on a HPC, there's a good chance that you'll need to talk to your sysadmin about doing this.

An alternative is to create a "child" singularity container based on the one that we have made and you can manually add in base directories to create file system mount points. This is a bit of a pain, and we have plans to write a tool to automate this process for people in the future.

I hope this makes sense! Let us know if we can help.

Phil

[mhebrard]

Hey !

No such file or directory

That was an issue with my installation of nextflow and interference with singularity.
In short, I needed sudo to run nextflow, and the /home of sudo that was mount within singularity didn't contain my data ...

I installed properly nextflow to be able to run it as simple user. Then the (right) file system is now mounted within singularity image.

This option uses one hyphen: -with-singularity.

I didn't saw that update. I will use one hyphen from now.

Working commands

singularity pull --name nfcore-rnaseq.img docker://nfcore/rnaseq

nextflow run nf-core/rnaseq 
-with-singularity nfcore-rnaseq.img 
-profile standard,singularity 
--genome GRCm38 
--reads 'data/*_R{1,2}*'

The first wave of process are running now !!

(I let you know if I have other issues)

[ewels]

Great! Thanks for letting us know :)

[efratushava]

This one is tougher. File not found errors like this are usually because of problems with the way that Singularity is installed and configured. In short, the easiest way to do this is to enable overlays and hosted file systems. You can do this by editing the singularity.conf file:

# MOUNT HOSTFS: [BOOL]
# DEFAULT: no
# Probe for all mounted file systems that are mounted on the host, and bind
# those into the container?
mount hostfs = yes

# ENABLE OVERLAY: [yes/no/try]
# DEFAULT: try
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container.  If 'try' is chosen,
# overlayfs will be tried but if it is unavailable it will be silently ignored.
enable overlay = yes

If you're running on a HPC, there's a good chance that you'll need to talk to your sysadmin about doing this.

An alternative is to create a "child" singularity container based on the one that we have made and you can manually add in base directories to create file system mount points. This is a bit of a pain, and we have plans to write a tool to automate this process for people in the future.

I hope this makes sense! Let us know if we can help.

Phil

What do you mean "create a "child" singularity container"? How can I run singularity inside singularity? And how to add the base directory?
Have you automated this process?

Thanks!

[drpatelh]

Is this not something that can be fixed by enabling autoMounts via Nextflow. Newer versions of pipelines are activating this by default so it shouldnt be a problem:
https://github.com/nf-core/rnaseq/blob/3b6df9bd104927298fcdf69e97cca7ff1f80527c/nextflow.config#L125

@efratushava Is this something you have observed with the latest release of the rnaseq pipeline?

[efratushava]

Is this not something that can be fixed by enabling autoMounts via Nextflow. Newer versions of pipelines are activating this by default so it shouldnt be a problem:
https://github.com/nf-core/rnaseq/blob/3b6df9bd104927298fcdf69e97cca7ff1f80527c/nextflow.config#L125

@efratushava Is this something you have observed with the latest release of the rnaseq pipeline?

Yes. It was actually with a different pipeline, but even when extracting only the fastqc process to a differrent file, as follows, it didn't work:

params.outdir = './results_try'
params.single_end = false
params.reads =  false 



/*
 * Create a channel for input read files
 */
if (params.readPaths) {
    if (params.single_end) {
        Channel
            .from(params.readPaths)
            .map { row -> [ row[0], [ file(row[1][0], checkIfExists: true) ] ] }
            .ifEmpty { exit 1, "params.readPaths was empty - no input files supplied" }
            .into { ch_read_files_for_fastqc; ch_read_files_for_trim_galore }
    } else {
        Channel
            .from(params.readPaths)
            .map { row -> [ row[0], [ file(row[1][0], checkIfExists: true), file(row[1][1], checkIfExists: true) ] ] }
            .ifEmpty { exit 1, "params.readPaths was empty - no input files supplied" }
            .into { ch_read_files_for_fastqc; ch_read_files_for_trim_galore }
    }
} else {
    Channel
        .fromFilePairs( params.reads, size: params.single_end ? 1 : 2 )
        .ifEmpty { exit 1, "Cannot find any reads matching: ${params.reads}\nNB: Path needs to be enclosed in quotes!\nIf this is single-end data, please specify --single_end on the command line." }
        .into { ch_read_files_for_fastqc; ch_read_files_for_trim_galore }
}



/*
 * STEP 1 - FastQC
 */
process fastqc {
    tag "$name"
    label 'process_medium'
    publishDir "${params.outdir}/fastqc", mode: 'copy',
        saveAs: { filename ->
                      filename.indexOf(".zip") > 0 ? "zips/$filename" : "$filename"
                }

    input:
    set val(name), file(reads) from ch_read_files_for_fastqc

    output:
    file '*_fastqc.{zip,html}' into ch_fastqc_results_for_multiqc

    script:
    """
    fastqc --quiet --threads 1 $reads
    """
}

Using this config:

process.container = 'nfcore/methylseq:1.4'

profiles {
 singularity {
      singularity.enabled = true
      singularity.autoMounts = true
  }
}

The command:
nextflow -log methylseq_try run try/try_fastqc.nf --reads 'data/*sorted_{1,2}.fastq' -with-singularity /home/vagrant/nfcore-methylseq.img

And I don't have root permission to change the singularity.conf file

[drpatelh]

Ok. So Im assuming you are using the latest version of that pipeline too? and running the command below fails?

nextflow run nf-core/methylseq -profile test,singularity -with-singularity /home/vagrant/nfcore-methylseq.img

autoMounts should be activated there by default too so you shouldnt need to provide another config.

[apeltzer]

Where are these reads stored? That sounds a bit like a missing bind path in your Singularity Configuration on the cluster you're working on... https://nf-co.re/usage/troubleshooting#cannot-find-input-files-when-using-singularity

[efratushava]

I'm so so sorry! I think it's working now...
Thank you!

[drpatelh]

Just out of interest, how did you get it working?

[efratushava]

I just didn't noticed that the result outdir was created...