Tools: Can't change igenomes_base in a custom config file

[ewels]

Description of bug

The iGenomes config files use params.igenomes_base to make it possible to use a local copy of iGenomes:

https://github.com/nf-core/tools/blob/9e33e7f02758964a1a09436af1eb6afe0e596e4c/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/conf/igenomes.config#L14

This works great when using a configuration _profile_ - the config profiles are loaded before the iGenomes config file is imported _(note line numbers)_:

https://github.com/nf-core/tools/blob/9e33e7f02758964a1a09436af1eb6afe0e596e4c/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/nextflow.config#L51-L56

https://github.com/nf-core/tools/blob/9e33e7f02758964a1a09436af1eb6afe0e596e4c/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/nextflow.config#L76-L79

However, if you use a custom configuration file, this is loaded after the pipeline nextflow.config. This means that igenomes.config has already been imported and parsed, so changing params.igenomes_base has no effect and the pipeline tries to use the s3 locations.

@adrlar had a nice worked example of this for nf-core/sarek on slack:

If I run:

nextflow -c icr.config config /mnt/scratch/DDS/SDDS/BIOINFCO/SHARED/software/sarek-2.5.2

with the config specifying only:

params {
   igenomes_base = '/mnt/scratch/DDS/SDDS/BIOINFCO/SHARED/genomes/igenomes'
   // ...

I get:

params {
   // ...
   igenomes_base = '/mnt/scratch/DDS/SDDS/BIOINFCO/SHARED/genomes/igenomes'
   // ...
   genomes {
      GRCh37 {
         acLoci = 's3://ngi-igenomes/igenomes//Homo_sapiens/GATK/GRCh37/Annotation/ASCAT/1000G_phase3_20130502_SNP_maf0.3.loci'
         acLociGC = 's3://ngi-igenomes/igenomes//Homo_sapiens/GATK/GRCh37/Annotation/ASCAT/1000G_phase3_20130502_SNP_maf0.3.loci.gc'
         bwaIndex = 's3://ngi-igenomes/igenomes//Homo_sapiens/GATK/GRCh37/Sequence/BWAIndex/human_g1k_v37_decoy.fasta.{amb,ann,bwt,pac,sa}'

Proposed fix

I think it should be possible to fix this by deferring the execution of the iGenomes params paths, by surrounding them in squiggly brackets ({}).

However, I have a feeling that we used to do this and removed them for some reason. I just can't remember what that was.

Any issues with adding these back? Would be great if someone has some time to test this 👍

[ibwoo]

For anybody wanting a quick workaround, --igenomes_base /path/to/igenomes/ works for me.

[adrlar]

The --igenomes_base command line param does indeed work to set the base before any config file gets a chance.

[ewels]

Ok, that's good - at least the command line option takes preference and is parsed in time.

I definitely have deja-vu with this one, I'm sure we've dealt with it before. Can anyone remember why we aren't using squiggly brackets @nf-core/core ?

[apeltzer]

Can't remember unfortunately, sorry :-(

[drpatelh]

This looks like the oldest commit for that file in the nf-core/tools repo? No {{ }}'s 😕
https://github.com/nf-core/tools/blob/c2d6d4a81f15428a2cfdcce3d0140c3df98c8f99/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/conf/igenomes.config

[ewels]

Challenge accepted! Predecessor was https://github.com/nf-core/cookiecutter (which used to be called SciLifeLab/NGI-NFcookiecutter).

And yes, it seems that the initial commit for that repo also had no squiggly brackets: https://github.com/nf-core/cookiecutter/blob/f871965a6fd038c82ecb0e841531dfd003b2cecf/conf/igenomes.config

But before _that_ repository existed, it was developed as part of SciLifeLab/NGI-RNAseq. And finally I find my squiggly brackets: https://github.com/SciLifeLab/NGI-RNAseq/commit/483a3bef2787af267052daadb8f2e1d108bd0cc8 (7 Sep 2017).

But, weirdly I added the squiggly brackets and then immediately took them away again within the same pull-request: https://github.com/SciLifeLab/NGI-RNAseq/pull/152

I wonder if there was some discussion / comments that went missing from this PR or something. @Hammarn - is your memory any better than mine??

Maybe we should just try adding them in and seeing what happens 😅

[Hammarn]

@ewels it usually is, at least I tend to remember to have lunch ;)
Eh, all I remember from this is that there was a lot of changing back and forth to try and get things to work properly. 🤷‍♂

[lconde-ucl]

Hi, I tried this with the rnaseq pipeline and it does not work for me.

Reproducing the bug described above

Specifically, this is the original igenomes.config:

[regmond@login12 nf-core_rnaseq]$ more ~/.nextflow/assets/nf-core/rnaseq/conf/igenomes.config     
/*
 * -------------------------------------------------
 *  Nextflow config file for iGenomes paths
 * -------------------------------------------------
 * Defines reference genomes, using iGenome paths
 * Can be used by any config that customises the base
 * path using $params.igenomes_base / --igenomes_base
 */

params {
  // illumina iGenomes reference file paths
  genomes {
    'GRCh37' {
      fasta       = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/WholeGenomeFasta/genome.fa"
      bwa         = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/BWAIndex/genome.fa"
      bowtie2     = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/Bowtie2Index/"
      star        = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/STARIndex/"
      bismark     = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/BismarkIndex/"
      gtf         = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.gtf"
      bed12       = "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.bed"
      mito_name   = "MT"
      macs_gsize  = "2.7e9"
      blacklist   = "${baseDir}/assets/blacklists/GRCh37-blacklist.bed"
    }
[...]

When I run it with my own config, it does not pick up the igenomes that I specified:

----------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/rnaseq v1.4.2
----------------------------------------------------
Pipeline Release  : master
Run Name          : agitated_swirles
Reads             : fastq_files/*{1,2}_001.fastq.gz
Data Type         : Paired-End
Genome            : GRCh37
Strandedness      : None
Trimming          : 5'R1: 0 / 5'R2: 0 / 3'R1: 0 / 3'R2: 0 / NextSeq Trim: 0
Aligner           : STAR
STAR Index        : s3://ngi-igenomes/igenomes//Homo_sapiens/Ensembl/GRCh37/Sequence/STARIndex/
GTF Annotation    : s3://ngi-igenomes/igenomes//Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.gtf
BED Annotation    : s3://ngi-igenomes/igenomes//Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.bed
[...]

---

Then I changed the config to add {}'s:

[regmond@login12 nf-core_rnaseq]$ more ~/.nextflow/assets/nf-core/rnaseq/conf/igenomes.config     
/*
 * -------------------------------------------------
 *  Nextflow config file for iGenomes paths
 * -------------------------------------------------
 * Defines reference genomes, using iGenome paths
 * Can be used by any config that customises the base
 * path using $params.igenomes_base / --igenomes_base
 */

params {
  // illumina iGenomes reference file paths
  genomes {
    'GRCh37' {
      fasta       = { "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/WholeGenomeFasta/genome.fa" }
      bwa         = { "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/BWAIndex/genome.fa" }
      bowtie2     = { "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/Bowtie2Index/" }
      star        = { "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/STARIndex/" }
      bismark     = { "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Sequence/BismarkIndex/" }
      gtf         = { "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.gtf" }
      bed12       = { "${params.igenomes_base}/Homo_sapiens/Ensembl/GRCh37/Annotation/Genes/genes.bed" }
      mito_name   = "MT"
      macs_gsize  = "2.7e9"
      blacklist   = "${baseDir}/assets/blacklists/GRCh37-blacklist.bed"
    }
[...]

The I run the pipeline again and still does not work as expected (groovy/java errors and empty reference paths)"

N E X T F L O W  ~  version 19.10.0
Launching `nf-core/rnaseq` [deadly_mendel] - revision: 3b6df9bd10 [master]
org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.nio.file.FileSystem
org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.nio.file.FileSystem
org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.nio.file.FileSystem
----------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/rnaseq v1.4.2
----------------------------------------------------
Pipeline Release  : master
Run Name          : deadly_mendel
Reads             : fastq_files/*{1,2}_001.fastq.gz
Data Type         : Paired-End
Genome            : GRCh37
Strandedness      : None
Trimming          : 5'R1: 0 / 5'R2: 0 / 3'R1: 0 / 3'R2: 0 / NextSeq Trim: 0
Aligner           : STAR
STAR Index        : 
GTF Annotation    : 
BED Annotation    : 
Remove Ribosomal RNA: false
Biotype GTF field : gene_biotype
Save prefs        : Ref Genome: No / Trimmed FastQ: No / Alignment intermediates: No
Max Resources     : 128 GB memory, 10 cpus, 3d time per job
[...]

Hopefully this feedback is useful.. :?

[ewels]

@pditommaso - any ideas why we are getting the groovy error org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.nio.file.FileSystem when we try to use a deferred execution string in a params variable for use as an input file?

[pditommaso]

Ugly, do you have the complete error stack trace and/or the line where it's happening?

[lconde-ucl]

Attaching the .nextflow.log file.
I didn't let the run continue, I killed it as soon as I saw the header log info
nextflow.log

[ewels]

Minimal test example:

#!/usr/bin/env nextflow

params.path_1 = '/dev/null'
params.path_2 = { "/dev/null" }

in_ch_1 = Channel.fromPath(params.path_1)
in_ch_2 = Channel.fromPath(params.path_2)

println("Path 1: ${params.path_1}")
println("Path 2: ${params.path_2}")

$ nextflow run main.nf
N E X T F L O W  ~  version 20.02.0-edge
Launching `main.nf` [sick_lorenz] - revision: c70000192e
Path 1: /dev/null
Path 2:
java.lang.String cannot be cast to java.nio.file.FileSystem

.nextflow.log

Mar-18 18:19:36.816 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf
Mar-18 18:19:36.974 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 20.02.0-edge
Mar-18 18:19:37.013 [main] INFO  nextflow.cli.CmdRun - Launching `main.nf` [golden_curran] - revision: c70000192e
Mar-18 18:19:37.207 [main] DEBUG nextflow.Session - Session uuid: b9e40fa6-0b74-49dc-9c82-b1fa38c31780
Mar-18 18:19:37.207 [main] DEBUG nextflow.Session - Run name: golden_curran
Mar-18 18:19:37.209 [main] DEBUG nextflow.Session - Executor pool size: 4
Mar-18 18:19:37.236 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 20.02.0-edge build 5276
  Created: 29-02-2020 10:17 UTC (11:17 CEST)
  System: Mac OS X 10.15.3
  Runtime: Groovy 2.5.8 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_144-b01
  Encoding: UTF-8 (UTF-8)
  Process: [email protected] [127.0.0.1]
  CPUs: 4 - Mem: 16 GB (231.3 MB) - Swap: 1 GB (570.8 MB)
Mar-18 18:19:37.263 [main] DEBUG nextflow.Session - Work-dir: /Users/philewels/minimal_nf_example/work [Mac OS X]
Mar-18 18:19:37.264 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/philewels/minimal_nf_example/bin
Mar-18 18:19:37.297 [main] DEBUG nextflow.Session - Observer factory: TowerFactory
Mar-18 18:19:37.300 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Mar-18 18:19:37.583 [main] DEBUG nextflow.Session - Session start invoked
Mar-18 18:19:38.430 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Mar-18 18:19:38.473 [main] DEBUG nextflow.Session - Workflow process names [dsl1]: 
Mar-18 18:19:38.584 [PathVisitor-1] ERROR nextflow.Channel - java.lang.String cannot be cast to java.nio.file.FileSystem
java.lang.ClassCastException: java.lang.String cannot be cast to java.nio.file.FileSystem
    at com.sun.proxy.$Proxy18.getFileSystem(Unknown Source)
    at nextflow.file.PathVisitor.applyGlobPattern0(PathVisitor.groovy:114)
    at nextflow.file.PathVisitor.apply(PathVisitor.groovy:68)
    at nextflow.file.PathVisitor$_applyAsync_closure1.doCall(PathVisitor.groovy:77)
    at nextflow.file.PathVisitor$_applyAsync_closure1.call(PathVisitor.groovy)
    at groovy.lang.Closure.run(Closure.java:486)
    at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705)
    at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:687)
    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Mar-18 18:19:38.607 [main] DEBUG nextflow.script.ScriptRunner - > Await termination 
Mar-18 18:19:38.613 [PathVisitor-1] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.String cannot be cast to java.nio.file.FileSystem
Mar-18 18:19:38.614 [main] DEBUG nextflow.Session - Session await
Mar-18 18:19:38.614 [main] DEBUG nextflow.Session - Session await > all process finished
Mar-18 18:19:38.614 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-18 18:19:38.629 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Mar-18 18:19:38.733 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Mar-18 18:19:38.782 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

[pditommaso]

Ah, this is very useful.

[pditommaso]

Umm, what exactly are you expecting from the following syntax?

params.path_2 = { "/dev/null" }

(please add @pditommaso to your reply).

[ewels]

@pditommaso In this case, I would expect it to resolve to /dev/null as well.

The back-story here is that we have some file paths with a params variable in it. When this is set / overwritten in config profiles that are _before_ this variable is defined it works fine. But if you try to set it in a -c config file it’s too late and it has no effect. I was hoping that by surrounding the path in squiggly brackets that it would not be resolved until later in the execution so that the overwritten params variable would be used.

[pditommaso]

Unfortunately no. The brackets can only be used in the process configuration block because the process has the logic to resolve them later.

[ewels]

Ok, thanks @pditommaso

I wonder if it would work if we import the igenomes.config file in main.nf instead of in nextflow.config 🤔 Are command-line config files parsed before nextflow gets to main.nf? I think maybe that they are..

This chunk of code:

https://github.com/nf-core/tools/blob/9e33e7f02758964a1a09436af1eb6afe0e596e4c/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/nextflow.config#L76-L79

[pditommaso]

Yes, config and command line args are persed before the script run. Also, I would try to avoid conditional config includes .. :confounded:

Have you taken in consideration to include igenomes config in its own profile? then it could be activated as an extra -profile

[ewels]

@pditommaso That could work, but it's not the conditional load which is the problem here. It's the fact that the igenomes config file is being loaded before the -c config file. I don't think that having iGenomes in a profile would change that.

[pditommaso]

It should because, the order in which the profiles are merged it depends on how you mention in the -profile argument I mean -profile foo,bar is different from -profile bar,foo. Could not address your issue?

[ewels]

@pditommaso No, because this is specifically for configs loaded with -c, which is always after all profiles.

[pditommaso]

I see. What is preventing to move it as a base profile as it is done for docker, test, etc.? Asking for a friend :)

https://github.com/nf-core/rnaseq/blob/master/nextflow.config#L119-L128

[ewels]

@pditommaso Same thing - we _always_ want it to load as late as possible so that the variable in the string can be changed in as many places as possible. We moved it after the profiles so that all profiles could change that string and that worked. But it's not enough for -c config flags. I think if we move it to main.nf then that might work for all cases though.. (hopefully)

[pditommaso]

Ok, I go through all the thread, I had started from the error comment. Unfortunately, this is expected. The config was designed to allow the params override from the command, not across config files.

Maybe it should be taken in consideration to handle the igenome dataset not as a config file, but for example, as a ymal file that could be read the from the NF script. The igenomes_base would remain as a parameter that could be used to parametrized the yaml file.

Currently, it could be done with a few lines of custom code. But it would be nice to have a NF built-in support for this.

[ewels]

Sure, that could be nice! Though to be honest I am hesitant to completely replace what we currently have as generally it works so well 😅 It's super nice that you can add to this list of genomes / overwrite specific paths and generally mess around with it using the built-in nextflow config handling. It would probably be possible to maintain this functionality with a YAML approach, but it would need some thought.

@lconde-ucl - as our chief tester, do you think you could try my suggestion mentioned here? https://github.com/nf-core/tools/issues/528#issuecomment-601153463 - just pop those lines in near the top of main.nf and give it a try..

[lconde-ucl]

@ewels - adding these lines in main.nf doesn't work:

[regmond@login12 nf-core_rnaseq]$ nextflow run nf-core/rnaseq --reads 'fastq_files/*{1,2}_001.fastq.gz' --genome GRCh37 -c "/scratch/scratch/regmond/ucl.config" 
N E X T F L O W  ~  version 19.10.0
Launching `nf-core/rnaseq` [small_stonebraker] - revision: 3b6df9bd10 [master]
Unknown method `includeConfig`

 -- Check script '/home/regmond/.nextflow/assets/nf-core/rnaseq/main.nf' at line: 119 or see '.nextflow.log' file for more details

[pditommaso]

includeConfig is not available in the script context.

[drpatelh]

Yeah, I've failed with trying that in the past too. What if we were to add igenomes.config to nf-core/configs and pull it from there? 🤔 Would we be able to get around the config parsing order that way?

[ewels]

Nope, I don't think so. Every config thing that we can possibly do is always going to be before the -c option. I think our only option now is to build this config array in the main script using custom code.

To be completely honest, I'm not super keen on a complete rewrite for what is a fairly minor edge case. I'm more inclined to just document this and how to manually resolve it. What does everyone think?

[drpatelh]

I think its fine to leave it as it is and document it as you suggested. Worse case scenario you just have to provide --igenomes_base on the command-line which is is fine. If and when the custom config is uploaded to nf-core/configs and contains the --igenomes_base parameter it should work as expected anyway.

[ewels]

I guess we can add this to https://nf-co.re/usage/troubleshooting ..?

[drpatelh]

Or here: https://nf-co.re/usage/reference_genomes#illumina-igenomes ? or to both 😅

[MaxUlysse]

I guess we can add this to https://nf-co.re/usage/troubleshooting ..?

Or here: https://nf-co.re/usage/reference_genomes#illumina-igenomes ? or to both sweat_smile

Can we actually mention that somewhere, just spend 30 min on that issue again...

[MaxUlysse]

Actually it was not exactly this one, but a similar one with igenomes_ignore