Nextflow: _undefined parameter_ even though it is defined in a config file.

[huguesfontenelle]

(migrated from #1032 since this is a different bug)

Bug report

Expected behavior and actual behavior

_undefined parameter_ even though it is defined in a config file.

Steps to reproduce the problem

nextflow run huguesfontenelle/nextflow-bug-config --genome smallGRCh37

Program output

N E X T F L O W  ~  version 19.01.0
Launching `huguesfontenelle/nextflow-bug-config` [sick_wozniak] - revision: 88fad15780 [master]
WARN: Access to undefined parameter `genomes` -- Initialise it to a default value eg. `params.genomes = some_value`
ERROR ~ Genome smallGRCh37 not found in configuration

 -- Check '.nextflow.log' file for details

For comparison, I'm expecting the same result as the one I obtain locally:

nextflow run pipelines/mapping.nf --genome smallGRCh37

N E X T F L O W  ~  version 19.01.0
Launching `pipelines/mapping.nf` [nauseous_cuvier] - revision: 329ad7c4ac
[max_cpus:2, max_memory:7 GB, max_time:1h, outDir:/Users/hugues/learn/nextflow-bug-config, out-dir:/Users/hugues/learn/nextflow-bug-config, genomes:[smallGRCh37:[cosmic:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz, cosmicIndex:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz.tbi, dbsnp:dbsnp_138.b37.small.vcf.gz, dbsnpIndex:dbsnp_138.b37.small.vcf.gz.tbi, genomeFile:human_g1k_v37_decoy.small.fasta, bwaIndex:human_g1k_v37_decoy.small.fasta.{amb,ann,bwt,pac,sa}, genomeDict:human_g1k_v37_decoy.small.dict, genomeIndex:human_g1k_v37_decoy.small.fasta.fai, knownIndels:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf, knownIndelsIndex:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf.idx]], genome:smallGRCh37]

Environment

• Nextflow version: 19.01.0
• Java version: 1.8.0_191
• Operating system: tested on both macOS, Linux

Additional context

(Add any other context about the problem here)

[huguesfontenelle]

Also, the remote run uses whichever nextflow.config it finds in the local work dir, so make sure to try this in a empty dir, and with no ~/.nextflow/config file.

[MaxUlysse]

What I found very disturbing is that when you do:

nextflow config huguesfontenelle/nextflow-bug-config

and

nextflow config main.nf

on a local copy, you get the same output.

[drpatelh]

I was able to reproduce the problem too. Seems like nextflow.config is being read but the includeConfig statements arent being evaluated.

[MaxUlysse]

@drpatelh Do you have the same config output as well?

[pditommaso]

The config reported is the following

$ nextflow config huguesfontenelle/nextflow-bug-config

manifest {
   author = 'Hugues Fontenelle'
   defaultBranch = 'dev'
   description = 'Demonstrating a bug'
   homePage = 'https://github.com/huguesfontenelle/nextflow-bug-config'
   mainScript = 'main.nf'
   nextflowVersion = '>=19.01.0'
}

params {
   genomes {
      smallGRCh37 {
         genomeFile = 'human_g1k_v37_decoy.small.fasta'
         bwaIndex = 'human_g1k_v37_decoy.small.fasta.{amb,ann,bwt,pac,sa}'
         genomeDict = 'human_g1k_v37_decoy.small.dict'
         genomeIndex = 'human_g1k_v37_decoy.small.fasta.fai'
      }
   }
}

What are you expecting instead ?

[drpatelh]

@pditommaso I dont get any config reported with that command. The name of the main script isnt main.nf either:
https://github.com/huguesfontenelle/nextflow-bug-config/blob/88e1f738840e964eb71a87e672224e4b1e416fb5/nextflow.config#L6

Im wondering how you got that output!

I managed to find the solution to this issue yesterday. It was failing because the main script wasnt in the top-level directory. Im assuming this is expected behaviour.
https://nfcore.slack.com/archives/CE6P95170/p1551366630039700

[huguesfontenelle]

@pditommaso Please wipe out my previous attempt..

rm -rf ~/.nextflow/assets/huguesfontenelle

[pditommaso]

You are right. Therefore the reported config is:

$ nextflow config huguesfontenelle/nextflow-bug-config

manifest {
   author = 'Hugues Fontenelle'
   defaultBranch = 'master'
   description = 'Demonstrating a bug'
   homePage = 'https://github.com/huguesfontenelle/nextflow-bug-dict'
   mainScript = 'pipelines/mapping.nf'
   nextflowVersion = '>=19.01.0'
}

process {
   cpus = 1
   memory = '1 GB'
   time = '5m'
}

params {
   max_cpus = 2
   max_memory = '7 GB'
   max_time = '1h'
   outDir = '/Users/pditommaso/Projects/nextflow'
   genomes {
      smallGRCh37 {
         cosmic = 'b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz'
         cosmicIndex = 'b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz.tbi'
         dbsnp = 'dbsnp_138.b37.small.vcf.gz'
         dbsnpIndex = 'dbsnp_138.b37.small.vcf.gz.tbi'
         genomeFile = 'human_g1k_v37_decoy.small.fasta'
         bwaIndex = 'human_g1k_v37_decoy.small.fasta.{amb,ann,bwt,pac,sa}'
         genomeDict = 'human_g1k_v37_decoy.small.dict'
         genomeIndex = 'human_g1k_v37_decoy.small.fasta.fai'
         knownIndels = '{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf'
         knownIndelsIndex = '{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf.idx'
      }
   }
}

What's wrong with this?

[huguesfontenelle]

Nothing wrong with what's reported by the config command.
Have you tried running

nextflow run huguesfontenelle/nextflow-bug-config --genome smallGRCh37

in a fresh directory?
Even though params.genomes is defined (as you saw in the config, the nextflow script doesn't see it)

[pditommaso]

Soooo, there are two mapping.nf scripts, one in the project root and another in the pipelines directory. Is this a NF puzzle ?

[MaxUlysse]

The two of them are the same, but the one in the root should be removed.

[huguesfontenelle]

Aargh. The redundant file is now removed. I'm so sorry; I realise that it is difficult to investigate this case if I keep on producing bad examples...

The buggy behaviour remains though.

And no, I'm not making puzzle for the fun of it. There is a bug!

[pditommaso]

Yes, there could be a glitch when using subdirectories for the main script. I would suggest to keep the main script in the project root for now.

[pditommaso]

well I run

$ nextflow run mapping.nf 
N E X T F L O W  ~  version 19.02.0-edge
Launching `mapping.nf` [maniac_spence] - revision: c458ec614f
[max_cpus:2, max_memory:7 GB, max_time:1h, outDir:/Users/pditommaso/Projects/nextflow/yyy, out-dir:/Users/pditommaso/Projects/nextflow/yyy, genomes:[smallGRCh37:[cosmic:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz, cosmicIndex:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz.tbi, dbsnp:dbsnp_138.b37.small.vcf.gz, dbsnpIndex:dbsnp_138.b37.small.vcf.gz.tbi, genomeFile:human_g1k_v37_decoy.small.fasta, bwaIndex:human_g1k_v37_decoy.small.fasta.{amb,ann,bwt,pac,sa}, genomeDict:human_g1k_v37_decoy.small.dict, genomeIndex:human_g1k_v37_decoy.small.fasta.fai, knownIndels:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf, knownIndelsIndex:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf.idx]]]

$ nextflow run pipelines/mapping.nf 
N E X T F L O W  ~  version 19.02.0-edge
Launching `pipelines/mapping.nf` [chaotic_albattani] - revision: c989d1a3b0
[max_cpus:2, max_memory:7 GB, max_time:1h, outDir:/Users/pditommaso/Projects/nextflow/yyy, out-dir:/Users/pditommaso/Projects/nextflow/yyy, genomes:[smallGRCh37:[cosmic:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz, cosmicIndex:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz.tbi, dbsnp:dbsnp_138.b37.small.vcf.gz, dbsnpIndex:dbsnp_138.b37.small.vcf.gz.tbi, genomeFile:human_g1k_v37_decoy.small.fasta, bwaIndex:human_g1k_v37_decoy.small.fasta.{amb,ann,bwt,pac,sa}, genomeDict:human_g1k_v37_decoy.small.dict, genomeIndex:human_g1k_v37_decoy.small.fasta.fai, knownIndels:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf, knownIndelsIndex:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf.idx]]]

$ nextflow run . 
N E X T F L O W  ~  version 19.02.0-edge
Launching `./pipelines/mapping.nf` [gigantic_euclid] - revision: c989d1a3b0
[max_cpus:2, max_memory:7 GB, max_time:1h, outDir:/Users/pditommaso/Projects/nextflow/yyy, out-dir:/Users/pditommaso/Projects/nextflow/yyy, genomes:[smallGRCh37:[cosmic:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz, cosmicIndex:b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz.tbi, dbsnp:dbsnp_138.b37.small.vcf.gz, dbsnpIndex:dbsnp_138.b37.small.vcf.gz.tbi, genomeFile:human_g1k_v37_decoy.small.fasta, bwaIndex:human_g1k_v37_decoy.small.fasta.{amb,ann,bwt,pac,sa}, genomeDict:human_g1k_v37_decoy.small.dict, genomeIndex:human_g1k_v37_decoy.small.fasta.fai, knownIndels:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf, knownIndelsIndex:{1000G_phase1,Mills_and_1000G_gold_standard}.indels.b37.small.vcf.idx]]]

The params is identical in all cases

[huguesfontenelle]

Another way to state this is that a remote run fails, while a local run succeed.

remote run:

nextflow run huguesfontenelle/nextflow-bug-config --genome smallGRCh37

local run:

nextflow run pipelines/mapping.nf --genome smallGRCh37

The remote run fails, allegedly, because of an undefined parameter.