Rnaseq: Increasing memory above 8G for markDuplicates leads to groovy error

[robsyme]

If I change my nextflow.config to include something like:

process {
  withName: markDuplicates {
    memory = '10G'
    time = '48h'
  }
}

Nextflow run exits with the message:

Sep-30 11:13:00.607 [Actor Thread 22] ERROR nextflow.processor.TaskProcessor - Error executing process > 'markDuplicates (6.MPS12343123-B01.4648.2.pair1AlignedByCoord.out)'

Caused by:
  No signature of method: Script_902d0cea.$() is applicable for argument types: (Script_902d0cea$_runScript_closure48$_closure131$_closure132) values: [Script_902d0cea$_runScript_closure48$_closure131$_closure132@2781ae83]
Possible solutions: is(java.lang.Object), run(), run(), any(), tap(groovy.lang.Closure), use([Ljava.lang.Object;)

Source block:
  markdup_java_options = (task.memory.toGiga() > 8) ? ${params.markdup_java_options} : "\"-Xms" +  (task.memory.toGiga() / 2 )+"g "+ "-Xmx" + (task.memory.toGiga() - 1)+ "g\""
  """
      picard ${markdup_java_options} MarkDuplicates \\
          INPUT=$bam \\
          OUTPUT=${bam.baseName}.markDups.bam \\
          METRICS_FILE=${bam.baseName}.markDups_metrics.txt \\
          REMOVE_DUPLICATES=false \\
          ASSUME_SORTED=true \\
          PROGRAM_RECORD_ID='null' \\
          VALIDATION_STRINGENCY=LENIENT
      samtools index ${bam.baseName}.markDups.bam
      """

If I reduce the memory to 8G or below, nextflow runs, but the markDuplicates step fails is killed for consuming too much RAM.

Is there a better way to set the markDuplicates memory?

[drpatelh]

@robsyme It looks like the error is coming from the syntax you are using to increase the memory requirement. Can you try adding this to a file called custom.config and then running the pipeline with an additional parameter i.e. -c custom.config -resume. Probably best to do it this way than to use nextflow.config.

process {
  withName:markDuplicates {
    memory = '10 GB'
    time = '48h'
  }
}

[robsyme]

Hi Harshil.

Thanks for the recommendation. Unfortunately, the problem persists even after moving the memory specification from nextflow.config to custom.config (with accompanying -c custom.config argument) , after changing 10G to 10 GB, and after changing withName: markDuplicates to withName:markDuplicates.

[drpatelh]

Thanks for trying @robsyme . So if you decrease the memory then markDuplicates completes? Are you using any other custom config settings? and can you send the command you are using to run the pipeline please?

[apeltzer]

tl;dr: MarkDuplicates behaves odd when working with large files, use default parameters such as these for running it (could be added here to RNAseq workflow too as a standard):

  markdup_java_options = '"-Xms4000m -Xmx7g"' //Established values for markDuplicate memory consumption, see issue PR #689 for details

Long Version:

We had some discussion on MarkDuplicates in the past and it showed some rather odd behavior. In the scilifelab/sarek (now being ported over to nf-core) pipeline, for example, we followed the recommendation in the GATK WDL pipelines to use certain parameters for the JVM to keep it as stable as possible.

  markdup_java_options = '"-Xms4000m -Xmx7g"' //Established values for markDuplicate memory consumption, see issue PR #689 for details

Discussion on this is linked here: https://github.com/SciLifeLab/Sarek/pull/689

[robsyme]

@apeltzer : my difficulty isn't MarkDuplicates the picard subcommand, but rather the nf-core/rnaseq nextflow script. When I increase the memory directive for that process above 8G, nextflow encounters a groovy error and is unable to schedule the job.

@drpatelh :

My full custom.config file looks like:

params {
  genomes {
    'L.pneumophila.reannotated' {
      fasta  = 'data/genome/genome.fasta'
      gtf    = 'data/genome/cds.gtf'
      star   = 'data/genome/star'
    }
  }
  genome = 'L.pneumophila.reannotated'
  reads = "data/reads/*/*.pair{1,2}.fastq.gz"
  forward_stranded = true
}

singularity.autoMounts = true

process {
  scratch = '$SLURM_TMPDIR'

  withName:genebody_coverage {
    time = '24h'
  }
  withName:markDuplicates {
    memory = '20 GB'
    time = '48h'
  }
}

I have also tried adding in markdup_java_options myself, but running with this config returns the same No signature of method: $() is applicable for argument types: (Script_902d0cea$_runScript_closure48$_closure131$_closure132) error:

params {
  genomes {
    'L.pneumophila.reannotated' {
      fasta  = 'data/genome/genome.fasta'
      gtf    = 'data/genome/cds.gtf'
      star   = 'data/genome/star'
    }
  }
  genome = 'L.pneumophila.reannotated'
  reads = "data/reads/*/*.pair{1,2}.fastq.gz"
  forward_stranded = true
  markdup_java_options = '"-Xms4000m -Xmx12g"'
}

singularity.autoMounts = true

process {
  scratch = '$SLURM_TMPDIR'

  withName:genebody_coverage {
    time = '24h'
  }
  withName:markDuplicates {
    memory = '20 GB'
    time = '48h'
  }
}

I've just realised that I was using NXF_VER=19.09.0-edge (in ~/.bashrc), which might be the problem. I'm trying again now using 19.07.0 and will update if that clears up the problem. Still, evern if the new NF version is breaking nf-core/rnaseq, it's probably good to understand why.
UPDATE: No - the problem persists using nextflow 19.07.0

[robsyme]

Are we sure that we need the string interpolation dollar-braces (${}) in:

• boolean_test ? ${params.markdup_java_options} : "other options"

rather than just

• boolean_test ? params.markdup_java_options : "other options"

at main.nf:871

I'm running a test now, and if removing the dollar-braces fixes the problem, I've got a PR ready to go (to the dev branch).
Update: Removing the dollar-braces fixes the problem.

Also confirmed with a minimal groovy example:

parameterValue = "this"

10 > 8 ? parameterValue : "that"
// Returns "this"

10 > 8 ? ${parameterValue} : "that"
// Errors with:
// No signature of method: _GroovyUserScript_.$() is applicable for argument types: (_GroovyUserScript_$_run_closure1)

[apeltzer]

Looks like you resolved your own problem :-)

[robsyme]

Yay open source!
Thanks @apeltzer and the nf-core team.