Nextflow: Add containerOptions as a process directive

[KevinSayers]

It would be useful to be able to modify the containerOptions for each process block. This would be beneficial as some containers for some processes may need unique flags not shared among other process containers.

process foo{
    container 'whatever.img'
    containerOptions '--nv'
}

should yield

singularity exec --nv ....

[pditommaso]

Here there are two main alternatives:

1) add a containerOptions directive allowing to add free style container run options (somehow similar to clusterOptions

2) extend syntax of container providing the ability to specify extra parameters eg
container 'user/foo', options: '-c --this --that', mounts: true

[chuckpr]

I vote for option 2. We invariably are defining the container within the process block. Feels more explicit than defining containers in nextflow.config.

[MaxUlysse]

I define the containers in the nextflow.config files, so I would vote for option 1.

[tdudgeon]

Option 2 would be better for me, but looks like both approaches are needed?

[pditommaso]

Since this feature won't be available for container native executors such us k8s and awsbacth, I think it's better to keep separated with a specific setting. I mean solution 1.

In this way it will be easier to provide container specific settings only for some target platforms.

[KevinSayers]

Seems reasonable to me. So the two ways to define the containerOptions would be:

In process block:

process foo{
    container 'whatever.img'
    containerOptions '--nv'
}

In config:

process.$foo.containerOptions = '--nv'

Would the runOptions setting for docker/singularity scope be additive with the containerOptions? I could see that being useful.

[pditommaso]

Would the runOptions setting for docker/singularity scope be additive with the containerOptions? I could see that being useful.

yes.

Also note with the new config syntax would be:

process { 
  withName: foo { containerOptions = '--nv' }
}

or

process { 
  withLabel: gpu { containerOptions = '--nv' }
}

provide you have annotated process with label 'gpu'.

(cool, isn't it :))

[pditommaso]

Available for testing using version 0.29.0-SNAPSHOT.

[KevinSayers]

Tried the containerOptions in a process, in the config, and with the new withLabel syntax in the config. Worked perfectly with both Singularity and Docker.

[pditommaso]

Included in version 0.29.0-RC1.