Nextflow: Add enabled to conda scope

[KevinSayers]

I suggest adding enabled to the conda scope. If you define a conda directive at the process level it could be useful to be able turn on/off conda.

Similar to what is currently implemented for containers e.g. docker.enabled = true in the config, conda.enabled = true would enable more portability.

[pditommaso]

Similar to module environment, conda is implicitly enabled when specified. For container the main reason was to allow to switch between different engines, ie. docker, singularity, etc.

If you want to specify a conda environment without enabling it by default use a profile, eg

profiles {
  withConda { process.conda = '/env.yml'  }
}

[KevinSayers]

I didn't think about modules and how they function. I think it could be useful to enable/disable anything that manages dependencies whether that be container, module, or conda. In my opinion it would be better for reproducible/portable workflows as it could reduce the need to modify a workflow that you have pulled if you aren't using the dependency management.

Obviously this could all be avoided specifying this in the nextflow.config as different profiles, as you suggested.

[pditommaso]

I think it could be useful to enable/disable anything that manages dependencies whether that be container, module, or conda.

Well, this is a different thing.

In my opinion it would be better for reproducible/portable workflows as it could reduce the need to modify a workflow that you have pulled if you aren't using the dependency management.

I agree. I guess, you are defining the conda deps in the main script. You should do that only for small scripts / testing purpose. In all other cases you should define the (conda) deps in nextflow.config file. Using a separate profile only for that allows you to enable/disable the conda support an you need.

I'm not supporting the idea of having an enabled flag because that will promote that definition of conda properties in the main script file, that's not a good practice.

[ewels]

I asked @pditommaso for the same thing and got the same response 😉 In the end we switched to using config profiles to manage all of this. See for example the nf-core/methylseq pipeline:

profiles {
  conda { process.conda = "$baseDir/environment.yml" }
  docker { docker.enabled = true }
  singularity { singularity.enabled = true }
}

So now instead of doing nextflow run -with-docker you do nextflow run -profile docker. Because you can put any config in you like, you can then have the nice and consistent:

nextflow run -profile (docker|singularity|conda)

Note that profiles can be used in combination, so for example on our UPPMAX cluster we have the choice of running with -profile singularity,uppmax or -profile conda,uppmax.

I hope this helps!

Phil

[pditommaso]

Exactly!

[KevinSayers]

@ewels @pditommaso I think this should be mentioned explicitly in the docs as currently the examples all show using the conda directive in a process block. I have created a PR with a 'best practices' section #802 as a suggestion.

[pditommaso]

It has been added a note in the documentation. See 6f6d5c9234824647a78b6249af187684cd4f8949.

[ewels]

With the advent of DSL2 this issue is coming around to bite us again. We want to define the conda environment in the module in the process that will be imported, and not in the pipeline config that is importing it, eg:

process FASTQC {
    container "docker.pkg.github.com/nf-core/$MODULE"
    conda "${moduleDir}/environment.yml"
    // [...]
}

However, by defining it in the process like this, conda is implicitly enabled.

Supporting conda.enabled / -with-conda etc in the same way as the container engines are handled would solve this for us in the most elegant way. The process could then define the software in every possible way without requiring a separate definition in the pipeline that imports it - much better for reproducibility and stability.

We're looking at using workarounds such as conda: if(params.use_conda) {"environment.yml"} else {null} (suggested by @grst, untested), but obviously native support would be much better.

[pditommaso]

My main concern is breaking exiting codebase and configs. What about a simple trick like this?:

conda params.conda ? "/some.env" : null 

ps. put conditional expressions at the top of nf-core best practice! 😄

[pditommaso]

haven't tried, it may require parentheses eg.

conda (params.conda ? "/some.env" : null)

[ewels]

Ok, we'll stick with the workaround for now then. Maybe we'll use params.with-conda so that we can have --with-conda and -with-docker 😅 (idea from @JoseEspinosa!)

[pditommaso]

-1 on the option naming! The use of - and -- is already confusing for newbie users. Adding the -- alternative to the with-something scheme will further increase uncertainty.

[ewels]

Yes I know, poor joke sorry 😂

[JoseEspinosa]

I have the feeling that I am not the only one that wrote-with-conda in my nextflow command thinking that it will just work 🤣

[ewels]

Yeah, personally I feel that the _future_ confusion caused by -with-conda not working in the same way as the other flags is greater than the potential confusion of changing the behaviour. But that's not my call to make.. 😄

[pditommaso]

In the future the manual Conda handling won't be needed anymore. We'll automate the building of containers from Conda .. 😎

[ewels]

What about people who can't use containers though? They'll still want Conda..?

[pditommaso]

Not saying it will be removed, but IMO it will be less relevant.