Nextflow: Suggestion: auto assign profiles based on hostname

[ewels]

Just a thought that I had earlier. We usually have a bunch of different config profiles for different systems. I've previously implemented warnings that check the system hostname and flag if it looks like the wrong profile was used. It would be really nice if config profiles could be automatically used based on this.

A simple toy example:

nextflow.config

profiles {
  standard {
    includeConfig 'conf/base.config'
  }
  uppmax {
    auto: { host.hostname.contains('.uppmax.uu.se') }
    includeConfig 'conf/base.config'
    includeConfig 'conf/uppmax.config'
  }
  system_two {
    auto: { host.hostname.contains('.system_two.com') }
    includeConfig 'conf/base.config'
    includeConfig 'conf/system_two.config'
  }
}

I'm trying to generalise a little here, but hopefully you get the idea. These profiles could still be specified normally using -profile uppmax, but would automatically be used when the auto call returns true.

I suspect that this would prevent some of the common user errors that we see with our pipelines and make it them easier to use.

Phil

[apeltzer]

Feels like I could try to work on this one during nfhack today :-)

[pditommaso]

This is not complicated. The only thing I'm not sure is that the auto: syntax can be managed.

[ewels]

NB: when could be a better name instead of auto, as that's consistent with pipeline syntax. But minor thing. What do you mean about managing the syntax @pditommaso?

[apeltzer]

A hint on where you think this code change should be implemented would be helpful as well :-) I currently struggle quite a bunch to find the appropriate place for this

[pditommaso]

Yes, maybe when is slightly better than auto, however the tricky thing is to handle the colon character. TLDR; the nextflow config file is a GroovyObject class that must be reduced to Java Map object, therefore everything must be converted to a key-value pair.

When using the property dot notation, it's easy to understand how nested object are created, e.g.

foo.bar = 1 

creates a foo map object containing the bar:1 pair.

When using the curly brackets notation, that code should be read as a dynamic method invocation to which it's passed the closure, then the closure set the key-value pairs in the nested pairs, eg.

foo {
  bar =1 
}

Any identifier postfixed with a : is supposed to be a java/groovy label keyword, therefore it's possible to write,

foo: bar = 1 

or

foo: bar { 
  x = 1 
}  

In both cases foo: is a label preceding that statement i.e. a variable assignment or a method invocation.

I'm not sure that's possible to write

foo: {
  /* boolean expression */
}

Because that it's not a method invocation (there's no method name). I think it's reduced to a closure definition (that should be used to define the profile activation logic), but it could be tricky to handle in the config object.

The config object is parsed using the ConfigParser. The first goal is to understand how the when definition is mapped in the config object. The ConfigParserTest can help to make some experiments.

Then once we have the auto profile information it should be enforce in the ConfigBuilder at this point.

Hope it helps 😄

[ewels]

So... would this work?

profiles {
  standard {
    includeConfig 'conf/base.config'
  }
  uppmax {
    when = host.hostname.contains('.uppmax.uu.se')
    includeConfig 'conf/base.config'
    includeConfig 'conf/uppmax.config'
  }
  system_two {
    when = host.hostname.contains('.system_two.com')
    includeConfig 'conf/base.config'
    includeConfig 'conf/system_two.config'
  }
}

[pditommaso]

yep

[apeltzer]

Thanks both! I'll try to fit this in at some point in the next weeks :-) Hopefully can be of help here :-)

[ewels]

FYI for googlers: we're now doing this in a custom manner in nf-core pipelines.
Config setup is here and function is here..

Of course, a native method for doing this within nextflow as discussed above is always nicer 😉

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.