Nextflow: Allow a process to access input file metadata

[pditommaso]

It should be possible to access input file metadata such as size, lastModified, etc.

This would allow ease the definition of dynamic resource allocations. For example:

𝚒𝚗_𝚌𝚑 = 𝙲𝚑𝚊𝚗𝚗𝚎𝚕.𝚏𝚛𝚘𝚖𝙿𝚊𝚝𝚑('𝚑𝚎𝚕𝚕𝚘.𝚝𝚡𝚝')  
𝚙𝚛𝚘𝚌𝚎𝚜𝚜 𝚏𝚘𝚘 {   
  𝚖𝚎𝚖𝚘𝚛𝚢 { 𝚡.𝚜𝚒𝚣𝚎() < 𝟷_000_000 ? 𝟸.𝙶𝙱 : 𝟾.𝙶𝙱 }
  𝚒𝚗𝚙𝚞𝚝: 𝚏𝚒𝚕𝚎 𝚡 𝚏𝚛𝚘𝚖 𝚒𝚗_𝚌𝚑
  𝚜𝚌𝚛𝚒𝚙𝚝:
  """
  𝚌𝚘𝚖𝚖𝚊𝚗𝚍 --𝚒𝚗𝚙𝚞𝚝 $𝚡
  """
}

Currently it's not possible to access process input file metadata. A possibile workaround requires to fetch that information from outside the process, see here for an example.

[rsuchecki]

Are you sure? I obviously did not make it clear, but I tested that and it appears to work.

Given a slightly modified nf script:

#!/usr/bin/env nextflow 

input_ch = Channel.fromPath('hello.txt')

process foo {
  tag {x.size() < 1_000_000 ? 2.GB : 8.GB}
  input:
    file(x) from input_ch
  script:
  println(x.size())
  """
  cat $x
  """
}

The file size is 416 bytes

ll hello.txt 
-rw-rw-r-- 1 rad rad 416 Jun  8 12:22 hello.txt

nextflow dynamic-res.nf 
N E X T F L O W  ~  version 0.29.0
Launching `dynamic-res.nf` [thirsty_torricelli] - revision: a00ae2f982
[warm up] executor > local
416
[a1/50c6ee] Submitted process > foo (2 GB)

[pditommaso]

That's a lucky situation. I guess if the file is not in the launching directory is not working.

[apeltzer]

Just looked up in the documentation and found this ticket - I have a couple of use cases where a process is quite heavily dependent on input size and this would make it possible to specify the size as another parameter on resource consumption :-)

[ewels]

I just came up against this problem too... (a conditional process according to file size)

[pditommaso]

Have a look at a possible workaround.

[ewels]

Nice, thanks! Yes, I have it working quite nicely now: https://github.com/nf-core/rnaseq/pull/37 - suggestions for improvement welcome!

[pditommaso]

This will be included in the next version.

Bonus feature, it's now possibile to compare numeric values with memory and duration units. This allows to write expression such as file.size() > 1.GB.

[apeltzer]

Awesome work @pditommaso! This is really neat :-)