Tools: Refactor input reads channel

[ewels]

At the moment we have a pretty nasty and hacky bit of code that handles the input read pairs for pipelines:
https://github.com/nf-core/tools/blob/aa947db2782715433fe1b9216de691e942370165/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/main.nf#L101-L123

This is because when it was written, fromFilePairs _required_ a glob expression. When working with remote files (eg. a -profile test config) we want to specify URLs for files. Nextflow can not download files from a URL that have any wildcards.

Because of this incompatibility I wrote the above workaround. However, I think @pditommaso now updated fromFilePairs to accept both glob patterns and a list of paths. This should mean that we can revert the above hack. We need to confirm that this is the case and that it works, then revert the template code.

[ewels]

x-ref the issue where @pditommaso says that this is updated: https://github.com/nextflow-io/nextflow/issues/874#issuecomment-427063183

[pditommaso]

A big +1 to this. The nf-core project visibility is growing a lot and people also use these pipelines as reference implementation, therefore it's important to promote good best practices whenever possible.

[ewels]

Is it ready? Just need to figure out how to implement it if so...

[apeltzer]

Hm, I tried it out like this:

  reads = ['https://github.com/nf-core/test-datasets/raw/eager2/testdata/Mammoth/JK2782_TGGCCGATCAACGA_L008_R1_001.fastq.gz.tengrand.fq.gz','https://github.com/nf-core/test-datasets/raw/eager2/testdata/Mammoth/JK2782_TGGCCGATCAACGA_L008_R2_001.fastq.gz.tengrand.fq.gz']

And then using the same Channel.fromFilePairs(params.reads) as usual, but it doesn't work - or am I too stupid?

Looking at the docs, this seems not to be mentioned (atm?)... maybe I understood it incorrectly...

[apeltzer]

https://github.com/nextflow-io/nextflow/issues/946

might help now that 19.01 is out!

[maxibor]

According to Nextflow issue 946 this will only come with 19.04.
As of Nextflow 19.01.0.5050 this is not yet available

nextflow run main.nf -profile docker,test --reads '/Users/borry/Documents/nf-core/data/Testdata_R1.tiny.fastq.gz'
N E X T F L O W  ~  version 19.01.0
Launching `main.nf` [cranky_montalcini] - revision: 2ede9e3f42
WARN: Access to undefined parameter `genome` -- Initialise it to a default value eg. `params.genome = some_value`
WARN: Access to undefined parameter `fasta` -- Initialise it to a default value eg. `params.fasta = some_value`
ERROR ~ No signature of method: nextflow.Channel$_fromFilePairs0_closure6.call() is applicable for argument types: (sun.nio.fs.UnixPath) values: [/Users/borry/Documents/nf-core/data/Testdata_R1.tiny.fastq.gz]

[apeltzer]

Fair point - I think this is somewhat loosely scheduled for in 2 weeks ;-)

[ewels]

@pditommaso - I just had another go with this, but I'm still struggling to get it to work as I expect..

ch_read_files = Channel.fromFilePairs([
  'https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub1.fastq.gz',
  'https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub2.fastq.gz',
  'https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub3.fastq.gz'
], size: 1)

However, I get this error from the above code:

Invalid method `call` invocation with arguments: /nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub1.fastq.gz (nextflow.file.http.XPath) on _fromFilePairs0_closure10 type

What am I doing wrong here? How can I specify a number of FastQ file URLs for a fromFilePairs channel?

Thanks!

Phil

[pditommaso]

Unfortunately, it looks like a bug.

[ewels]

x-ref: https://github.com/nextflow-io/nextflow/issues/1405

[pditommaso]

Scheduled for 19.12.0-edge / 20.01.0

[ewels]

Great!

[drpatelh]

This may be indirectly solved by replacing the glob reads input with a design file. See here

[ewels]

Will still need fixing in pipelines that use this input style. From a quick search:

• rnaseq
• methylseq
• smrnaseq
• scrnaseq
• crispvar
• hlatyping
• cageseq
• smartseq2
• hic
• hlatyping
• rnafusion
• coproid
• nascent
• ampliseq
• mag
• eager

[drpatelh]

Should this issue be moved to rnaseq then since it won't be fixed in tools anymore? At least in the long-term/whenever we merge dsl2_template? Or you planning to get a quick fix in and synced with the existing code?

[ewels]

A cheeky fix would be lovely if we can manage it 🤔

[ewels]

Had a stab at this earlier today and talked to @pditommaso for some help. I think it's still going to be difficult to refactor this code with the way that fromFilePairs works without needing a lot of downstream changes or making the code more complex than it currently is.

Seeing as we are going to move towards sample sheets anyway, I think we can just close this issue. Current code in production is stable, so even if it feels a little hacky I think it's fine.