Nextflow: Error message reporting file name collision is misleading

[Hammarn]

https://travis-ci.org/SciLifeLab/NGI-RNAseq/builds/298050472

This is the offending code: https://github.com/SciLifeLab/NGI-RNAseq/blob/master/main.nf#L1041

[pditommaso]

I'm not able to replicate it. I've tried the following but it's working fine:

process foo {
  input:
  file 'foo/*' from file('a')
  file 'bar/*' from file('a')

  '''
  echo true
  '''
}

[Hammarn]

I'm running a test on my local machine and will paste a full log shortly.

[Hammarn]

nextflow_log.txt

[pditommaso]

Ar you able to provide a minimal failing example like the above ?

[Hammarn]

Hmm, not at the top of my head, but I guess I could give it a try. Will probably take me some time though.

[MaxUlysse]

I have a similar problem with CAW, but in our case, the problem with the two or more input files with the same name doesn't always happen.
I know in which case it's working, and in which it's failing, trying to understand why, and I'll see if I can make a minimal example.

[pditommaso]

I've uploaded a snapshot producing a more detailed log to better troubleshoot this problem. Please define the variable NXF_VER=0.26.1-SNAPSHOT, try to replicate the issue and please include the resulting .nextflow.log file.

[pditommaso]

However I don't think it's a bug. One the file reporting the problem is produced at this line. Therefore there many tasks creating the output .command.log, which are all collected and expected to be put in the directory stringtie, hence there's a file name collision.

Either you get just one (maybe the first) or rename the log file to avoid the name collision eg.

input:
file ('stringtie/.command.log.*') from stringtie_log.collect() 

[Hammarn]

Ok, new log attached here:
updated_log.txt
You might be right, and that we have to do some extra renaming. I'm still pussled by the error message though.

Process `multiqc` is staging two or more input files with the same name -- offending files: stringtie/.command.log, rseqc/log.txt

Do that mean that mean that both stringtie/.command.log, rseqc/log.txt are problematic? Because the way it's written now it implies that those two file are named the same which is clearly not the case.

[pditommaso]

The log file you have attached is reporting you still are using 0.26.0 not the snapshot. However the problem look clear.

Do that mean that mean that both stringtie/.command.log, rseqc/log.txt are problematic? Because the way it's written now it implies that those two file are named the same which is clearly not the case.

No. I mean there's more than one file names stringtie/.command.log and more than one rseqc/log.txt. This a potential bug in your code, because only one copy of these files will be kept.

To replicate the problem you can write:

files = [ file('a'), file('a') ].channel()

process foo {
  input:
  file 'foo/*' from files.collect()

  '''
  echo true
  '''
}

and a possible solution:

files = [ file('a'), file('a') ].channel()

process foo {
  input:
  file 'foo/a.*' from files.collect()

  '''
  echo true
  '''
}

[Hammarn]

You are right, here is the output of actually using the snapshot:
updated_log.txt

No. I mean there's more than one file names stringtie/.command.log and more than one
rseqc/log.txt

So what I wrote first then, that it's a problem with both of those processes.
With the new log it's quite clear that there are indeed multiple occurences of the same file, (if I'm reading the output correctly) rseqc/log.txt:4,.
I'm not sure if we actually care in this case though, I have a suspiction that we just want _a_ file to get a version name or similar. Anyhow I will look into that and fix the code accordingly so you can close this issue.

One thing though is that I think the error message has some room to be misinterpreted, since that was what both me and Phil apparently did. It would be nice if it said that it found four instances of files all named a _and_ four instances of file b. as of now it implies that file a & b are named the same.
I'll try and get back with a suggestion for how to formulate the error to make it a bit clearer.

[pditommaso]

One thing though is that I think the error message has some room to be misinterpreted

Yes, we thought the same with @skptic. We have changed as following

Process $name input file name collision -- There are multiple input files for each of the following file names: xxx, ...

Feel free to provide a better/improved one.

[Hammarn]

Seems like a very good improvement! Thanks for your help!

[ewels]

Hi @pditommaso,

Could you clarify how your possible solution above works? The two snippets look identical to me.. How does the a. prefix change anything?

Cheers,

Phil

[pditommaso]

Having * means maintain the original file names. Instead prefix* means stage the files naming them prefix1, prefix2 .. etc. Look at this table.

[ewels]

[pditommaso]

LOL

[pditommaso]

Included in version 0.26.1-SNAPSHOT build 4732