Nextflow: AWS Batch S3 upload function does not work if output is a directory

[MaxUlysse]

Expected outputs:

output:
    set file("*multiqc_report.html"), file("*multiqc_data") into multiQCReport

Error:

Caused by:
  Missing output file(s) `*multiqc_data` expected by process `RunMultiQC`

There is a multiqc_report.html but it seems that the multiqc_data directory is not found.

here are the logs:
nextflow.command.log
task.command.log
task.command.err
task.command.out
task.command.run

[pditommaso]

In the task wrapper there this command at the end

nxf_s3_upload '*multiqc_data' s3://caw-test-results/work/ce/562f50f7277af34c4dc7d39e727612

Does that folder exist in the S3 path s3://caw-test-results/work/ce/562f50f7277af34c4dc7d39e727612 ?

[MaxUlysse]

Not at all

[pditommaso]

so there's a problem in the upload, I will check it.

[MaxUlysse]

Thanks again.

[pditommaso]

how is exactly the name of that directory ?

[MaxUlysse]

the directory is just multiqc_data, I copied the output directive from NGI-RNAseq, and they include possibilities of personnalization, so at some point it could be anything_multiqc_data I guess.

[pditommaso]

It is a side effect of #535

[MaxUlysse]

Sorry about that

[pditommaso]

that's life! :)

[pditommaso]

But we have already a patch for it! Please try version 0.26.3-SNAPSHOT

[MaxUlysse]

Still the same problem.

[pditommaso]

Can you include the command.run?

On 30 Nov 2017 4:08 p.m., "Maxime Garcia" [e-mail] wrote:

Still the same problem.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextflow-io/nextflow/issues/539#issuecomment-348215899,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAx3SGF7YIpde_XhxpsTyETiZhx-2oyqks5s7sUKgaJpZM4QwlZ7
.

[MaxUlysse]

https://gist.github.com/MaxUlysse/fd038806878bb29835809863a578a7bb

[pditommaso]

The wrapper does not contain the patch https://github.com/nextflow-io/nextflow/commit/613b30a7251cb36e70581d533dcbe5a28ef56dad.

It should be eval "ls -d $pattern" instead of eval "ls $pattern"

Make sure you are using

` $ nextflow info Version: 0.26.3-SNAPSHOT build 4739 Modified: 30-11-2017 14:23 UTC (15:23 CEST)

[MaxUlysse]

I'm using NXF_VER=0.26.3-SNAPSHOT already. How can I specify to use a more recent version?

[pditommaso]

does the build number match with mine?

[MaxUlysse]

not at all:

NXF_VER=0.26.3-SNAPSHOT nextflow info
  Version: 0.26.3-SNAPSHOT build 4738
  Modified: 28-11-2017 15:16 UTC (16:16 CEST)
...

[pditommaso]

if so do the following

CAPSULE_RESET=1 NXF_VER=0.26.3-SNAPSHOT nextflow info

It will force to update your local snapshot.

Then remove CAPSULE_RESET=1 for the following runs

[MaxUlysse]

Thanks a lot, I got the same build now.
Trying at the moment, will tell you how it goes in a moment

[pditommaso]

:crossed_fingers:

[MaxUlysse]

It worked without any problem.
Thanks a lot for everything.

[pditommaso]

thanks to you for the troubleshooting! 🎉 🎉 🎉

[pditommaso]

Included in version 0.26.3.