Nextflow: Allow trace storage on remote object storage

[apeltzer]

Bug report

When storing a trace file/work dir on an S3 bucket, a pipeline fails with an error message

Expected behavior and actual behavior

The pipeline runs through when specifying an accessible S3 bucket for storing trace/work directories or files.

Steps to reproduce the problem

Use e.g. a working nf-core pipeline and specify to store a trace file on an S3 bucket.

Environment

• Nextflow version: [0.30.2]
• Java version: [1.8.188]
• Operating system: [Linux, AWS EC2 AMI]

[apeltzer]

I'm not sure whether just the trace file is problematic, but other pipeline metrics/reports would be interesting, too:

timeline {
  enabled = true
  file = "./piperun/pipeline_info/ICGC-FeatureCounts_timeline.html"
}
report {
  enabled = true
  file = "./piperun/pipeline_info/ICGC-FeatureCounts_report.html"
}
trace {
  enabled = true
  file = "./piperun/pipeline_info/ICGC-FeatureCounts_trace.txt"
}
dag {
  enabled = true
  file = "./piperun/pipeline_info/ICGC-FeatureCounts_dag.svg"
}

[veeravalli]

@apeltzer
Hi Alex, how do you bypass this issue now? Thanks.
Am just deleting trace.txt before I invoke the workflow with -resume option.

[lorenzgerber]

@pditommaso I had a look into this issue. The simplest fix would be to implement the S3 move operation as done in the upstream library :

        copy(source, target, options);
        delete(source);

I am willing to implement and test this, but I would need some hints on how to temporarily include a forked nextflow-s3fs repo in the gradle/capsule/jitpack build process for testing.

[pditommaso]

Hi Lorenz, couldn't check the code. What's the benefit over the current
implementation? p

On Tue, May 7, 2019, 04:24 Lorenz Gerber [e-mail] wrote:

@pditommaso https://github.com/pditommaso I had a look into this issue.
The simplest fix would be to implement the S3 move operation as done in the upstream
library
https://github.com/Upplication/Amazon-S3-FileSystem-NIO2/blob/4fe1a64942cd565bb871a4e8c5add19687313f6b/src/main/java/com/upplication/s3fs/S3FileSystemProvider.java#L429
:

    copy(source, target, options);
    delete(source);

I am willing to implement and test this, but I would need some hints on
how to temporarily include a forked nextflow-s3fs repo in the
gradle/capsule/jitpack build process for testing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextflow-io/nextflow/issues/813#issuecomment-489882021,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGHOSB2QFNKZXUR6TDDSSLPUDR7FANCNFSM4FMAYJXQ
.

[lorenzgerber]

Implementing the the move operation should fix this present issue (#813)

Current code in nextflow-s3fs/S3FileSystemProvider.java:

@Override
public void move(Path source, Path target, CopyOption... options)
        throws IOException {
    throw new UnsupportedOperationException();
}

Code upstream in Amazon-S3-FileSystem-NIO2/S3FileSystemProvider.java:

@Override
public void move(Path source, Path target, CopyOption... options) throws IOException {
    if (options != null && Arrays.asList(options).contains(StandardCopyOption.ATOMIC_MOVE))
        throw new AtomicMoveNotSupportedException(source.toString(), target.toString(), "Atomic not supported");
    copy(source, target, options);
    delete(source);
}

[pditommaso]

Oh, I've never realised that move was not implemented 👍. Tho not 100% this fix #813.

[lorenzgerber]

I would like to test it locally. I understand that nextflow builds nextflow-s3fs from the source of your repo, right? Is there a simple way to make nextflow use a local build/build from local source?

[pditommaso]

Yes, that's the project used by NF. Unfortunately, the tests are a complete mess. I was never able to re-organise a consistent test set when forked the project. When a need to make a change I test it packaging the jar which is really bad I know.

[olgabot]

Hello, has there been any progress on this? This is a pain point for using AWS as -resume doesn't work out of the box and requires workarounds, which is feasible for power users but not for everyone.

[pditommaso]

No sorry

[ggabernet]

Hi, in case it helps someone -resume works for me if specifying a new trace directory path with every run. For _nf-core_ pipelines: --tracedir

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ewels]

Hi all, this issue was closed but still presents a fairly major problem. As @olgabot says, not being able to use -resume directly on failed pipelines is very annoying, as the user has to go into s3 to delete the publishDir before resuming, or use a different output directory.

@pditommaso could this issue be reopened? The discussion thread is a little scrappy, is the problem clear?

[pditommaso]

The problem here is that NF finds a file with the same name already existing, try to rename it, but the AWS S3 does not have an API for it and therefore fails.

Possible workarounds:

• delete those files beforehand
• use a name that's unique, for example

trace {
    enabled = true
    file = "s3://nextflow-ci/work/foo-${params.random}.txt"
}

[ewels]

I guess this is also an issue for pipeline processed result files saved with publishDir, not just the trace files? Deleting the files beforehand would be fine for me 👍🏻

[pditommaso]

No, because there the policy if to override the file. Instead here NF rollover the log/trace files.

[ewels]

Ahh, right now I understand, sorry. Can Nextflow copy (with a new name) and delete the old file, instead of doing a move / rename?

Otherwise yes, deleting the old file is better than having to use a new bucket 👍🏻

[pditommaso]

Copy and deleting works for small files like this, but it should be implemented which would generalize this behavior which is very bad for common pipeline input/output files (hundreds of gigas..)

I see to add flag to overwrite logs instead or rolling them.

[pditommaso]

The overwrite option was added to report, timeline, trace and dag config scopes. Using this any remote files is deleted beforehand allowing the new files generated on resume to be stored remotely.