Nextflow: Error when trying to use grep in Nextflow

[Z-Zen]

Bug report

Expected behavior and actual behavior

• Expected behavior: grep commands should return the lines if a string matches the start of the line.
• actual behavior: Nextflow gives an error without any details on why this happened. However, the command exit status is 1 but gives the expected behavior in the command output.

Steps to reproduce the problem

Create a file aaaa.txt that contains the following:

dawdsdv
grfghhrth
dsvfskdmgf
test1 dadgfdgdfgdfg
test2 awdawdawd
sdgsdg test3
fbfgbfgb

#!/usr/bin/env nextflow

def samplesList = ["test1", "test2", "test3"]

process testProcess {
    echo true

    script:
        """
        for i in ${samplesList.join(' ')}
        do
            grep -P "^\$i" ~/Documents/aaaa.txt
        done 
        """
}

Program output

mai-14 18:04:50.426 [main] DEBUG nextflow.cli.Launcher - $> nextflow run test.nf
mai-14 18:04:50.637 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 19.04.1
mai-14 18:04:50.650 [main] INFO  nextflow.cli.CmdRun - Launching `test.nf` [thirsty_cajal] - revision: 563dfc8a8e
mai-14 18:04:50.670 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /media/Nextflow-ANGES/nextflow.config
mai-14 18:04:50.671 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /media/Nextflow-ANGES/nextflow.config
mai-14 18:04:50.704 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
mai-14 18:04:51.253 [main] DEBUG nextflow.Session - Session uuid: 3d0e8dd8-4d95-4032-b052-3063688e5170
mai-14 18:04:51.253 [main] DEBUG nextflow.Session - Run name: thirsty_cajal
mai-14 18:04:51.254 [main] DEBUG nextflow.Session - Executor pool size: 30
mai-14 18:04:51.269 [main] DEBUG nextflow.cli.CmdRun -
  Version: 19.04.1 build 5072
  Modified: 03-05-2019 12:29 UTC (14:29 CEST)
  System: Linux 4.9.0-7-amd64
  Runtime: Groovy 2.5.6 on OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b01-1~deb9u1-b01
  Encoding: UTF-8 (UTF-8)
  Process: 4074@custom
  CPUs: 30 - Mem: 30.7 GB (18.7 GB) - Swap: 4.8 GB (4.8 GB)
mai-14 18:04:51.307 [main] DEBUG nextflow.Session - Work-dir: /media/Nextflow-ANGES/work [nfs]
mai-14 18:04:51.468 [main] DEBUG nextflow.Session - Session start invoked
mai-14 18:04:51.473 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
mai-14 18:04:51.476 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
mai-14 18:04:51.690 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
mai-14 18:04:51.759 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
mai-14 18:04:51.760 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
mai-14 18:04:51.766 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
mai-14 18:04:51.768 [main] INFO  nextflow.executor.Executor - [warm up] executor > local
mai-14 18:04:51.773 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=30; memory=30.7 GB; capacity=30; pollInterval=100ms; dumpInterval=5m
mai-14 18:04:51.776 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: LocalPollingMonitor
mai-14 18:04:51.777 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
mai-14 18:04:51.780 [main] DEBUG nextflow.executor.Executor - Invoke register for executor: local
mai-14 18:04:51.843 [main] DEBUG nextflow.Session - >>> barrier register (process: testProcess)
mai-14 18:04:51.846 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > testProcess -- maxForks: 30
mai-14 18:04:51.878 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
mai-14 18:04:51.878 [main] DEBUG nextflow.Session - Session await
mai-14 18:04:52.021 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
mai-14 18:04:52.026 [Task submitter] INFO  nextflow.Session - [1e/3d1ac2] Submitted process > testProcess (null)
mai-14 18:04:52.081 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: testProcess (null); status: COMPLETED; exit: 1; error: -; workDir: /media/Nextflow-ANGES/work/1e/3d1ac2d586c34051251b0aecfbc851]
mai-14 18:04:52.112 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'testProcess (null)'

Caused by:
  Process `testProcess (null)` terminated with an error exit status (1)

Command executed:

  for i in test1 test2 test3
  do
        grep -P "^$i" ~/Documents/awdawd.txt
  done

Command exit status:
  1

Command output:
  test1 dadgfdgdfgdfg
  test2 awdawdawd

Command wrapper:
  test1 dadgfdgdfgdfg
  test2 awdawdawd

Work dir:
  /media/Nextflow-ANGES/work/1e/3d1ac2d586c34051251b0aecfbc851

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
mai-14 18:04:52.121 [Actor Thread 2] DEBUG nextflow.Session - <<< barrier arrive (process: testProcess)
mai-14 18:04:52.122 [main] DEBUG nextflow.Session - Session await > all process finished
mai-14 18:04:52.122 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `testProcess (null)` terminated with an error exit status (1)
mai-14 18:04:52.144 [main] DEBUG nextflow.Session - Session await > all barriers passed
mai-14 18:04:52.144 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
mai-14 18:04:52.150 [main] DEBUG nextflow.trace.StatsObserver - Workflow completed > WorkflowStats[succeedCount=0; failedCount=1; ignoredCount=0; cachedCount=0; succeedDuration=0ms; failedDuration=3ms; cachedDuration=0ms]
mai-14 18:04:52.333 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
mai-14 18:04:52.358 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Environment

• Nextflow version: version 19.04.1 build 5072
• Java version: openjdk version "1.8.0_212"
• Operating system: debian 9

Additional context

The given command works in bash without any issue but when executed in Nextflow, it fails.

[micans]

The issue is that the match does not succeed for pattern test3; thus grep fails, thus the whole script section fails. If you change the grep line to grep -P "^\$i" ~/Documents/aaaa.txt || true this should work.

[Z-Zen]

The issue is that the match does not succeed for pattern test3; thus grep fails, thus the whole script section fails. If you change the grep line to grep -P "^\$i" ~/Documents/aaaa.txt || true this should work.

@micans yes I know that this works but this is not supposed to be the right way of how Nextflow handles things, right ? I also got another case that happens with MongoDB queries. If I don't put the || true, it gives the same error. I spent lots of time on it before I could figure it out so I though other people may encounter the same problem.
I mean I am not sure if it could be fixed but that would be awesome !

[pditommaso]

NF has a strict error checking by default. If can change it setting

process.shell = ['/bin/bash','-u']

or whatever you prefer.

[Z-Zen]

NF has a strict error checking by default. If can change it setting

process.shell = ['/bin/bash','-u']

or whatever you prefer.

@pditommaso ahhhh so that's why ! I understand now ! Thanks for this useful information. This issue can be closed now !

[pditommaso]

Good, closing then.