Rnaseq: ERROR ~ Cannot get property 'Galgal4' on null object

[Tichaboni]

Hi @ewels , @pditommaso , @marchoeppner I am trying to use the nf-core/rnaseq pipeline to analyze some RNAseq data. I am trying to run it on a docker using the following command,

nextflow run nf-core/rnaseq --reads '*_{1,2}.fastq.gz' --genome Galgal4 -profile docker

and am getting the following:

N E X T F L O W  ~  version 0.31.1
Launching `nf-core/rnaseq` [desperate_shirley] - revision: 44f1525d7a [master]
WARN: Access to undefined parameter `genomes` -- Initialise it to a default value eg. `params.genomes = some_value`
ERROR ~ Cannot get property 'Galgal4' on null object

 -- Check script 'main.nf' at line: 113 or see '.nextflow.log' file for more details

I have looked at the log but unfortunately I can not tell what I am looking for.
How do I Initialise it to a default value eg. params.genomes = some_value as suggested in the message above?
Thanks

[jun-wan]

@Tichaboni I think you need to use profile docker combined with standard, ie. -profile standard, docker since docker profile does not contain base and igenomes config. Could you have a try?

[ewels]

Yes exactly - this is similar to the problem reported in #33. Hopefully specifying the standard profile as suggested by @jun-wan will solve the problem. For the next release we'll try to refactor the way that the config works so that -profile docker works as expected 👍

[Tichaboni]

@ewels @jun-wan
Thanks for the replies. I tried with -profile standard but it is throwing another error

ERROR ~ Error executing process > 'trim_galore (SRR2017999)'
Caused by:
  Process requirement exceed available memory -- req: 16 GB; avail: 15.6 GB
$ `free` 
            total        used        free      shared  buff/cache   available
Mem:       16312664     3631752      306916      683984    12373996    12749232
Swap:       2097148           0     2097148

How should I proceed from here?
Thanks

[ewels]

Ok great, we're making progress! You can limit how much memory the pipeline requests with the --max_memory argument, see the docs: https://github.com/nf-core/rnaseq/blob/master/docs/usage.md#--max_memory

If you have 15.6 GB memory available, I'd recommend requesting something like 12GB. Note that STAR may not run with such limited memory, so you may need to use HiSAT2 for alignment instead (--aligner hisat2).

Phil

[Tichaboni]

@ewels @jun-wan Thanks for the swift response. I made the suggested changes, rerun the command as follows

nextflow run nf-core/rnaseq --reads '*moja_{1,2}.fastq.gz' --genome Galgal4 --max_memory 12.GB --aligner hisat2 -profile standard

and got the following error:

ERROR ~ Error executing process > 'trim_galore (SRR)'

Caused by:
  Process `trim_galore (SRR)` terminated with an error exit status (127)

Any suggestions?

Thanks

[apeltzer]

nextflow run nf-core/rnaseq --reads '*moja_{1,2}.fastq.gz' --genome Galgal4 --max_memory 12.GB --aligner hisat2 -profile docker,standard

..should run. Error code 127 means program not found, which is likely because you didn't specify to use docker/singularity. You can also run the same command with -profile standard,singularity if you can't run docker on your machine.

[ewels]

Yes, you need -profile standard,docker instead of just -profile standard or -profile docker in your two posts. This was in @jun-wan's answer above, but for me it wraps over two lines so it's easy to miss!

[Tichaboni]

@ewels @jun-wan @apeltzer Sorry I missed where it said I need to use both. I have changed the command accordingly. Tried it and it threw the error

docker: Got permission denied while trying to connect to the Docker daemon socket....

I used sudo (Just in case someone gets the same error they can do this) and it is running. I will report on the progress.
Thanks.

[ewels]

Sounds good, thanks! For future reference, what kind of system are you running on? eg. Laptop / server / operating system etc...

[apeltzer]

Make sure you have been added to the group docker on your system, e.g. by running this:

sudo gpasswd -a <yourusername> docker

Then logout, login again and you will most likely not need the sudo anymore.

[Tichaboni]

I am using Ubuntu 18.04.1 LTS (bionic) on a Lenovo ThinkCentre M910z.

Update:

[warm up] executor > local
[7c/e73193] Submitted process > fastqc (SRR)
[52/e8f0d3] Submitted process > get_software_versions
[76/4bd546] Submitted process > workflow_summary_mqc
[52/e8f0d3] NOTE: Process `get_software_versions` terminated with an error exit status (126) -- Error is ignored
[c3/350544] Submitted process > trim_galore (SRR)
[13/19c28b] Submitted process > output_documentation
ERROR ~ Error executing process > 'output_documentation'

Caused by:
  Process `output_documentation` terminated with an error exit status (126)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

[ewels]

Exit code 126 usually means command not executable. This is probably a permissions problem again. Did you manage to get @apeltzer's suggestion with the docker group to work?

[Tichaboni]

@ewels I added myself as a user to docker as suggested by @apeltzer. I have a large download in progress. I will log out after it's done and try again.
Thanks

[Tichaboni]

Hi @ewels @jun-wan @apeltzer
Update 2. (after log out and log in)

[warm up] executor > local
[1b/743dfd] Submitted process > fastqc (SRR)
[ed/d9fcef] Submitted process > workflow_summary_mqc
[20/7e0170] Submitted process > get_software_versions
[1b/743dfd] NOTE: Process `fastqc (SRR)` terminated with an error exit status (1) -- Execution is retried (1)
[cf/4494f0] Submitted process > trim_galore (SRR)
[cf/4494f0] NOTE: Process `trim_galore (SRR)` terminated with an error exit status (1) -- Execution is retried (1)
[2b/08dff3] Submitted process > output_documentation
[2b/08dff3] NOTE: Process `output_documentation` terminated with an error exit status (1) -- Execution is retried (1)
.....(truncated)....
[0e/e0ec8f] Re-submitted process > fastqc (SRR)
[0e/e0ec8f] NOTE: Process `fastqc (SRR)` terminated with an error exit status (1) -- Execution is retried (3)
[34/8f9df7] Re-submitted process > trim_galore (SRR)
[34/8f9df7] NOTE: Process `trim_galore (SRR)` terminated with an error exit status (1) -- Execution is retried (3)
[d8/9982ec] Re-submitted process > output_documentation
[d8/9982ec] NOTE: Process `output_documentation` terminated with an error exit status (1) -- Execution is retried (3)
[8c/a3682d] Re-submitted process > fastqc (SRR)
[5f/0e72c4] Re-submitted process > trim_galore (SRR)
ERROR ~ Error executing process > 'fastqc (SRR)'

`````

Caused by:
Process fastqc (SRR) terminated with an error exit status (1)
Command executed:
fastqc -q SRRmoja_1.fastq.gz SRRmoja_2.fastq.gz
Command exit status:
1
Command output:
(empty)
Command wrapper:
ln: failed to create symbolic link 'SRRmoja_1.fastq.gz': Function not implemented
`` Says I can continue with-resume` after fixing the problem. Would you please help me with finding the problem and ways of fixing it.

Cheers
p.s. I a changed into new working directory on an external drive (which has larger storage).

[apeltzer]

https://askubuntu.com/questions/1042833/failed-to-create-symbolic-link-function-not-implemented

(2nd answer: Is the filesystem you start from on a different partition than where your work directory is?)
What kind of filesystem do you run on? (ext?)

If the function is not implemented occurs, that means your filesystem running does potentially not support symbolic links.

Run uname -a so we know which linux kernel you run, maybe also some more information on the system would be helpful here...

[Tichaboni]

@apeltzer
uname -a
Linux bkimathi-ThinkCentre-M910z 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 GNU/Linux

[apeltzer]

Sorry off to vacation now - maybe check your filesystem?

[Tichaboni]

@jun-wan @ewels
Thanks for your help @apeltzer . The drive (FreeAgent) I was using originally uses exfat, the new drive (muthuri_st) uses exfat. I was able to create some space on FreeAgent and run the commands again. Unfortunately I ran out of memory at makeHISATindex (genome.fa
lsblk -f

sda                                                                   
├─sda1     vfat   EFI            5F66-17ED                            
└─sda2     exfat  muthuri_st     59D4-FAEA                            /media/bki
sdb                                                                   
└─sdb1     ntfs   FreeAgent Drive
                                 8C68286B682855EC                     /media/bki

Update 3:

[warm up] executor > local
[c0/1c1f73] Cached process > fastqc (SRR)
[1b/667444] Cached process > trim_galore (SRR)
[be/5c7314] Cached process > output_documentation
[59/f8aebf] Submitted process > workflow_summary_mqc
[eb/3b4c4c] Submitted process > get_software_versions
[43/fa5619] Cached process > makeHisatSplicesites (genes.gtf)
[HISAT2 index build] Available memory: 12 GB
[HISAT2 index build] Over 1 GB available, so using splice sites and exons in HISAT2 index
[e3/3f0a6c] Submitted process > makeHISATindex (genome.fa)
ERROR ~ Error executing process > 'makeHISATindex (genome.fa)'

Caused by:
  Process exceeded running time limit (5h)

More info, I have generated a the index and splice sites for this genome using hisat2. Is there a way I can skip the makeHISATindex step and use what I already have? (If that helps with the memory)
I will increase the running time to more than 5 and retry with -resume .
Any help will be appreciated.

[jun-wan]

Hi @Tichaboni

You can use --hisat2_index to specify the full path to your reference genome, please see the docs: https://github.com/nf-core/rnaseq/blob/master/docs/usage.md#--star_index---hisat2_index---fasta---gtf---bed12.

[Tichaboni]

Hi @jun-wan @ewels @apeltzer
Thanks for taking your time to assist me with this.
I increased the running time to 12 hours and used --hisat2_index option. The index is in form of 8 files. I included _.*.ht2 (will that capture all the index files?)
Update 4.
...

[HISAT2 index build] Over 1 GB available, so using splice sites and exons in HISAT2 index
[9b/935cf9] Cached process > makeHISATindex (genome.fa)
[aa/7c2c0b] Submitted process > hisat2Align (SRRmoja_1)
ERROR ~ Error executing process > 'hisat2Align (SRRmoja_1)'

Caused by:
  Process exceeded running time limit (1d)

Looks like there is progress.
Where do we go from here? by Charles Bradley

[ewels]

Great, now we're getting there! Though timing-out after a day? Ouch! 😖

I guess that must have been the third attempt too, as it usually gives a limit of 8 hours and multiplies by the number of retry attempts (8x3 = 24 hours).

I'm not sure that there is currently an easy way to increase this limit without changing some code now. So you'll need to start using a custom config file - see the nextflow docs. In short, I'd recommend creating a file called ~/.nextflow/config so that it is automatically applied to all runs on your computer. Alternatively you can make nextflow.config in your working directory, or make a file anywhere and reference it in the command with -c /path/to/config

Once you have this file, add the following:

process {
  withName:hisat2Align {
    time = 48.h
  }
}

This will increase the maximum running time for the alignment to 48 hours. You can change this to whatever makes sense for your system. The overwrites the default config set in the pipeline (see conf/base.config).

Fingers crossed! 🤞

Phil

[Tichaboni]

Hello @ewels @jun-wan @apeltzer
I made nextflow.config in the working directory and added the script.
The program has been running for more than 2 days and it seems to be stuck at hisat2Align.

=========================================
[warm up] executor > local
[25/1cc3ac] Submitted process > fastqc (SRR)
[2e/ff39f4] Submitted process > workflow_summary_mqc
[b5/21aaba] Submitted process > get_software_versions
[1c/fc729c] Submitted process > trim_galore (SRR)
[6c/0dae49] Submitted process > output_documentation
[26/6f26b7] Submitted process > makeHisatSplicesites (genes.gtf)
[HISAT2 index build] Available memory: 12 GB
[HISAT2 index build] Over 1 GB available, so using splice sites and exons in HISAT2 index
[9c/3e947f] Submitted process > makeHISATindex (genome.fa)
[09/19de5a] Submitted process > hisat2Align (SRRmoja_1)

Should I have more patience or is there some form of intervention we need to do?

[ewels]

Hmm, this is very suspicious. Can you paste the contents of work/09/19de5a[...]/.command.out please? Also the ls -la for that folder.

I guess that nextflow isn't detecting the exit of hisat2 for some reason, or it's hanging.

[Tichaboni]

The work directory has 31 sub-directories with two of them named 9 (9c and9f) their contents are directories with the following names (numbers) ab60f50fa2a338b41e9a347e8514d8 and dec86a83933e07791b58f1a395b80f respectively, which are empty. I do not have work/09/19de5a[...]/.command.out

[ewels]

Yup - nextflow executes each task in its own directory. The log message prefixes identify the hash which is name of that directory. So the reasoning for my question was to see what was happening with the hisat2Align task ([09/19de5a]). The log message truncates the second full hash, hence my [...]: you'll have to find a path that starts with that string.

The directories you found (9c/ab60f5 and 9f/dec86a) don't show up in the above pasted log, so must be from a different run of the pipeline.

Phil

[Tichaboni]

Hi @ewels
Sorry for the above mixup. I had checked in the wrong directory. The power went out on my computer last weekend. I rerun the command this morning. I have the the folder work/09/19de5a... but the new log also shows [16/0f5b74] Submitted process > hisat2Align (SRRmoja_1).

[warm up] executor > local
[25/1cc3ac] Cached process > fastqc (SRR)
[1c/fc729c] Cached process > trim_galore (SRR)
[6c/0dae49] Cached process > output_documentation
[11/c770a4] Submitted process > workflow_summary_mqc
[23/dfac39] Submitted process > get_software_versions
[26/6f26b7] Cached process > makeHisatSplicesites (genes.gtf)
[HISAT2 index build] Available memory: 12 GB
[HISAT2 index build] Over 1 GB available, so using splice sites and exons in HISAT2 index
[9c/3e947f] Cached process > makeHISATindex (genome.fa)
[16/0f5b74] Submitted process > hisat2Align (SRRmoja_1)

ls work/16/0f5b74..

genes.hisat2_splice_sites.txt  genome.hisat2_index.2.ht2  genome.hisat2_index.4.ht2  genome.hisat2_index.8.ht2  SRRmoja_1_val_1.fq.gz  where_are_my_files.txt
genome.hisat2_index.1.ht2      genome.hisat2_index.3.ht2  genome.hisat2_index.7.ht2  SRRmoja_1.bam              SRRmoja_2_val_2.fq.gz

ls -la

total 22
drwxrwxrwx 1 bkimathi bkimathi 4096 Sep 10 10:44 .
drwxrwxrwx 1 bkimathi bkimathi    0 Sep 10 10:44 ..
-rwxrwxrwx 1 bkimathi bkimathi    0 Sep 10 10:44 .command.begin
-rwxrwxrwx 1 bkimathi bkimathi    0 Sep 10 10:44 .command.err
-rwxrwxrwx 1 bkimathi bkimathi    0 Sep 10 10:44 .command.log
-rwxrwxrwx 1 bkimathi bkimathi    0 Sep 10 10:44 .command.out
-rwxrwxrwx 1 bkimathi bkimathi 4375 Sep 10 10:44 .command.run
-rwxrwxrwx 1 bkimathi bkimathi  421 Sep 10 10:44 .command.sh
-rwxrwxrwx 1 bkimathi bkimathi 3526 Sep 10 10:44 .command.stub
-rwxrwxrwx 1 bkimathi bkimathi  170 Sep 10 11:08 .command.trace
lrwxrwxrwx 1 bkimathi bkimathi  123 Sep 10 10:44 genes.hisat2_splice_sites.txt -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/26/6f26b73c9ad040944ff7651acb3236/genes.hisat2_splice_sites.txt
lrwxrwxrwx 1 bkimathi bkimathi  119 Sep 10 10:44 genome.hisat2_index.1.ht2 -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/9c/3e947f2d9188c54347656ac5faaf3e/genome.hisat2_index.1.ht2
lrwxrwxrwx 1 bkimathi bkimathi  119 Sep 10 10:44 genome.hisat2_index.2.ht2 -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/9c/3e947f2d9188c54347656ac5faaf3e/genome.hisat2_index.2.ht2
lrwxrwxrwx 1 bkimathi bkimathi  119 Sep 10 10:44 genome.hisat2_index.3.ht2 -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/9c/3e947f2d9188c54347656ac5faaf3e/genome.hisat2_index.3.ht2
lrwxrwxrwx 1 bkimathi bkimathi  119 Sep 10 10:44 genome.hisat2_index.4.ht2 -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/9c/3e947f2d9188c54347656ac5faaf3e/genome.hisat2_index.4.ht2
lrwxrwxrwx 1 bkimathi bkimathi  119 Sep 10 10:44 genome.hisat2_index.7.ht2 -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/9c/3e947f2d9188c54347656ac5faaf3e/genome.hisat2_index.7.ht2
lrwxrwxrwx 1 bkimathi bkimathi  119 Sep 10 10:44 genome.hisat2_index.8.ht2 -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/9c/3e947f2d9188c54347656ac5faaf3e/genome.hisat2_index.8.ht2
-rwxrwxrwx 1 bkimathi bkimathi    0 Sep 10 10:44 SRRmoja_1.bam
lrwxrwxrwx 1 bkimathi bkimathi  115 Sep 10 10:44 SRRmoja_1_val_1.fq.gz -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/1c/fc729c081e6d4c1826411a1ab9f845/SRRmoja_1_val_1.fq.gz
lrwxrwxrwx 1 bkimathi bkimathi  115 Sep 10 10:44 SRRmoja_2_val_2.fq.gz -> /media/bkimathi/FreeAgentDrive/Reads_Genome_Annotation/work/1c/fc729c081e6d4c1826411a1ab9f845/SRRmoja_2_val_2.fq.gz
lrwxrwxrwx 1 bkimathi bkimathi   92 Sep 10 10:44 where_are_my_files.txt -> /home/bkimathi/miniconda3/share/nextflow/assets/nf-core/rnaseq/assets/where_are_my_files.txt

I notice that two of the index files 5 and 6 are not listed.
p.s. Rookie question, .command.out is not a directory, how do I show it's contents?

[ewels]

but the new log also shows [16/0f5b74] Submitted process > hisat2Align (SRRmoja_1).

Yup, this is because you ran with -resume which skipped the successfully completed tasks (listed as Cached process) and restarted those that did not complete successfully (Submitted process). Because the hisat2Align didn't finish before, it was restarted and gets a new hash and new work directory.

Rookie question, .command.out is not a directory, how do I show it's contents?

These are just regular files, so you can use cat or less, or whatever you like to read their contents (I recently discovered bat, which is awesome).

The fact that .command.out is 0 bytes is strange. That means that it's totally empty - that there was no standard out from the command at all. It is also slightly strange about index files 5 and 7 missing too, as you say.

Ok - next investigative steps:

1. Could you do ls -la on the makeHISATindex work folder 9c/3e947f to check whether those index files 5 and 7 are generated?
2. Could you please paste the contents of the hisat2 command: $ cat .command.sh in work/09/19de5a...

I'm running out of ideas here though....

[MaxUlysse]

Hi, I'll be joining this quest as well.
Can you join also the .nextflow.log file it could contains some interesting information?

[apeltzer]

Back from vacation and will have a look as well - sharing the .nextflow.log, .command.* files would be great. Could you upload them to e.g. https://pastebin.com/ maybe?

[MaxUlysse]

@apeltzer I believe you can drag and drop files directly here, they are automatically uploaded in GitHub.

[apeltzer]

Yeah but only if you add .txt as file extension i believe? .log isn't accepted if IIRC :disappointed:

[Tichaboni]

@ewels @apeltzer @MaxUlysse
ls -la on 9c/3e947f

-rwxrwxrwx 1 bkimathi bkimathi 1280502476 Sep  5 20:29 genome.hisat2_index.0.rf
-rwxrwxrwx 1 bkimathi bkimathi     129732 Sep  5 15:03 genome.hisat2_index.1.ht2
-rwxrwxrwx 1 bkimathi bkimathi 1282807906 Sep  5 20:29 genome.hisat2_index.1.rf
-rwxrwxrwx 1 bkimathi bkimathi          4 Sep  5 15:03 genome.hisat2_index.2.ht2
-rwxrwxrwx 1 bkimathi bkimathi 1814041802 Sep  5 20:30 genome.hisat2_index.2.rf
-rwxrwxrwx 1 bkimathi bkimathi      97172 Sep  5 14:57 genome.hisat2_index.3.ht2
-rwxrwxrwx 1 bkimathi bkimathi  829817366 Sep  5 20:28 genome.hisat2_index.3.rf
-rwxrwxrwx 1 bkimathi bkimathi  248098489 Sep  5 14:57 genome.hisat2_index.4.ht2
-rwxrwxrwx 1 bkimathi bkimathi 1247347906 Sep  5 20:29 genome.hisat2_index.4.rf
-rwxrwxrwx 1 bkimathi bkimathi 2331284084 Sep  5 20:30 genome.hisat2_index.5.rf
-rwxrwxrwx 1 bkimathi bkimathi 2153147864 Sep  5 20:30 genome.hisat2_index.6.rf
-rwxrwxrwx 1 bkimathi bkimathi    6189132 Sep  5 14:57 genome.hisat2_index.7.ht2
-rwxrwxrwx 1 bkimathi bkimathi 2955818220 Sep  5 20:31 genome.hisat2_index.7.rf
-rwxrwxrwx 1 bkimathi bkimathi    1253816 Sep  5 14:57 genome.hisat2_index.8.ht2
drwxrwxrwx 1 bkimathi bkimathi          0 Sep  5 14:57 tmp

cat .command.sh in work/09/19de5a...

#!/bin/bash -euo pipefail
hisat2 -x genome.hisat2_index \
       -1 SRRmoja_1_val_1.fq.gz \
       -2 SRRmoja_2_val_2.fq.gz \
        \
       --known-splicesite-infile genes.hisat2_splice_sites.txt \
       --no-mixed \
       --no-discordant \
       -p 8 \
       --met-stderr \
       --new-summary \
       --summary-file SRRmoja_1.hisat2_summary.txt  \
       | samtools view -bS -F 4 -F 8 -F 256 - > SRRmoja_1.bam

cat .command.log in work/16/ 0f5b7... does not return anything.
cat .nextflow.log
I have uploaded to Pastebin
https://pastebin.com/j2sGLY7K

[MaxUlysse]

@Tichaboni

I have uploaded to Pastebin

Can you give us the address?

[Tichaboni]

@MaxUlysse
I have included it in the post above
https://pastebin.com/j2sGLY7K
but when I click it it leads to the issues page(copy paste on google works)

[MaxUlysse]

The link in your last comment links to the right page, the one in your previous comment links indeed to the issues page...
Strange

[MaxUlysse]

Wait, your pastebin doesn't seem to contain the right .nextflow.log

/home/bkimathi/miniconda3/bin/nextflow run skptic/lncRNA-Annotation-nf -profile docker

This is not the right pipeline I believe

[Tichaboni]

Sorry to ask this but how do I get the .nextflow.log specific to this workflow?

[MaxUlysse]

I'm guessing you're launching all your pipelines from the same location?

[Tichaboni]

The other pipeline was run from a different location. Is is not active right now. I am not sure but the two might have been installed from within the same location.

[ewels]

but when I click it it leads to the issues page(copy paste on google works)

Some dodgy markdown going on there I think. I edited the issue to fix it (markdown syntax is [nice text](http://url.com) and you did [http://url.com](nice text) - the link didn't exist, so GitHub linked to the issues page for some reasons.

Sorry to ask this but how do I get the .nextflow.log specific to this workflow?

This file is saved in the current work directory, wherever you launch the pipeline. It is independent from where the pipeline files themselves are kept.

[Tichaboni]

@ewels @apeltzer @MaxUlysse
Thanks for the edit @ewels
I have found the right log file and edited the Paste accordingly

[MaxUlysse]

Thanks a lot.
As far as I can tell from exploring the .nextflow.log there's no error, only this hisat2Align process ongoing and taking forever.

[Tichaboni]

Has anyone found what might be the issue here?

[ewels]

Not yet sorry. @pditommaso - any idea what's happening here? Pipeline runs fine until the hisat2Align step and then stalls forever. Task launches, but .command.out is empty and no sign that HiSAT2 is actually running.

Log: https://pastebin.com/j2sGLY7K, work directory listing above. Running locally with nextflow version 0.31.1.

[pditommaso]

Is there the .exitcode file in the task work directory?

p

[apeltzer]

Another idea, could you run like this:

export NXF_DEBUG=3

and then run your normal nextflow commandline? The log should contain more info then...

[ewels]

Is there the .exitcode file in the task work directory?

No, no .exitcode file (see dir listing).

[pditommaso]

The fact that the .exitcode is missing suggests the process is still running. Since it's running locally in that machine have you checked with a simple ps fx if the linux process exists ?

[Tichaboni]

Just an update before working on the suggestions above:

[16/0f5b74] Submitted process > hisat2Align (SRRmoja_1)
ERROR ~ Error executing process > 'hisat2Align (SRRmoja_1)'
Caused by:
  Process exceeded running time limit (2d)
Command executed:
  hisat2 -x genome.hisat2_index \
         -1 SRRmoja_1_val_1.fq.gz \
         -2 SRRmoja_2_val_2.fq.gz \
          \
         --known-splicesite-infile genes.hisat2_splice_sites.txt \
         --no-mixed \
         --no-discordant \
         -p 8 \
         --met-stderr \
         --new-summary \
         --summary-file SRRmoja_1.hisat2_summary.txt  \
         | samtools view -bS -F 4 -F 8 -F 256 - > SRRmoja_1.bam
Command exit status:
  -
Command output:
  (empty)

[Tichaboni]

@ewels @apeltzer @MaxUlysse @pditommaso
I have done the export and rerun the nextflow commandline

...
[9c/3e947f] Cached process > makeHISATindex (genome.fa)
[b2/e63455] Submitted process > hisat2Align (SRRmoja_1)

I have cat .nextflow.log and pasted it at
nextflow log with DEBUG
ps fx has the following

.......
8446 ?        Ssl    1:32  \_ /usr/lib/gnome-terminal/gnome-terminal-server
 8589 pts/1    Ss     0:00  |   \_ bash
 8311 pts/1    Sl+    0:15  |   |   \_ /home/bkimathi/miniconda3/bin/java -Xverify:none -Djava.
 8929 pts/1    S+     0:00  |   |       \_ /bin/bash -uex .command.run
 8964 pts/1    S+     0:00  |   |           \_ tee .command.out
 8965 pts/1    S+     0:02  |   |           \_ tee .command.err
 8966 pts/1    Sl+    0:03  |   |           \_ docker run -i --memory 12288m -e NXF_DEBUG=3 -v 
.........

I hope this sheds more light.
Thanks
BK

[pditommaso]

This means that jobs is still running (in the container). Now, you should try to understand if just slow or it hangs, either case it's a task level issue, not a nextflow problem.

Best way to troubleshoot it, change in the task work dir (after you killed it) and launch it using bash .command.run.

[ewels]

Closing for now. Please let us know if there are any updates!