Tools: nf-core download cannot find nextflow
Hi,
I'm getting an error that states: AssertionError: It looks like Nextflow is not installed. It is required for most nf-core functions.
However nextflow has been installed in my system & is available in the path.
I can download these manually, however posting here FYI.
(nfcore) [burosj01@li03c03]$ nf-core download rnaseq --singularity
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
INFO: Saving rnaseq
Pipeline release: 1.3
Pull singularity containers: Yes
Output directory: nf-core-rnaseq-1.3
INFO: Downloading workflow files from GitHub
Traceback (most recent call last):
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/nf_core/utils.py", line 49, in fetch_wf_config
nfconfig_raw = subprocess.check_output(['nextflow', 'config', '-flat', wf_path], stderr=devnull)
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nextflow': 'nextflow'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/sc/hydra/work/burosj01/conda/envs/nfcore/bin/nf-core", line 273, in <module>
nf_core_cli()
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/sc/hydra/work/burosj01/conda/envs/nfcore/bin/nf-core", line 129, in download
dl.download_workflow()
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/nf_core/download.py", line 69, in download_workflow
self.find_container_images()
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/nf_core/download.py", line 182, in find_container_images
self.config = nf_core.utils.fetch_wf_config(os.path.join(self.outdir, 'workflow'))
File "/sc/hydra/work/burosj01/conda/envs/nfcore/lib/python3.6/site-packages/nf_core/utils.py", line 52, in fetch_wf_config
raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.")
AssertionError: It looks like Nextflow is not installed. It is required for most nf-core functions.
Here is the info for nextflow:
(nfcore) [burosj01@li03c03]$ which nextflow
~/bin/nextflow
(nfcore) [burosj01@li03c03]$ nextflow info
Version: 19.04.1 build 5072
Modified: 03-05-2019 12:29 UTC (08:29 EDT)
System: Linux 3.10.0-957.el7.x86_64
Runtime: Groovy 2.5.6 on OpenJDK 64-Bit Server VM 11-ea+28
Encoding: UTF-8 (UTF-8)
I am running nf-core in a conda environment, installed via pip.
(nfcore) [burosj01@li03c03]$ conda info
active environment : nfcore
active env location : /sc/hydra/work/burosj01/conda/envs/nfcore
shell level : 1
user config file : /hpc/users/burosj01/.condarc
populated config files : /hpc/users/burosj01/.condarc
conda version : 4.6.14
conda-build version : 3.17.6
python version : 3.7.1.final.0
base environment : /hpc/packages/minerva-centos7/anaconda3/2018.12 (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /sc/hydra/work/burosj01/conda/pkgs
envs directories : /sc/hydra/work/burosj01/conda/envs
/hpc/users/burosj01/.conda/envs
/hpc/packages/minerva-centos7/anaconda3/2018.12/envs
platform : linux-64
user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.1 Linux/3.10.0-957.el7.x86_64 centos/7.6.1810 glibc/2.17
UID:GID : 22153:31227
netrc file : None
offline mode : False
jburos
All 13 comments
Thanks for reporting this - I think that could be because Nextflow isn't in the same conda environment and rather installed in the ~/bin/ folder - could you install nextflow to the same nfcore environment? If that resolves the issue, we might have to investigate further why this causes issues then :+1:
apeltzer
on 22 Jun 2019
Thanks @apeltzer - I moved the binary to the conda environment's bin directory & this resolved the issue.
(nfcore) [burosj01@li03c03 ]$ cp ~/bin/nextflow /sc/hydra/work/burosj01/conda/envs/nfcore/bin/
(nfcore) [burosj01@li03c03 ]$ which nextflow
/sc/hydra/work/burosj01/conda/envs/nfcore/bin/nextflow
(nfcore) [burosj01@li03c03 ]$ nf-core download rnaseq --singularity
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
INFO: Saving rnaseq
Pipeline release: 1.3
Pull singularity containers: Yes
Output directory: nf-core-rnaseq-1.3
INFO: Downloading workflow files from GitHub
INFO: Downloading 1 singularity container
INFO: Building singularity image from dockerhub: docker://nfcore/rnaseq:1.3
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO: Starting build...
Getting image source signatures
[... truncated ...]
I have a feeling that if we add shell=True to the subprocess command, then this will resolve this problem, without having to move the nextflow binary around. It's a famous security issue though, but I think it should be fine in this context. Anyone have any thoughts? @nf-core/core
ewels
on 3 Jul 2019
I guess it should be fine for this context - as we expect users of nf-core tools anyways to have permissions on the system they want to use the application 馃憤
apeltzer
on 3 Jul 2019
It seems that it has been fixed in the current dev branch (as of 61a2ccc):
With nextflow in /usr/local/bin, but not in the current conda environment (here test):
(test)$ nf-core download rnaseq
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/tools version 1.10.dev0
INFO: Saving rnaseq
Pipeline release: 1.4.2
Pull singularity containers: No
Output file: nf-core-rnaseq-1.4.2.tar.gz
INFO: Downloading workflow files from GitHub
INFO: Downloading centralised configs from GitHub
INFO: Compressing download..
INFO: Command to extract files: tar -xzf nf-core-rnaseq-1.4.2.tar.gz
INFO: MD5 checksum for nf-core-rnaseq-1.4.2.tar.gz: d47cf8ce566ae35e3fe501dfcb761fbd
maxibor
on 14 Jul 2020
I'm a bit lost sorry @maxibor - what's wrong here?
ewels
on 14 Jul 2020
I don't think that this issue has been solved yet - the line of code that triggered the exception above still doesn't use shell=True:
https://github.com/nf-core/tools/blob/61a2ccc6e92bbfeb7c76e86eeda9a3113d1b6566/nf_core/utils.py#L69
ewels
on 14 Jul 2020
@ewels if you use Popen you can pass in an env arg; could you just pass in the environment from the parent Python instance to fix this? That would avoid needing shell=True
stevekm
on 15 Jul 2020
Ah sure, sounds good! 馃憤
ewels
on 15 Jul 2020
@ewels
Nextflow in /usr/local/bin
$ ls -1 /usr/local/bin | grep nextflow
nextflow
But not in currently activated environment:
$ conda list | grep nextflow
Yet no issue :
$ nf-core download rnaseq --singularity
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/tools version 1.10.dev0
INFO: Saving rnaseq
Pipeline release: 1.4.2
Pull singularity containers: Yes
Output file: nf-core-rnaseq-1.4.2.tar.gz
INFO: Downloading workflow files from GitHub
INFO: Downloading centralised configs from GitHub
INFO: Downloading 1 singularity container
INFO: Building singularity image from Docker Hub: docker://nfcore/rnaseq:1.4.2
ERROR: Singularity is not installed!
INFO: Compressing download..
INFO: Command to extract files: tar -xzf nf-core-rnaseq-1.4.2.tar.gz
INFO: MD5 checksum for nf-core-rnaseq-1.4.2.tar.gz: c879174f2ae9935d4c03c4d3176aad14
$ git log | head
commit 400e33e3f1e4f96ef49a427f04b8e508dec19e64
Merge: dc8d453 7874b8b
Author: Phil Ewels <[email protected]>
Date: Thu Jul 16 09:38:40 2020 +0200
Merge pull request #680 from ewels/rich-lint-progress
Slight tweak to main lint results output
commit 7874b8b17a78fe20b0f54bf5e3c0e2bd84ec28bb
ok - so it was fixed at some point you mean?
ewels
on 16 Jul 2020
Tried confirming in a fresh and clean Virtual Machine and can't reproduce either :-(
apeltzer
on 16 Jul 2020
Not sad face - happy face! That means we can close this issue 馃槄
ewels
on 16 Jul 2020
Related issues
lucacozzuto
路
5Comments
ewels
路
5Comments
matq007
路
7Comments
sven1103
路
3Comments
MaxUlysse
路
6Comments
Most helpful comment
Thanks @apeltzer - I moved the binary to the conda environment's
bindirectory & this resolved the issue.