Multiqc: Network Share Problem: Invalid cross-device link when calling os.rename

Created on 26 Jul 2018 · 6Comments · Source: ewels/MultiQC

Dear @ewels ,

First off all thanks for your neat tool and all the effort you put in it!

Description of bug:
When using multiqc on data stored on a CIFS share I get an error. CIFS shares do not support symbolic links. However, the problem here seems to pertain to the usage of os.rename() for multiqc_sources.txt if temporary (/tmp) and output directory (CIFS) reside on different shares. _Is there an option to specify the multiqc temporary directory via a commandline option?_ I would try to map it to the CIFS share.

MultiQC Error log:

[2018-07-26 14:06:51,065] multiqc.modules.fastqc.fastqc                      [INFO   ]  Found 2 reports
[2018-07-26 14:06:51,132] multiqc                                            [DEBUG  ]  No samples found: clusterflow
[2018-07-26 14:06:51,135] multiqc                                            [DEBUG  ]  No samples found: bcl2fastq
[2018-07-26 14:06:51,139] multiqc                                            [DEBUG  ]  No samples found: interop
[2018-07-26 14:06:51,171] multiqc                                            [INFO   ]  Compressing plot data
[2018-07-26 14:06:51,673] multiqc                                            [INFO   ]  Report      : qc/LB18-0784-Sequencing-Analysis_multiqc_report.html
[2018-07-26 14:06:51,673] multiqc                                            [INFO   ]  Data        : qc/LB18-0784-Sequencing-Analysis_multiqc_report_data
[2018-07-26 14:06:51,674] multiqc                                            [DEBUG  ]  Moving data file from '/tmp/tmp7gzkuz5a/multiqc_data/multiqc_sources.txt' to 'qc/LB18-0784-Sequencing-Analysis_multiqc_report_data'
Traceback (most recent call last):
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/shutil.py", line 544, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmp7gzkuz5a/multiqc_data/multiqc_sources.txt' -> 'qc/LB18-0784-Sequencing-Analysis_multiqc_report_data/multiqc_sources.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/bin/multiqc", line 767, in <module>
    multiqc()
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/bin/multiqc", line 611, in multiqc
    shutil.move(fn, config.data_dir)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/shutil.py", line 558, in move
    copy_function(src, real_dst)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/shutil.py", line 258, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/shutil.py", line 201, in copystat
    follow_symlinks=follow)
PermissionError: [Errno 1] Operation not permitted

MultiQC run details (please complete the following):

Command used to run MultiQC: multiqc --force --interactive --verbose --outdir qc/ .
MultiQC Version: MultiQC v1.5
Operating System: Linux s-labb-ngs02 4.4.0-131-generic #157-Ubuntu SMP
Python Version: Python 3.5.4
Method of MultiQC installation: conda

Thanks in advance!

bug

Source

your-highness

Most helpful comment

Fixed in #1130

ewels on 29 Apr 2020

🚀2

All 6 comments

I found a workaround on my Linux machine for the "Invalid cross-device link" Error:
export TMPDIR=/mnt/cifs/tmp

However, now I get the following "Operation not permitted" Error:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/bin/multiqc", line 767, in <module>
    multiqc()
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/bin/multiqc", line 644, in multiqc
    copy_tree(template_mod.template_dir, tmp_dir)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/distutils/dir_util.py", line 163, in copy_tree
    dry_run=dry_run)
  File "/opt/conda/miniconda3/envs/HUM-NGSDiag/lib/python3.5/distutils/file_util.py", line 158, in copy_file
    os.utime(dst, (st[ST_ATIME], st[ST_MTIME]))
PermissionError: [Errno 1] Operation not permitted

I assume this is related to the CIFS share not allowing to set file attributes.

For now setting the output directory to a XFS mount works as a charme and I will move the report after computation.

your-highness on 26 Jul 2018

Hi @your-highness (nice GitHub handle!)

Sorry to hear that you were having difficulties, and glad that you got them sorted. I've not come across this before, but your fix makes sense.

I'll close this now, but let me know if there's anything that I can help with 👍

Phil

ewels on 30 Jul 2018

Ran into this today too - its apparently the case that somewhere you use os.rename() which doesn't work between two filesystems, a shutil.move() though would be able to do this properly. Might actually work on this if you re-open and assign me ;-)

apeltzer on 29 Jan 2020

Hi,
I also ran into this problem today with multiqc Version 1.8.
The solution with creating the output somewhere else (e.g. in the /tmp directory and then moving it works.
@apeltzer After looking a little further the problem seems to be the shutil.move command (Python version 3.6 and it should have been fixed in 3.3). And it breaks while handling this error (see also error log of @your-highness)

alex-seitz on 27 Feb 2020

👍2

@ewels, @apeltzer Let's reopen this. We could slightly modify source code (replace shutil.move with another command) and will get rid of this error. I know that there is a workaround with TMPDIR, but sometimes it isn't convenient to use it. E.g. I run MultiQC from Snakemake pipeline using LSF cluster submission system with workers from Docker containers. I could struggle with all these things and force them not to change TMPDIR to something more appropriate but I prefer not to use all these complicated environment hacks and fix MultiQC to work out of the box.

Same issue on python 3.6/3.8 with MultiQC 1.7/1.8

shutil.move command doesn't work correctly if source and dest are in different file systems, so fails with:

[2020-03-19 10:22:44,535] multiqc                                            [DEBUG  ]  Latest MultiQC version is v1.8
[2020-03-19 10:22:44,535] multiqc                                            [INFO   ]  This is MultiQC v1.8 (3508b91)
...
[2020-03-19 10:22:46,247] multiqc                                            [INFO   ]  Data        : multiqc_data
[2020-03-19 10:22:46,249] multiqc                                            [DEBUG  ]  Moving data file from '/tmp/tmpzw8x7unp/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt' to '/scratch1/fs1/m'
Traceback (most recent call last):
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 780, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpzw8x7unp/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt' -> '/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/multiqc_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/bin/multiqc", line 6, in <module>
    from multiqc.__main__ import multiqc
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/site-packages/multiqc/__main__.py", line 44, in <module>
    multiqc.run_cli(prog_name='multiqc')
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/site-packages/multiqc/multiqc.py", line 204, in run_cli
    sys_exit_code = run(
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/site-packages/multiqc/multiqc.py", line 749, in run
    shutil.move(fn, config.data_dir)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 794, in move
    copy_function(src, real_dst)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 427, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 370, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 322, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt'

So this is acutally a python issue but there is a workaround. This issue already discussed in https://stackoverflow.com/a/43967659/418358 (see comments), python > 3.3 is supposed to fix it but it still doesn't work in my environment using python 3.6 or 3.8 and it doesn't work in for some other people (see comments to the stackoverflow issue)

Minimal test case:

# issue_806.py
import shutil

fn = '/tmp/tmpzw8x7unp/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt'
dd = '/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt'

shutil.move(fn, dd)

Fails with:

$ python3.8 issue_806.py
Traceback (most recent call last):
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 780, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpzw8x7unp/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt' -> '/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/multiqc_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "foo.py", line 9, in <module>
    shutil.move(fn, dd)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 794, in move
    copy_function(src, real_dst)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 427, in copy2
    copystat(src, dst, follow_symlinks=follow_symlinks)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 370, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/.snakemake/conda/482ac789/lib/python3.8/shutil.py", line 322, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt'

According to StackOverflow suggestion this workaround works ok:

# issue_806_workaround.py
import os
import shutil

fn = '/tmp/tmpzw8x7unp/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt'
dd = '/scratch1/fs1/martyomov/rcherniatchik/aging-rnaseq-smk-pipeline-small/multiqc_data/multiqc_qualimap_bamqc_genome_results.txt'

# shutil.move(fn, dd)
shutil.copy(fn, dd)
os.remove(fn)

Works OK!