Multiqc: Multiqc - conda issue

[jweger1988]

Hello All!

I'm running anaconda on on Ubuntu 14.04. I have the python 3.6 version and I am getting an error message when I try to run MultiQC - I have never had this problem before on an older version of ubuntu. If anyone can help me with this problem it would be greatly appreciated. I've tried with both python environments version 2.7 and 3.6

Here is the error message

[INFO   ]         multiqc : This is MultiQC v1.3
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching '.'
Searching 5 files..  [####################################]  100%
[INFO   ]         bowtie2 : Found 2 reports
[INFO   ]         multiqc : Compressing plot data
[WARNING]         multiqc : Previous MultiQC output found! Adjusting filenames..
[WARNING]         multiqc : Use -f or --force to overwrite existing reports instead
[INFO   ]         multiqc : Report      : multiqc_report_4.html
[INFO   ]         multiqc : Data        : multiqc_data_4

Traceback (most recent call last):
  File "/home/vignuzzilab/anaconda2/envs/py3-2/bin/multiqc", line 724, in <module>
    multiqc()
  File "/home/vignuzzilab/anaconda2/envs/py3-2/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/vignuzzilab/anaconda2/envs/py3-2/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/vignuzzilab/anaconda2/envs/py3-2/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vignuzzilab/anaconda2/envs/py3-2/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/vignuzzilab/anaconda2/envs/py3-2/bin/multiqc", line 572, in multiqc
    shutil.move(fn, config.data_dir)
  File "/home/vignuzzilab/anaconda2/envs/py3-2/lib/python2.7/shutil.py", line 316, in move
    copy2(src, real_dst)
  File "/home/vignuzzilab/anaconda2/envs/py3-2/lib/python2.7/shutil.py", line 145, in copy2
    copystat(src, dst)
  File "/home/vignuzzilab/anaconda2/envs/py3-2/lib/python2.7/shutil.py", line 112, in copystat
    os.utime(dst, (st.st_atime, st.st_mtime))
OSError: [Errno 1] Operation not permitted: '/media/Fastq/VR/test/multiqc_data_4/multiqc_data.json'

[lecorguille]

My 2 cents
@jweger1988 Are you sure that you have write access on your device?
Can you test something like that:

touch /media/Fastq/VR/test/multiqc_data_4/foobar

[jweger1988]

@lecorguille - Thanks for the tip. I ran that command and nothing happens, no message appears. Is this appropriate?

[lecorguille]

You should have created an empty file named foobar in /media/Fastq/VR/test/multiqc_data_4/. If you didn't get any message, it is because that wasn't a write access issue 😄

No more idea so far ...

[jweger1988]

OK, the empty file was created. Thanks for the suggestion!

[ewels]

Thanks both! As @lecorguille suggests, this error comes from when MultiQC is trying to move the files that it has created from a temporary directory into their final location.

A few questions / things to try:

1. How are you running MultiQC? Are you running it as your own user manually, or is it within a script or a pipeline or something?
2. You have multiple MultiQC outputs in your working directory, so some files must be created. Could you do a ls -la in a clean working directory before and after you run MultiQC please? Also ls -la multiqc_data so that we can see exactly which files are successfully created.
3. The file that's failing is new in MultiQC v1.3, which was released last week. This kind of thing makes me paranoid! :) Can you try installing MultiQC v1.2 and see if you get the same error?

Cheers,

Phil

[jweger1988]

1. I'm running it directly on the command line. Command is just the basic multiqc .
2. Before:

vignuzzilab@rasmus-OptiPlex-9010:/media/Fastq/VR/new_test$ ls -la
total 1
drwxrwxrwx 2 root root   0 nov.  10 11:07 .
drwxrwxrwx 2 root root   0 nov.  10 11:07 ..
-rwxrwxrwx 1 root root 204 nov.  10 11:07 Vp5L1_S15.log
-rwxrwxrwx 1 root root 204 nov.  10 11:07 Vp5L4_S18.log

After:

vignuzzilab@rasmus-OptiPlex-9010:/media/Fastq/VR/new_test$ ls -la
total 1
drwxrwxrwx 2 root root   0 nov.  10 11:07 .
drwxrwxrwx 2 root root   0 nov.  10 11:07 ..
-rwxrwxrwx 1 root root 204 nov.  10 11:07 Vp5L1_S15.log
-rwxrwxrwx 1 root root 204 nov.  10 11:07 Vp5L4_S18.log

and just in case in the multiqc_data directory:

(py3) vignuzzilab@rasmus-OptiPlex-9010:/media/Fastq/VR/new_test/multiqc_data$ ls -la
total 5
drwxrwxrwx 2 root root    0 nov.  10 11:08 .
drwxrwxrwx 2 root root    0 nov.  10 11:08 ..

1. It doesn't appear to be specific to version 1.3 - here is the error message I'm getting with 1.2

[WARNING]         multiqc : MultiQC Version v1.3 now available!
[INFO   ]         multiqc : This is MultiQC v1.2
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching '.'
Searching 2 files..  [####################################]  100%
[INFO   ]         bowtie2 : Found 2 reports
[INFO   ]         multiqc : Compressing plot data
[WARNING]         multiqc : Previous MultiQC output found! Adjusting filenames..
[WARNING]         multiqc : Use -f or --force to overwrite existing reports instead
[INFO   ]         multiqc : Report      : multiqc_report_1.html
[INFO   ]         multiqc : Data        : multiqc_data_1

Traceback (most recent call last):
  File "/home/vignuzzilab/anaconda2/envs/py3/bin/multiqc", line 4, in <module>
    __import__('pkg_resources').run_script('multiqc==1.2', 'multiqc')
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/pkg_resources/__init__.py", line 748, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1517, in run_script
    exec(code, namespace, namespace)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/EGG-INFO/scripts/multiqc", line 723, in <module>
    multiqc()
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/EGG-INFO/scripts/multiqc", line 571, in multiqc
    shutil.move(fn, config.data_dir)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/shutil.py", line 316, in move
    copy2(src, real_dst)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/shutil.py", line 145, in copy2
    copystat(src, dst)
  File "/home/vignuzzilab/anaconda2/envs/py3/lib/python2.7/shutil.py", line 112, in copystat
    os.utime(dst, (st.st_atime, st.st_mtime))
OSError: [Errno 1] Operation not permitted: '/media/Fastq/VR/new_test/multiqc_data_1/multiqc_bowtie2.txt'

Hopefully this is helpful and thank you very much for your help.

James

[ewels]

Thanks! I don't understand that multiqc_data isn't there after you've run MultiQC, yet you are able to cd into it and run ls -la?

[ewels]

Next request - could you try running again but with the -v flag to get the verbose log output please? It'll be long, so best to save it to a file and attach that to the issue..

multiqc . -v >> multiqc.log 2>&1

[jweger1988]

Shoot I made a mistake, here is the ls -la after running multiqc .

(py3) vignuzzilab@rasmus-OptiPlex-9010:/media/Fastq/VR/new_test$ ls -la
total 1
drwxrwxrwx 2 root root   0 nov.  10 11:18 .
drwxrwxrwx 2 root root   0 nov.  10 11:07 ..
drwxrwxrwx 2 root root   0 nov.  10 11:08 multiqc_data
drwxrwxrwx 2 root root   0 nov.  10 11:18 multiqc_data_1
-rwxrwxrwx 1 root root 204 nov.  10 11:07 Vp5L1_S15.log
-rwxrwxrwx 1 root root 204 nov.  10 11:07 Vp5L4_S18.log

[jweger1988]

multiqc.log

Here is that log. Interestingly, with that I get a file called /media/Fastq/VR/new_test/multiqc_data_2/multiqc_bowtie2.txt that has the relevant information.

Thanks!

[marcelm]

My guess would be a conflict or problem with mounting options for the target /media/ directory. The 'permission denied' error is thrown when os.utime() is used to set access and modification times for an already copied file.

What does mount | grep /media/Fastq say?

[ewels]

Hi @jweger1988,

Any updates? I may close this issue soon as we are unable to replicate the error and it seems increasingly likely that it's a system specific problem..

Cheers,

Phil

[jweger1988]

Thanks, Phil. Still doesn't work but I also believe it to be system specific. I really appreciate all of your help.

[ewels]

Ok, thanks for the reply @jweger1988 - did you run the command suggested by @marcelm?

[mackellardrew]

Hey @ewels et al., I know this is an old thread, but I thought I'd chime in that I found this thread today when googling essentially the same error. Multiqc was failing at the call to copy/move files; it was successfully generating the multiqc_data dir and multiqc_data.json file, but no others. The end of the trace was:

File "/home/dcm0303/python/lib/python3.7/shutil.py", line 205, in copystat
    follow_symlinks=follow)
PermissionError: [Errno 1] Operation not permitted

@marcelm 's comment prompted me to check how the dir I was working in was mounted; I had mounted it via an /etc/fstab line (it's a Windows shared drive operating on Ubuntu via the WSL) that didn't pass a uid or gid, and so was listing the owner as root. Apparently multiqc or one of its dependencies don't like that. Changing how it was mounted resolved the problem in my case. Just FYI.

[Mushin]

Hey @ewels et al., I know this is an old thread, but I thought I'd chime in that I found this thread today when googling essentially the same error. Multiqc was failing at the call to copy/move files; it was successfully generating the multiqc_data dir and multiqc_data.json file, but no others. The end of the trace was:

File "/home/dcm0303/python/lib/python3.7/shutil.py", line 205, in copystat
    follow_symlinks=follow)
PermissionError: [Errno 1] Operation not permitted

@marcelm 's comment prompted me to check how the dir I was working in was mounted; I had mounted it via an /etc/fstab line (it's a Windows shared drive operating on Ubuntu via the WSL) that didn't pass a uid or gid, and so was listing the owner as root. Apparently multiqc or one of its dependencies don't like that. Changing how it was mounted resolved the problem in my case. Just FYI.

Please can you advise how you fixed the mount in this case?

[mackellardrew]

@Mushin I just had to pass my user ID and group ID to the line in /etc/fstab that mounts the drive where I was storing my reads. It originally read

H: /mnt/h drvfs defaults 0 0

and is now

H: /mnt/h drvfs defaults,uid=1000,gid=1000 0 0

1000 is a pretty typical value for each, but you could also substitute uid=$(id -u),gid=$(id -g) if you're unsure of your own situation.

[j-andrews7]

For anyone mounting using the mount command directly, the above becomes:

sudo mount -t drvfs -o defaults,uid=1000,gid=1000 H: /mnt/h