[richam@milou1 skr盲t]$ multiqc /proj/b2013064/private/nobackup/rickard/RNA_clusterflow/Mouse_vs_Mouse_SE/
[INFO ] multiqc : This is MultiQC v0.5
Traceback (most recent call last):
File "/sw/comp/python/2.7.6_milou/lib/python2.7/logging/__init__.py", line 874, in emit
stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 121: ordinal not in range(128)
Logged from file multiqc, line 138
[INFO ] multiqc : Searching '/proj/b2013064/private/nobackup/rickard/RNA_clusterflow/Mouse_vs_Mouse_SE/'
[INFO ] feature_counts : Found 14 reports
[INFO ] tophat : Found 14 reports
[INFO ] cutadapt : Found 14 reports
[INFO ] fastqc : Found 21 reports
/sw/comp/python/2.7.6_milou/lib/python2.7/genericpath.py:71: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
s1 = min(m)
Traceback (most recent call last):
File "/sw/apps/bioinfo/MultiQC/0.5/milou/bin/multiqc", line 416, in <module>
multiqc()
File "/sw/apps/bioinfo/MultiQC/0.5/milou/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/sw/apps/bioinfo/MultiQC/0.5/milou/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/sw/apps/bioinfo/MultiQC/0.5/milou/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/sw/apps/bioinfo/MultiQC/0.5/milou/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/sw/apps/bioinfo/MultiQC/0.5/milou/bin/multiqc", line 317, in multiqc
logger.info("Report : {}".format(os.path.relpath(config.output_fn)))
File "/sw/comp/python/2.7.6_milou/lib/python2.7/posixpath.py", line 426, in relpath
i = len(commonprefix([start_list, path_list]))
File "/sw/comp/python/2.7.6_milou/lib/python2.7/genericpath.py", line 71, in commonprefix
s1 = min(m)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
It seems that simply standing in a directory with 脜/脛/脰 or similar in the name casuses the error. It does not have to be the target directory
Thanks!
Hammarn
All 10 comments
Yay, encoding errors - my favourite! Best of all, I thought I'd managed to recreate your error but it's actually somewhere else:
[INFO ] multiqc : This is MultiQC v0.6.dev0
[INFO ] multiqc : Searching 'data'
[INFO ] feature_counts : Found 6 reports
[INFO ] star : Found 6 reports
[INFO ] cutadapt : Found 6 reports
[INFO ] fastq_screen : Found 6 reports
[INFO ] fastqc : Found 6 reports
Traceback (most recent call last):
File "/Users/philewels/.miniconda/envs/work/bin/multiqc", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/Users/philewels/Work/MultiQC/scripts/multiqc", line 416, in <module>
multiqc()
File "/Users/philewels/.miniconda/envs/work/lib/python2.7/site-packages/click/core.py", line 700, in __call__
return self.main(*args, **kwargs)
File "/Users/philewels/.miniconda/envs/work/lib/python2.7/site-packages/click/core.py", line 680, in main
rv = self.invoke(ctx)
File "/Users/philewels/.miniconda/envs/work/lib/python2.7/site-packages/click/core.py", line 873, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/philewels/.miniconda/envs/work/lib/python2.7/site-packages/click/core.py", line 508, in invoke
return callback(*args, **kwargs)
File "/Users/philewels/Work/MultiQC/scripts/multiqc", line 298, in multiqc
report.data_sources_tofile()
File "/Users/philewels/Work/MultiQC/multiqc/utils/report.py", line 250, in data_sources_tofile
body = '\n'.join(["\t".join(l) for l in lines])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 53: ordinal not in range(128)
ewels
on 8 Apr 2016
The latest bcbio-nextgen/tests/integration/test_automated_analysis/test_2_hisat2 failed at:
[2017-06-29T16:03Z] Run multiqc
[2017-06-29T16:03Z] [INFO ] multiqc : This is MultiQC v1.0
[2017-06-29T16:03Z] [INFO ] multiqc : Template : default
[2017-06-29T16:03Z] Searching 12 files..
[2017-06-29T16:03Z] [INFO ] samtools : Found 1 stats reports
[2017-06-29T16:03Z] [INFO ] fastqc : Found 1 reports
[2017-06-29T16:03Z] [INFO ] multiqc : Compressing plot data
[2017-06-29T16:03Z] [INFO ] multiqc : Report : ../../bcbiotx/tmpSObcmy/multiqc_report.html
[2017-06-29T16:03Z] [INFO ] multiqc : Data : ../../bcbiotx/tmpSObcmy/multiqc_data
[2017-06-29T16:03Z] Traceback (most recent call last):
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/bin/multiqc", line 4, in <module>
[2017-06-29T16:03Z] __import__('pkg_resources').run_script('multiqc==1.0', 'multiqc')
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 744, in run_script
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1499, in run_script
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/EGG-INFO/scripts/multiqc", line 640, in <module>
[2017-06-29T16:03Z] multiqc()
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 722, in __call__
[2017-06-29T16:03Z] return self.main(*args, **kwargs)
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 697, in main
[2017-06-29T16:03Z] rv = self.invoke(ctx)
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 895, in invoke
[2017-06-29T16:03Z] return ctx.invoke(self.callback, **ctx.params)
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 535, in invoke
[2017-06-29T16:03Z] return callback(*args, **kwargs)
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/EGG-INFO/scripts/multiqc", line 548, in multiqc
[2017-06-29T16:03Z] report_output = j_template.render(report=report, config=config)
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/jinja2/environment.py", line 1008, in render
[2017-06-29T16:03Z] return self.environment.handle_exception(exc_info, True)
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
[2017-06-29T16:03Z] reraise(exc_type, exc_value, tb)
[2017-06-29T16:03Z] File "/tmp/tmp4sttwZ/base.html", line 19, in top-level template code
[2017-06-29T16:03Z] {% include 'includes.html' %}
[2017-06-29T16:03Z] File "/tmp/tmp4sttwZ/includes.html", line 40, in top-level template code
[2017-06-29T16:03Z] <script type="text/javascript">{{ include_file('assets/js/packages/highcharts.js') }}</script>
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/EGG-INFO/scripts/multiqc", line 536, in include_file
[2017-06-29T16:03Z] return f.read()
[2017-06-29T16:03Z] File "/home/user1/local/share/bcbio/anaconda/lib/python2.7/codecs.py", line 314, in decode
[2017-06-29T16:03Z] (result, consumed) = self._buffer_decode(data, self.errors, final)
[2017-06-29T16:03Z] UnicodeDecodeError: 'utf8' codec can't decode byte 0xa2 in position 185691: invalid start byte
[2017-06-29T16:03Z] Uncaught exception occurred
From the unicode character table, 0xa2 represents 芒?
Could we assume such 'random' character should be ignored (if the host providing the assets/js/packages/highcharts.js file does not send proper UTF-8) or replaced with certain character(s) on line #535 of multiqc.py:
with io.open (os.path.join(fdir, name), "r", encoding='utf-8') as f:
?
The local bcbio installation is automatic upgrade -u stable ofbcbio_nextgen.py showing version 1.0.3.
Thanks.
elmobile
on 5 Jul 2017
Hi @elmobile,
I guess you're talking about lines 535 and 536 of scripts/multiqc in the v1.0 MultiQC release (there is no multiqc.py file).
This is a strange error, as it's loading a file that's packaged with MultiQC (assuming that you haven't modified the files). highcharts.js has ascii encoding so it shouldn't be a problem. This code that reads the file hasn't changed in two years, so I'm not sure why you're getting this now.
It's not the same error as originally reported in this issue, as the file is already being opened in utf-8 mode. And I can't replicate the problem myself.
Could you find the md5sum of the assets/js/packages/highcharts.js file please? Just to check that the file hasn't been corrupted. I get the following:
MD5 (assets/js/packages/highcharts.js) = 0c20c11400c0bef7c0c8ed72e9b70a8c
Note that your test is called /test_2_hisat2, presumably because of the new HISAT2 module that I wrote and pushed a few minutes ago. However, the log is from MultiQC v1.0 (the latest release). The new HISAT2 module is only in v1.1dev.
Phil
ewels
on 5 Jul 2017
Am running the MultiQC v1.0 in the latest stable release.
Don't know if the following would help,
[user1@centos73 local]$ md5sum ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
8e861203b92818749b4872e88cf9b6ef ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
[user1@centos73 local]$ md5sum ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
8e861203b92818749b4872e88cf9b6ef ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
[user1@centos73 tests]$ python
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import hashlib
>>> print(hashlib.md5(open('./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js', 'rb').read()).hexdigest())
8e861203b92818749b4872e88cf9b6ef
>>> print(hashlib.md5(open('./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js', 'rb').read()).hexdigest())
8e861203b92818749b4872e88cf9b6ef
>>> quit()
[user1@centos73 local]$ ls -l ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
192 -rw-rw-r--. 2 user1 user1 192694 May 28 13:22 ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
192 -rw-rw-r--. 2 user1 user1 192694 May 28 13:22 ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
If the copy of /assets/js/packages/highcharts.js dated 'May 28, 2017' is not correct, is there a quick way to update the /assets/js/packages?
Thanks.
elmobile
on 5 Jul 2017
Hi @elmobile,
I just did a test installation of MultiQC v1.0 using bioconda Python 2.7 as you seem to have, both on mac and linux distros. On both I get the following:
$ conda create --name mqc_test -c bioconda python=2.7 multiqc
$ md5sum miniconda2/envs/mqc_test/lib/python2.7/site-packages/multiqc-1.0.dev0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
0c20c11400c0bef7c0c8ed72e9b70a8c miniconda2/envs/mqc_test/lib/python2.7/site-packages/multiqc-1.0.dev0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
$ ls -l miniconda2/envs/mqc_test/lib/python2.7/site-packages/multiqc-1.0.dev0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
-rw-rw-r-- 2 phil phil 192694 May 25 11:44 miniconda2/envs/mqc_test/lib/python2.7/site-packages/multiqc-1.0.dev0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
So the same number of bytes but a different md5sum and timestamp. I'm not sure how this could happen.. Can you try creating a fresh environment and reinstalling? Do @chapmanb or @lpantano have any thoughts about how bcbio could be affecting this?
Phil
ewels
on 6 Jul 2017
Phil and @elmobile;
It looks like the error comes from the bcbio hisat2 test (not your 1.1-dev MultiQC work). bcbio installs the MultiQC 1.0 bioconda package so nothing fancy is going on.
It seems like something might be corrupted in the install. You could try to do:
bcbio_conda remove multiqc
bcbio_conda install -y -c conda-forge -c bioconda click multiqc
to refresh the multiqc install. Fingers crossed that cleans it up.
chapmanb
on 6 Jul 2017
Somehow bcbio_conda remove multiqc does not remove previous copies of
/home/user1/local/share/bcbio/anaconda/pkgs/{multiqc-1.0-py27_4,multiqc-1.0-py27_4.tar.bz2}.
Need to manually delete the above folder and file before running the bcbio_conda install -y -c conda-forge -c bioconda click multiqc command.
After the multiqc refresh, MD5 gives the matching string:
[user1@centos73 local]$ md5sum ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
0c20c11400c0bef7c0c8ed72e9b70a8c ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
[user1@centos73 local]$ md5sum ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
0c20c11400c0bef7c0c8ed72e9b70a8c ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
[user1@centos73 local]$ ls -l ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
192 -rw-rw-r--. 2 user1 user1 192694 May 28 13:22 ./share/bcbio/anaconda/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
192 -rw-rw-r--. 2 user1 user1 192694 May 28 13:22 ./share/bcbio/anaconda/pkgs/multiqc-1.0-py27_4/lib/python2.7/site-packages/multiqc-1.0-py2.7.egg/multiqc/templates/default/assets/js/packages/highcharts.js
The test_2_hisat2 passed accordingly.
Thanks!
elmobile
on 6 Jul 2017
Ok great! Glad you got it sorted :) Thanks @chapmanb!
ewels
on 6 Jul 2017
@chapmanb - whilst you're here, MultiQC now supports HISAT2 (instead of reporting the logs as coming from Bowtie 2). But you have to add --new-summary to the hisat2 alignment command.. May be useful for the bcbio hisat2 pipeline! See #221 for more info.
Phil
ewels
on 6 Jul 2017
awesome, we鈥檒l add that!
lpantano
on 6 Jul 2017
Related issues
your-highness
路
6Comments
tluquez
路
6Comments
adirot
路
5Comments
mahesh-panchal
路
3Comments
pancheto
路
7Comments