Weblate: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)

[pc-jedi]

Steps to reproduce

I'm not sure what exactly the user did. But as far as I understand the trancelog he was searching for something.

Actual behaviour

Internal Server Error: /translate/sites/xxxx/es/

UnicodeDecodeError at /translate/sites/xxxx/es/
'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)

Request Method: GET
Request URL: http://translate.xxxx.com/translate/sites/xxxx/es/?q=Jugador&search=ftx&source=on&target=on&type=all&ignored=False
Django Version: 1.9.5
Python Executable: /usr/local/bin/python3.5
Python Version: 3.5.1
Python Path: ['/usr/local/bin', '/usr/local/lib/python35.zip', '/usr/local/lib/python3.5', '/usr/local/lib/python3.5/plat-linux', '/usr/local/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/site-packages']
Server time: Vie, 15 Apr 2016 16:28:41 +0200
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin',
 'django.contrib.admindocs',
 'django.contrib.sitemaps',
 'social.apps.django_app.default',
 'crispy_forms',
 'compressor',
 'rest_framework',
 'rest_framework.authtoken',
 'weblate.trans',
 'weblate.lang',
 'weblate.accounts',
 'weblate')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'social.apps.django_app.middleware.SocialAuthExceptionMiddleware',
 'weblate.accounts.middleware.RequireLoginMiddleware')


Traceback:  

File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py" in get_response
  149.                     response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py" in get_response
  147.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.5/site-packages/weblate/trans/views/edit.py" in translate
  481.     search_result = search(translation, request)

File "/usr/local/lib/python3.5/site-packages/weblate/trans/views/edit.py" in search
  131.             search_form.cleaned_data,

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/unit.py" in search
  255.                     params

File "/usr/local/lib/python3.5/site-packages/weblate/trans/search.py" in fulltext_search
  293.                 TargetSchema()

File "/usr/local/lib/python3.5/site-packages/weblate/trans/search.py" in base_search
  257.         return [result['pk'] for result in searcher.search(terms)]

File "/usr/local/lib/python3.5/site-packages/whoosh/searching.py" in search
  786.         self.search_with_collector(q, c)

File "/usr/local/lib/python3.5/site-packages/whoosh/searching.py" in search_with_collector
  819.         collector.run()

File "/usr/local/lib/python3.5/site-packages/whoosh/collectors.py" in run
  144.                 self.collect_matches()

File "/usr/local/lib/python3.5/site-packages/whoosh/collectors.py" in collect_matches
  214.         for sub_docnum in self.matches():

File "/usr/local/lib/python3.5/site-packages/whoosh/collectors.py" in matches
  415.             yield matcher.id()

File "/usr/local/lib/python3.5/site-packages/whoosh/codec/whoosh3.py" in id
  980.             self._read_ids()

File "/usr/local/lib/python3.5/site-packages/whoosh/codec/whoosh3.py" in _read_ids
  1082.             self._read_data()

File "/usr/local/lib/python3.5/site-packages/whoosh/codec/whoosh3.py" in _read_data
  1077.         self._data = loads(b)

Exception Type: UnicodeDecodeError at /translate/sites/xxxx/es/
Exception Value: 'ascii' codec can't decode byte 0x80 in position 4: ordinal not in range(128)
Request information:
GET:
source = 'on'
type = 'all'
target = 'on'
search = 'ftx'
ignored = 'False'
q = 'Jugador'

Server configuration

 * Weblate 2.6-dev
 * Python 3.5.1
 * Django 1.9.5
 * six 1.10.0
 * python-social-auth 0.2.16
 * Translate Toolkit 1.14.0-rc1
 * Whoosh 2.7.4
 * Git 2.1.4
 * Pillow (PIL) 1.1.7
 * dateutil 2.5.2
 * lxml 3.6.0
 * django-crispy-forms 1.6.0
 * compressor 1.6
 * djangorestframework 3.3.3
 * pyuca N/A
 * pyLibravatar N/A
 * Database backends: django.db.backends.mysql

[nijel]

It seems like whoosh is having some troubles with the index. Was it created with Python 3 or it was migrated from Python 2?

[pc-jedi]

The whole environment was created with Python 3.

[nijel]

Looks like Whoosh bug, I've filed it there: https://bitbucket.org/mchaput/whoosh/issues/441/unicodedecodeerror-on-search

[Mikanoshi]

https://gist.github.com/Mikanoshi/0e751ef3a496ebd340dcaa43362abdeb
Never used Python 3. Character is horizontal ellipsis …
Will updating to 2.6 help? I'm on 2.5-dev

[nijel]

Okay, this was fixed in d9da8a438ed06facc6ac32025bf5dc7d5e073ba3, see #996

[ihoru]

I've Python 2.7 and weblate-2.6 checked out, but I have the same problem:

[ihoru]

@nijel , here is my log when I'm trying to execute ./manage.py loadpo --all and it finds new locale:

INFO project/prod/zh_TW: processing resources/lang/i18n/zh_TW/LC_MESSAGES/messages.po, revision has changed
Traceback (most recent call last):
  File "./manage.py", line 31, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/opt/weblate/weblate/trans/management/commands/loadpo.py", line 42, in handle
    subproject.create_translations(options['force'], langs)
  File "/opt/weblate/weblate/trans/models/subproject.py", line 1002, in create_translations
    self, lang, code, path, force, request=request
  File "/opt/weblate/weblate/trans/models/translation.py", line 70, in check_sync
    translation.check_sync(force, request=request)
  File "/opt/weblate/weblate/trans/models/translation.py", line 461, in check_sync
    self, unit, pos
  File "/opt/weblate/weblate/trans/models/unit.py", line 107, in update_from_unit
    dbunit.update_from_unit(unit, pos, created)
  File "/opt/weblate/weblate/trans/models/unit.py", line 486, in update_from_unit
    same_state=same_state
  File "/opt/weblate/weblate/trans/models/unit.py", line 748, in save
    super(Unit, self).save(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 708, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 736, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 820, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 859, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1039, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 1060, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 112, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 207, in execute
    args = tuple(map(db.literal, args))
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 304, in literal
    s = self.escape(o, self.encoders)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 215, in string_literal
    return db.string_literal(obj)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

$ ./manage.py list_versions:

 * Weblate weblate-2.6
 * Python 2.7.11+
 * Django 1.9.5
 * six 1.10.0
 * python-social-auth 0.2.18
 * Translate Toolkit 1.14.0-rc1
 * Whoosh 2.7.4
 * Git 2.7.4
 * Pillow (PIL) 1.1.7
 * dateutil 2.5.3
 * lxml 3.6.0
 * django-crispy-forms 1.6.0
 * compressor 1.6
 * djangorestframework 3.3.3
 * pytz 2016.4
 * pyuca N/A
 * pyLibravatar N/A
 * Mercurial 3.7.3
 * Database backends: django.db.backends.mysql

[nijel]

@ihoru Can you please open separate issue for it? This is already third different problem in this bug...

[ihoru]

@nijel, new issue created: https://github.com/nijel/weblate/issues/1106