Weblate: Commit fails for i18next format with format changes

Created on 27 Oct 2017 · 28Comments · Source: WeblateOrg/weblate

I found an interesting way to break things. I had an old Dutch translation file that looks like this:

{
  "tip": {
    "title": "Barbara zegt over dit artikel"
  },
  "cta": {
    "settings": "Herken je jezelf niet in het advies? <1>Pas je instellingen aan.</1>"
  }
}

The canonical / master file has a few more items, but most importantly the tip key is now a mapping with subkeys instead of directly translatable:

{
  "tip": {
    "title": {
      "dress": "Barbara says about this dress",
      "skirt": "Barbara says about this skirt",
      "top": "Barbara says about this top",
      "generic": "Barbara says about this article"
    },
    "cta": {
      "dress": "<0>Register now</0> to see if Barbara recommends this dress for you.",
      "skirt": "<0>Register now</0> to see if Barbara recommends this skirt for you.",
      "top": "<0>Register now</0> to see if Barbara recommends this top for you.",
      "generic": "<0>Register now</0> to see if Barbara recommends this article for you."
    }
  },
  "cta": {
    "settings": "Don't recognise yourself in these tips <1>Change your settings.</1>"
  }
}

After adding translations for the tip/* entries trying to commit the result produces an error:

Environment:


Request Method: POST
Request URL: https://translate.curvetips.net/commit/public-websites/site-injection/nl/

Django Version: 1.11.6
Python Version: 2.7.13
Installed Applications:
(u'django.contrib.auth',
 u'django.contrib.contenttypes',
 u'django.contrib.sessions',
 u'django.contrib.sites',
 u'django.contrib.messages',
 u'django.contrib.staticfiles',
 u'django.contrib.admin.apps.SimpleAdminConfig',
 u'django.contrib.admindocs',
 u'django.contrib.sitemaps',
 u'social_django',
 u'crispy_forms',
 u'compressor',
 u'rest_framework',
 u'rest_framework.authtoken',
 u'weblate.trans',
 u'weblate.lang',
 u'weblate.permissions',
 u'weblate.screenshots',
 u'weblate.accounts',
 u'weblate.utils',
 u'weblate.wladmin',
 u'weblate',
 u'weblate.gitexport')
Installed Middleware:
(u'django.contrib.sessions.middleware.SessionMiddleware',
 u'django.middleware.common.CommonMiddleware',
 u'django.middleware.locale.LocaleMiddleware',
 u'django.middleware.csrf.CsrfViewMiddleware',
 u'weblate.accounts.middleware.AuthenticationMiddleware',
 u'django.contrib.messages.middleware.MessageMiddleware',
 u'django.middleware.clickjacking.XFrameOptionsMiddleware',
 u'social_django.middleware.SocialAuthExceptionMiddleware',
 u'weblate.accounts.middleware.RequireLoginMiddleware',
 u'weblate.middleware.SecurityMiddleware')



Traceback:

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/http.py" in inner
  40.             return func(request, *args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/weblate/trans/views/git.py" in commit_translation
  135.     return perform_commit(request, obj)

File "/usr/local/lib/python2.7/dist-packages/weblate/trans/views/git.py" in perform_commit
  67.         request,

File "/usr/local/lib/python2.7/dist-packages/weblate/trans/views/git.py" in execute_locked
  44.         result = call(*args, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/weblate/trans/models/translation.py" in commit_pending
  641.             request, last, self.last_change, True, True, skip_push

File "/usr/local/lib/python2.7/dist-packages/weblate/trans/models/translation.py" in git_commit
  751.                 self.update_units(author)

File "/usr/local/lib/python2.7/dist-packages/weblate/trans/models/translation.py" in update_units
  866.         self.store.save()

File "/usr/local/lib/python2.7/dist-packages/weblate/trans/formats.py" in save
  665.             self.store.serialize(handle)

File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py" in serialize
  169.         units = self.serialize_units()

File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py" in serialize_units
  480.                 self.nested_set(units, path, target)

File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py" in nested_set
  245.             self.nested_set(target[path[0]], path[1:], value)

File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py" in nested_set
  245.             self.nested_set(target[path[0]], path[1:], value)

File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py" in nested_set
  247.             target[path[0]] = value

Exception Type: TypeError at /commit/public-websites/site-injection/nl/
Exception Value: 'unicode' object does not support item assignment

bug translate-toolkit

Source

wichert

All 28 comments

Interestingly iIt seems weblate _did_ create a commit, which just removed the JSON file entirely:

commit 8e1eda32080f6402e7770986cdf48099743f1f82
Author: Wichert Akkerman <[email protected]>
Date:   Fri Oct 27 08:37:19 2017 +0000

    Translated using Weblate (Dutch)

    Currently translated at 55.5% (5 of 9 strings)

    Translation: Public Websites/Advice injection
    Translate-URL: https://translate.curvetips.net/projects/public-websites/site-injection/nl/

    [skip ci]

diff --git a/locales/nl.json b/locales/nl.json
index c2dd177..e69de29 100644
--- a/locales/nl.json
+++ b/locales/nl.json
@@ -1,8 +0,0 @@
-{
-  "tip": {
-    "title": "Barbara zegt over dit artikel"
-  },
-  "cta": {
-    "settings": "Herken je jezelf niet in het advies? <1>Pas je instellingen aan.</1>"
-  }
-}

wichert on 27 Oct 2017

Well the problem is not in the tip node, but that there is tip/title which is text while you want to add keys to it.

What indeed should not happen is deleting the file content...

nijel on 27 Oct 2017

On translate-toolkit side this should be already fixed as of translate/translate@1ab87665ba0cc9a98457162db4ac5df45ff61224

nijel on 7 Nov 2017

Thank you for your report, the issue you have reported has just been fixed.

In case you see problem with the fix, please comment on this issue.
In case you see similar problem, please open separate issue.
If you are happy with the outcome, consider supporting Weblate by donating.

nijel on 7 Nov 2017

Seems I have this issue as well (Same stacktrace it seems).
And I think we also changed 1 translation, to become a hash with nested keys instead.

After translating 3 hours translating.. now we can't

access the Commit page to reset the branch.
click Save on ANY translation
download json file (XLF/PO/MO files work fine)

We're using the docker setup, which was setup on Feb.22.
We're literrally going to launch our product in a few DAYS :(

Do I have to re-setup weblate now? :O (Loosing 3h of work, since we can't push those changes, plus copy-paste back all translations after i re-setup, since we can get them from downloading other file formats)


database_1  | DETAIL:  Key (unitid)=(1832) already exists.
database_1  | STATEMENT:  INSERT INTO "trans_indexupdate" ("unitid", "source", "to_delete", "language_code") VALUES (1832, true, false, 'zh_CN') RETURNING "trans_indexupdate"."id"
weblate_1   | 2018-03-11 11:00:52,130 DEBG 'uwsgi' stderr output:
weblate_1   | ERROR Internal Server Error: /commit/flately/app/zh_CN/
weblate_1   | Traceback (most recent call last):
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/exception.py", line 41, in inner
weblate_1   |     response = get_response(request)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 187, in _get_response
weblate_1   |     response = self.process_exception_by_middleware(e, request)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 185, in _get_response
weblate_1   |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/django/utils/decorators.py", line 185, in inner
weblate_1   |     return func(*args, **kwargs)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
weblate_1   |     return view_func(request, *args, **kwargs)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/http.py", line 40, in inner
weblate_1   |     return func(request, *args, **kwargs)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/weblate/trans/views/git.py", line 135, in commit_translation
weblate_1   |     return perform_commit(request, obj)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/weblate/trans/views/git.py", line 67, in perform_commit
weblate_1   |     request,
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/weblate/trans/views/git.py", line 44, in execute_locked
weblate_1   |     result = call(*args, **kwargs)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/weblate/trans/models/translation.py", line 419, in commit_pending
weblate_1   |     request, last, self.last_change, True, True, skip_push
weblate_1   |     self.store.serialize(temp)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py", line 169, in serialize
weblate_1   |     units = self.serialize_units()
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py", line 253, in serialize_units
weblate_1   |     self.nested_set(units, path, unit.target)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py", line 245, in nested_set
weblate_1   |     self.nested_set(target[path[0]], path[1:], value)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py", line 245, in nested_set
weblate_1   |     self.nested_set(target[path[0]], path[1:], value)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py", line 245, in nested_set
weblate_1   |     self.nested_set(target[path[0]], path[1:], value)
weblate_1   |   File "/usr/local/lib/python2.7/dist-packages/translate/storage/jsonl10n.py", line 247, in nested_set
weblate_1   |     target[path[0]] = value
weblate_1   | TypeError: 'unicode' object does not support item assignment

Frexuz on 11 Mar 2018

What Weblate version do you use? Anyway the biggest issue is that translate-tookit with the fix is not yet released...

nijel on 11 Mar 2018

Oh, i thought it was released since you closed this issue :)

I'm using https://github.com/WeblateOrg/docker (latest master)
Not sure how to check version? 🤔

Frexuz on 11 Mar 2018

The issue was fixed on Weblate to avoid writing empty file in case of failure. The original issue has been fixed in translate-toolkit, but it hasn't seen release since then. I've just asked for a release there.

nijel on 11 Mar 2018

👍1

Thanks

Frexuz on 12 Mar 2018

I've just pushed docker container 2.19.1-2, which uses latest translate-toolkit which should address this issue.

nijel on 12 Mar 2018

I'm having the same issue, i can't commit a project :(

imagem