Weblate: Android xml entities translation issue

[Viish]

Hi,

Describe the bug

Translating an Android string that contains an entity fails.

To Reproduce

Steps to reproduce the behavior:

1. Create a translation component for an Android project using xml entities
2. Translate a string using a xml entity to another language
3. Save changes
4. See error

Expected behavior

Translated string should only contain &; instead of &;&;

Screenshots

Web interface seems fine:

But commit ins't:

Server configuration and status

(weblate-env) [weblate@linphone ~]$ weblate list_versions
 * Weblate: 4.2.2
 * Django: 3.1.1
 * siphashc: 2.1
 * Whoosh: 2.7.4
 * translate-toolkit: 3.1.1
 * lxml: 4.5.2
 * Pillow: 7.2.0
 * bleach: 3.1.5
 * python-dateutil: 2.8.1
 * social-auth-core: 3.3.3
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4
 * djangorestframework: 3.11.1
 * django-filter: 2.3.0
 * django-appconf: 1.0.4
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 50.3.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.5
 * celery: 4.4.6
 * kombu: 4.6.11
 * translation-finder: 2.1
 * html2text: 2020.1.16
 * pycairo: 1.19.1
 * pygobject: 3.38.0
 * diff-match-patch: 20200713
 * requests: 2.24.0
 * django-redis: 4.12.1
 * hiredis: 1.1.0
 * sentry_sdk: 0.16.5
 * Cython: 0.29.21
 * misaka: 2.1.1
 * GitPython: 3.1.8
 * borgbackup: 1.1.13
 * pyparsing: 2.4.7
 * Python: 3.6.8
 * Git: 2.18.4
 * psycopg2-binary: 2.8.6
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.12
 * Redis server: 3.2.12
 * PostgreSQL server: 12.1
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.belledonne-communications.com
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://localhost:6379, redis://localhost:6379, regular
 * Platform: Linux 3.10.0-1127.19.1.el7.x86_64 (x86_64)

Weblate deploy checks

(weblate-env) [weblate@linphone ~]$ weblate check --deploy
System check identified some issues:

WARNINGS:
?: (weblate.W025.ass) Failure in loading handler for ass file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.ini) Failure in loading handler for ini file format: Missing iniparse library.
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.islu) Failure in loading handler for islu file format: Missing iniparse library.
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.laravel) Failure in loading handler for laravel file format: No module named 'phply'
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.php) Failure in loading handler for php file format: No module named 'phply'
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.srt) Failure in loading handler for srt file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.ssa) Failure in loading handler for ssa file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W025.sub) Failure in loading handler for sub file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#optional-deps
?: (weblate.W033.Gerrit) Failure in loading VCS module for Gerrit: git: 'review' is not a git command. See 'git --help'.
 (1)
    HINT: https://docs.weblate.org/en/weblate-4.2.2/vcs.html
?: (weblate.W033.GitHub) Failure in loading VCS module for GitHub: [Errno 2] No such file or directory: 'hub': 'hub'
    HINT: https://docs.weblate.org/en/weblate-4.2.2/vcs.html
?: (weblate.W033.GitLab) Failure in loading VCS module for GitLab: [Errno 2] No such file or directory: 'lab': 'lab'
    HINT: https://docs.weblate.org/en/weblate-4.2.2/vcs.html
?: (weblate.W033.Mercurial) Failure in loading VCS module for Mercurial: [Errno 2] No such file or directory: 'hg': 'hg'
    HINT: https://docs.weblate.org/en/weblate-4.2.2/vcs.html
?: (weblate.W033.Subversion) Failure in loading VCS module for Subversion: git: 'svn' is not a git command. See 'git --help'.

The most similar command is
    serve
 (1)
    HINT: https://docs.weblate.org/en/weblate-4.2.2/vcs.html

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-4.2.2/admin/backup.html

System check identified 15 issues (1 silenced).

Exception traceback

N/A

Additional context

Attempting to translate branch feature/kotlin_rewrite of https://gitlab.linphone.org/BC/public/linphone-android/

Thanks in advance.

Best regards,

[nijel]

Just to be sure: Was this change generated by the version you are listing or the commit was created earlier? I see you now have translate-toolkit 3.1.1 which was relesed yesterday and I think the 3.1 release did fix some related Android issues.

[github-actions[bot]]

The issue you've reported needs to be addressed in the translate-toolkit. Please file the issue there and do not forget to include links to any relevant specifications about the formats (if applicable).

[Viish]

Hi,

Was this change generated by the version you are listing

Yes, it was created this morning as I started to translate.

I see you now have translate-toolkit 3.1.1 which was relesed yesterday and I think the 3.1 release did fix some related Android issues.

So I have to downgrade the translate-toolkit ?

Thanks in advance.

[nijel]

So I have to downgrade the translate-toolkit ?

I don't think downgrading will help. As a workaround you can add doctype to the XML prior translating in Weblate, that should work.

[github-actions[bot]]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[Viish]

I don't think downgrading will help. As a workaround you can add doctype to the XML prior translating in Weblate, that should work.

I just tried it (deleted translation and started it again from scratch), doesn't work:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE resources [
<!ENTITY appName "Linphone">
]>
<resources>
    <string name="app_name">&amp;appName;&appName;</string>
</resources>

[nijel]

You need to start with a file that has the doctype declaration:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE resources [
<!ENTITY appName "Linphone">
]>
<resources>
</resources>

Weblate should be able to add the strings in this case.

Or install translate-toolkit with the fix:

pip install https://github.com/translate/translate/archive/master.zip

[Viish]

You need to start with a file that has the doctype declaration:

That's what I did but it didn't worked.

Or install translate-toolkit with the fix:

I'm going to try it right now

[Viish]

Still the same issue:

(weblate-env) [weblate@linphone ~]$ pip install https://github.com/translate/translate/archive/master.zip
Collecting https://github.com/translate/translate/archive/master.zip
  Downloading https://github.com/translate/translate/archive/master.zip
     / 1.6 MB 5.6 MB/s
Requirement already satisfied (use --upgrade to upgrade): translate-toolkit==3.1.1 from https://github.com/translate/translate/archive/master.zip in ./weblate-env/lib/python3.6/site-packages
Requirement already satisfied: lxml!=4.3.1,>=4.0 in ./weblate-env/lib/python3.6/site-packages (from translate-toolkit==3.1.1) (4.5.2)
Building wheels for collected packages: translate-toolkit
  Building wheel for translate-toolkit (setup.py) ... done
  Created wheel for translate-toolkit: filename=translate_toolkit-3.1.1-py3-none-any.whl size=1339346 sha256=1ce30e1b2fd14b1656f680f9243ca9622a9c496e322878d3cdbd62ceb2eaac29
  Stored in directory: /tmp/pip-ephem-wheel-cache-cc5r1bp5/wheels/94/4d/cd/ef17a9e09a40dd37ff4825a30b13efdb2c35bf101ac122fcf2
Successfully built translate-toolkit

->

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE resources [
<!ENTITY appName "Linphone">
]>
<resources>
    <string name="app_name">&amp;appName;&appName;</string>
</resources>

I can create you an account on the deployed weblate if that can help.

[nijel]

Did you restart both uwsgi and celery workers after the upgrade?

[Viish]

Did you restart both uwsgi and celery workers after the upgrade?

Httpd yes, not celery, I'm going to do it.

[nijel]

In many cases Celery is doing the writes, so that one is important as well :-).

[Viish]

Restarted celery:

[root@linphone ~]# service celery-weblate status
Redirecting to /bin/systemctl status celery-weblate.service
● celery-weblate.service - Celery Service (Weblate)
   Loaded: loaded (/etc/systemd/system/celery-weblate.service; enabled; vendor preset: disabled)
   Active: active (running) since jeu. 2020-09-24 15:10:54 CEST; 5min ago
  Process: 1206 ExecStop=/bin/sh -c /home/weblate/weblate-env/bin/celery multi stopwait celery notify backup translate --pidfile=${CELERYD_PID_FILE} (code=exited, status=0/SUCCESS)
  Process: 1211 ExecStart=/bin/sh -c /home/weblate/weblate-env/bin/celery multi start celery notify backup translate -A weblate.utils --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} --beat:celery --queues:celery=celery --prefetch-multiplier:celery=4  --queues:notify=notify --prefetch-multiplier:notify=10 --queues:translate=translate --prefetch-multiplier:translate=4  --concurrency:backup=1 --queues:backup=backup  --prefetch-multiplier:backup=2 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/celery-weblate.service
           ├─1224 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --beat --queues=celery --prefetch-multiplier=4 --logfile=/var/log/celery/weblate-celery%I.log --pidfile=/tmp/celery-weblate/celer...
           ├─1241 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --queues=notify --prefetch-multiplier=10 --logfile=/var/log/celery/weblate-notify%I.log --pidfile=/tmp/celery-weblate/notify.pid ...
           ├─1255 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --concurrency=1 --queues=backup --prefetch-multiplier=2 --logfile=/var/log/celery/weblate-backup%I.log --pidfile=/tmp/celery-webl...
           ├─1271 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --queues=translate --prefetch-multiplier=4 --logfile=/var/log/celery/weblate-translate%I.log --pidfile=/tmp/celery-weblate/transl...
           ├─1316 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --beat --queues=celery --prefetch-multiplier=4 --logfile=/var/log/celery/weblate-celery%I.log --pidfile=/tmp/celery-weblate/celer...
           ├─1324 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --queues=notify --prefetch-multiplier=10 --logfile=/var/log/celery/weblate-notify%I.log --pidfile=/tmp/celery-weblate/notify.pid ...
           ├─1325 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --beat --queues=celery --prefetch-multiplier=4 --logfile=/var/log/celery/weblate-celery%I.log --pidfile=/tmp/celery-weblate/celer...
           ├─1326 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --queues=notify --prefetch-multiplier=10 --logfile=/var/log/celery/weblate-notify%I.log --pidfile=/tmp/celery-weblate/notify.pid ...
           ├─1327 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --concurrency=1 --queues=backup --prefetch-multiplier=2 --logfile=/var/log/celery/weblate-backup%I.log --pidfile=/tmp/celery-webl...
           ├─1333 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --queues=translate --prefetch-multiplier=4 --logfile=/var/log/celery/weblate-translate%I.log --pidfile=/tmp/celery-weblate/transl...
           ├─1334 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --queues=translate --prefetch-multiplier=4 --logfile=/var/log/celery/weblate-translate%I.log --pidfile=/tmp/celery-weblate/transl...
           ├─1617 git push origin feature/kotlin_rewrite:weblate/translation
           ├─1619 ssh -o UserKnownHostsFile=/home/weblate/weblate-data/ssh/known_hosts -o IdentityFile=/home/weblate/weblate-data/ssh/id_rsa -o StrictHostKeyChecking=yes -o HashKnownHosts=no -F /dev/null [email protected] git-rec...
           └─2061 /home/weblate/weblate-env/bin/python3 -m celery worker -A weblate.utils --loglevel=INFO --beat --queues=celery --prefetch-multiplier=4 --logfile=/var/log/celery/weblate-celery%I.log --pidfile=/tmp/celery-weblate/celer...

sept. 24 15:10:54 linphone.org sh[1211]: > [email protected]: OK
sept. 24 15:10:54 linphone.org systemd[1]: Started Celery Service (Weblate).
sept. 24 15:11:15 linphone.org weblate[1323]: INFO linphone/linphone-android: updating repository
sept. 24 15:12:00 linphone.org weblate[1323]: INFO linphone/linphone-android: update took 44.68 seconds
sept. 24 15:12:00 linphone.org weblate[1323]: INFO linphone/linphone-android: repository updated from f6c2d2dd11c5d2a5469838aa606e0c0f7e86216e to f6c2d2dd11c5d2a5469838aa606e0c0f7e86216e
sept. 24 15:12:00 linphone.org weblate[1323]: INFO linphone/linphone-android: pushing to remote repo
sept. 24 15:12:00 linphone.org weblate[1325]: INFO linphone/linphone-android: updating repository
sept. 24 15:12:17 linphone.org weblate[1325]: INFO linphone/linphone-android: update took 17.45 seconds
sept. 24 15:12:17 linphone.org weblate[1325]: INFO linphone/linphone-android: repository updated from f6c2d2dd11c5d2a5469838aa606e0c0f7e86216e to f6c2d2dd11c5d2a5469838aa606e0c0f7e86216e
sept. 24 15:12:17 linphone.org weblate[1325]: INFO linphone/linphone-android: pushing to remote repo

But have still the same issue (I tried another string just in case):

(weblate-env) [weblate@linphone linphone]$ git status
fatal: not a git repository (or any of the parent directories): .git
(weblate-env) [weblate@linphone linphone]$ cd linphone-android
(weblate-env) [weblate@linphone linphone-android]$ git status
On branch feature/kotlin_rewrite
Your branch is ahead of 'origin/feature/kotlin_rewrite' by 2 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean
(weblate-env) [weblate@linphone linphone-android]$ git diff HEAD^^
diff --git a/app/src/main/res/values-fr/strings.xml b/app/src/main/res/values-fr/strings.xml
new file mode 100644
index 0000000..46b5680
--- /dev/null
+++ b/app/src/main/res/values-fr/strings.xml
@@ -0,0 +1,7 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE resources [
+<!ENTITY appName "Linphone">
+]>
+<resources>
+    <string name="service_name">&amp;appName; Service&appName; Service</string>
+</resources>
\ No newline at end of file

And some new errors:
weblate.log

[nijel]

I've also found related issue, but in this case it only produces more messed up output and will not fix it for you: https://github.com/translate/translate/pull/4137

[Viish]

I have found an interesting thing: the issue only happens if I don't change the text.
Example:
&appName; Service -> &appName; Service => Bug
&appName; Service-> &appName; Service2 => No bug

[nijel]

This still seems like you are hitting the old bug, do you run mod_wsgi inside httpd or standalone uwsgi? In the latter case the uwsgi should be restarted (and httpd doesn't have to).

Anyway to test, you can install again current master. I've just merged https://github.com/translate/translate/pull/4137 which will change the output in case of the failure, so that we will know for sure that the new code is in use.

[Viish]

I did another:

(weblate-env) [weblate@linphone linphone-android]$ pip install https://github.com/translate/translate/archive/master.zip
Collecting https://github.com/translate/translate/archive/master.zip
  Downloading https://github.com/translate/translate/archive/master.zip
     | 1.6 MB 5.2 MB/s
Requirement already satisfied (use --upgrade to upgrade): translate-toolkit==3.1.1 from https://github.com/translate/translate/archive/master.zip in /home/weblate/weblate-env/lib/python3.6/site-packages
Requirement already satisfied: lxml!=4.3.1,>=4.0 in /home/weblate/weblate-env/lib/python3.6/site-packages (from translate-toolkit==3.1.1) (4.5.2)
Building wheels for collected packages: translate-toolkit
  Building wheel for translate-toolkit (setup.py) ... done
  Created wheel for translate-toolkit: filename=translate_toolkit-3.1.1-py3-none-any.whl size=1339407 sha256=c058ea6fb76ecec93938bfe6719b78dca21c5b0e3ab922eb653d189b61771ece
  Stored in directory: /tmp/pip-ephem-wheel-cache-bs5fb8ji/wheels/94/4d/cd/ef17a9e09a40dd37ff4825a30b13efdb2c35bf101ac122fcf2
Successfully built translate-toolkit

Followed by:

[root@linphone ~]# service celery-weblate restart
Redirecting to /bin/systemctl restart celery-weblate.service
[root@linphone ~]# systemctl restart httpd24-httpd

Then I deleted the french translation, created it again and translated 2 strings and here's the result:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE resources [
<!ENTITY appName "Linphone">
]>
<resources>
    <string name="service_name">&amp;appName; Service&appName; Service</string>
    <string name="service_auto_start_description">&amp;appName; a été démarré automatiquement&appName; has been started automatically</string>
</resources>

[Viish]

By the way, when I make changes into settings.py, a systemctl reload httpd24-httpd is enough for them to be applied.

[nijel]

This IMHO still runs the old code. Maybe reload doesn't really realod the Python code? Can you try restart?

[Viish]

I did a restart to be sure, even if usually the reload is enough.
Mod_wsgi was installed through pip, and I see no trace of uwsgi anywhere except for /opt/rh/httpd24/root/usr/lib64/httpd/modules/mod_proxy_uwsgi.so and weblate/examples/weblate.uwsgi.ini

[nijel]

uwsgi is one of ways to run the server, you don't have to use it.

Still, it seems that the web server doesn't pick up the translate-toolkit update because at least it should no longer duplicate the content....

Requirement already satisfied (use --upgrade to upgrade): translate-toolkit==3.1.1 from https://github.com/translate/translate/archive/master.zip in /home/weblate/weblate-env/lib/python3.6/site-packages

Ah, that explains it, you have to do:

pip install --upgrade https://github.com/translate/translate/archive/master.zip

[Viish]

Ah, that explains it, you have to do:
pip install --upgrade https://github.com/translate/translate/archive/master.zip

It works!!!

Thanks a lot for your availability!!!

Best regards.

[github-actions[bot]]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.