Weblate: XLF missing <target> translation in latest version of Weblate

[UncleSamSwiss]

Describe the bug
We updated our (docker-compose) Weblate installation (by simply doing a docker-compose pull) to the latest version (unfortunately I don't remember the exact version we had before). Since that update, when we translate a string, it only ends up as <source> but not as <target> in the translated XLF file.

To Reproduce
Steps to reproduce the behavior:

1. Translate an existing (but not yet translated) string in an existing language.
2. Commit, rebase and push to GIT
3. Translated XLF contains new <trans-unit> with <source> (translated!) but without <target>

Expected behavior
The XLF file must contain both <source> and <target> tags.
It used to work with the previous version of Weblate.

Examples
Correct translation with a previous version of Weblate (unfortunately unknown version pulled on January 25, 2019):

    <trans-unit id="personsHeader" datatype="html">
        <source>Persons</source>
        <target>Personnes</target><context-group purpose="location">
          <context context-type="sourcefile">app/app.component.ts</context>
          <context context-type="linenumber">6</context>
        </context-group>
        <note priority="1" from="description">Header for the persons list</note>
      </trans-unit>

Wrong translation (in the same file) with the latest version of Weblate (version see below):

      <trans-unit id="networkConfigTitle" datatype="html">
        <source>Configuration Réseau</source>
        <context-group purpose="location">
          <context context-type="sourcefile">app/components/network-config/network-config.component.html</context>
          <context context-type="linenumber">13</context>
        </context-group>
        <note priority="1" from="description">Network configuration title</note>
      </trans-unit>

Server configuration and status

 * Weblate 3.5.1
 * Python 3.6.7
 * Django 2.1.7
 * Celery 4.2.1
 * celery-batches 0.2
 * six 1.12.0
 * social-auth-core 3.1.0
 * social-auth-app-django 3.1.0
 * django-appconf 1.0.3
 * translate-toolkit 2.3.1
 * translation-finder 1.0
 * Whoosh 2.7.4
 * defusedxml 0.5.0
 * Git 2.17.1
 * Pillow 5.1.0
 * python-dateutil 2.8.0
 * lxml 4.2.1
 * django-crispy-forms 1.7.2
 * django_compressor 2.2
 * djangorestframework 3.9.2
 * user-agents 1.1.0
 * jellyfish 0.7.1
 * diff-match-patch 20121119
 * pytz 2018.9
 * pyuca 1.2
 * PyYAML 3.12
 * tesserocr 2.4.0
 * Mercurial 4.5.3
 * git-svn 2.17.1
 * Database backends: django.db.backends.postgresql
 * Cache backends: avatar:FileBasedCache, default:RedisCache
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 4.15.0-46-generic (x86_64)

System check identified some issues:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W018) You should not have DEBUG set to True in deployment.

INFOS:
?: (weblate.I021) Error collection is not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-3.5.1/admin/install.html#collecting-errors

System check identified 5 issues (0 silenced).

Additional context
If you need additional information (like component configuration in Weblate), please feel free to ask here.

[nijel]

Can you please list your translation component configuration?

[nijel]

Possibly related to https://github.com/WeblateOrg/weblate/issues/2600

[UncleSamSwiss]

_Part 1:_

_Part 2:_

_Part 3:_

I hope, these are the relevant sections.

[UncleSamSwiss]

BTW, I think the <source> should be copied from the base file and only the <target> should be updated. With the old version it was just copying the same string in both places.

[UncleSamSwiss]

Just for completeness these are the translation units from the messages.xlf (base file):

      <trans-unit id="personsHeader" datatype="html">
        <source>Persons</source>
        <context-group purpose="location">
          <context context-type="sourcefile">app/components/homescreen/homescreen.component.html</context>
          <context context-type="linenumber">4</context>
        </context-group>
        <note priority="1" from="description">Header for the persons list</note>
      </trans-unit>

      <trans-unit id="networkConfigTitle" datatype="html">
        <source>
    Network Configuration
  </source>
        <context-group purpose="location">
          <context context-type="sourcefile">app/components/network-config/network-config.component.html</context>
          <context context-type="linenumber">13</context>
        </context-group>
        <note priority="1" from="description">Network configuration title</note>
      </trans-unit>

[UncleSamSwiss]

Indeed it looks like the same issue as @ShadowXVII is having. As I'm not allowing the editing of the base file, issue #2521 didn't bother me; but the change b21089b is a regression for me as well.

[nijel]

Unset monolingual base file in settings and it will work correctly. It should be set only if you are using monolingual translations, also see https://docs.weblate.org/en/latest/formats.html#bilingual-and-monolingual-formats.

[UncleSamSwiss]

Thanks for your reply. I'm a bit confused; we had the following setup until now:

1. Angular on the build server generates the (English) src/frontend/web/src/i18n/messages.xlf
2. Weblate uses messages.xlf to import new strings to be translated
3. Weblate generates the messages.de.xlf and messages.fr.xlf files with all available translations
4. Weblate commits and pushes those changes back to our Git

This used to work just fine for us until the mentioned recent change.

If we now remove the "Monolingual base language file", how will Weblate know when Angular added new translatable strings in messages.xlf? Angular does not generate messages.de.xlf and messages.fr.xlf.

[nijel]

Hmm, this is in conflict what was requested in https://github.com/WeblateOrg/weblate/issues/2521, we probably need some setting to decide between these two.

[ShadowXVII]

2521 should only apply to the base/source file when "allow editing base language" is on -- i.e messages.xlf (which only contains sources, right @UncleSamSwiss ?)

The other files should continue to work like normal (source and target).

[UncleSamSwiss]

@ShadowXVII: right.

I agree with all @ShadowXVII says, but it would be great if we had showing the original string from the base language file. This is not strictly necessary, but it would make more sense when you read a translated file: you would see the source and the translation, not twice the translation.

If what I'm writing here doesn't make sense, please let me know and I can elaborate with an example.

[SomeBdyElse]

I do have the same issue translating a TYPO3 project. It has the following (common?) file layout for its xliff files:

locallang.xlf is a monolingual source file. It contains only source and no target elements. It is mostly hand-written by the developers when they add new keys.

*.locallang.xlf are bilingual translation files. These files should contain the same source strings as locallang.xlf (unless stale) and target elements in their respective languages.

So I do need both: A monolingual translation base and bilingual translation files. This is how I tried to configure this in the component setup:

Expected behavior:
Editors should be allowed to edit source.

• If a translator edits a translation source, the _source_ element in locallang.xlf should be updated. The source elements of the translation files shall be left untouched (stale).
• If an editor edits a translation, he/she shall see the source from the monolingual locallang.xlf source file. Once the editor saves the translation, the _source_ element of the bilingual translation file *.locallang.xlf shall be updated to match the source from the monolingual source file. The _target_ element in the file should be updated to the newly provided translation.

Actual behavior:
If a translator edits a translation source, the _source_ element in locallang.xlf gets updated. Great!
If an editor edits a translation via GUI, the _source_ element in the respective *.locallang.xlf file gets set to the translated string. Instead, I would expect the source to be updated to the translation source from the loccallang.xlf file and the target to be set to the newly provided translation.

As far as I can see, we do need a mixed mode that allows to define a monolingual source file as well as bilingual translation files at the same time. I do believe that this is also what @UncleSamSwiss is asking for.

[nijel]

I think I now finally understand what is expected here, it would be great if somebody could test and comment fixes I've proposed in https://github.com/WeblateOrg/weblate/pull/2696

[nijel]

Thank you for your report, the issue you have reported has just been fixed by https://github.com/WeblateOrg/weblate/pull/2696.

• In case you see problem with the fix, please comment on this issue.
• In case you see similar problem, please open separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[MonsieurBon]

Will this only be released with 3.7 or is there any chance it could be shipped as a bugfix to 3.6?

[nijel]

There will be a bugfix release 3.6.1 in next days and it will contain this fix.

[UncleSamSwiss]

I just updated to 3.6.1 (using docker-compose) and the issue seems to stay the same (<target> is still missing).
Do I need to do something in the project/component configuration for this to work?
The string was already there before the update, but I translated it only after the update to 3.6.1.

A newly added translation looks like this:

    <trans-unit id="updateHeader" datatype="html">
        <source>Aktualisierungen</source>
        <context-group purpose="location">
          <context context-type="sourcefile">app/components/homescreen/homescreen.component.html</context>
          <context context-type="linenumber">13</context>
        </context-group>
        <note priority="1" from="description">Header for Update management</note>
      </trans-unit>

The configuration is still the same as reported above.

Server configuration and status

 * Weblate 3.6.1
 * Python 3.7.3rc1
 * Django 2.2
 * Celery 4.3.0
 * celery-batches 0.2
 * six 1.12.0
 * social-auth-core 3.1.0
 * social-auth-app-django 3.1.0
 * django-appconf 1.0.3
 * translate-toolkit 2.3.1
 * translation-finder 1.2
 * Whoosh 2.7.4
 * defusedxml 0.6.0
 * Git 2.20.1
 * Pillow 5.4.1
 * python-dateutil 2.8.0
 * lxml 4.3.2
 * django-crispy-forms 1.7.2
 * django_compressor 2.2
 * djangorestframework 3.9.2
 * user-agents 2.0
 * jellyfish 0.7.1
 * diff-match-patch 20121119
 * pytz 2019.1
 * pyuca 1.2
 * PyYAML 3.13
 * tesserocr 2.4.0
 * Mercurial 4.8.2
 * git-svn 2.20.1
 * Database backends: django.db.backends.postgresql
 * Cache backends: avatar:FileBasedCache, default:RedisCache
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 4.15.0-46-generic (x86_64)

[ktullemans]

Hi, We also see an issue within our hosted Weblate.
For us the target section isn't missing, but the source and target text have been switched after translating.

After pushing new translations, the source will be replaced by the translation and the target will be replaced by the source text.

Translation from 3 days ago:
<trans-unit xml:space="preserve" approved="no" id=".sg.ui.error.stream.geoblock"> <source>Afspelen niet toegestaan in deze regio</source> <target state="translated">Cannot playback in current region</target> <note from="developer">An error message</note> </trans-unit> <trans-unit xml:space="preserve" approved="no" id=".sg.ui.error.restart.cannot_play"> <source>Restart kan niet gestart worden op dit moment.</source> <target state="translated">Unable to restart the program at this time.</target> <note from="developer">An error message</note> </trans-unit>

Translation from more than a month ago:
<trans-unit xml:space="preserve" approved="no" id=".sg.ui.error.replay.offer_missing"> <source>Your subscriptions do not allow replaying this channel.</source> <target state="translated">Uw huidige abonnement is niet toereikend voor TV Gemist voor deze zender. Voor meer informatie over beschikbare abonnementen, bel onze klantenservice {phone}.</target> <note from="developer">An error message</note> </trans-unit>

We solved above issue for now by unchecking edit source/base file within component settings.
But it still means there's a bug while translating this file format.

[jilles-sg]

Turning off the component setting "Edit base file" "Whether users will be able to edit the base file for monolingual translations." appeared to fix the issue, but only once. Any further changes were made to <source> again. This is with Hosted Weblate, version 3.6.1.

[nijel]

This is not a 24x7 support, so please be patient when waiting for response. If need faster response times, you have an option to get it.

It seems there was bug in my fix which inverted the logic, sorry for that.

PS: The "Edit base file" will not change anything. I makes no sense for this to toggle how the file is being updated.

[SomeBdyElse]

Thank you @nijel for all your work on this issue. I just tested 78430b by modifying ./usr/local/lib/python3.7/dist-packages/weblate/formats/ttkit.py in the 3.6.1 docker container.

Unfortunately, the patch seems to have side effects:

My configuration is still the same as in https://github.com/WeblateOrg/weblate/issues/2668#issuecomment-481021716 .

Source language: German
Target languages: French, English, Italian
File mask: *.locallang.xlf
Monolingugal base language file: locallang.xlf
Edit base file: true

Content of fr.locallang.xlf

<trans-unit id="page.title" approved="yes">
   <source>Schweizer Alpen-Club SAC</source>
   <target state="translated">Club Alpin Suisse CAS</target>
</trans-unit>

Once I open the french translation screen of a translated object:

Actual:

• Target and source both show the same source value.
• Once I click save without changes, the target in the translation item is changed to equal the source. This shows as a committable change.

Expected:

• Target should show the target value from the bilingual translation file "Club Alpin Suisse CAS". This differs from the monolingual source "Schweizer Alpen-Club SAC".
• If I press save without changes, there is nothing to commit

[nijel]

Thanks for testing the patch. Just to clarify: saving does work as expected? What you've pointed out seems to be different issue...

[nijel]

The other issue should be addressed by a500b90bf7, testing is welcome.

[SomeBdyElse]

Thank you for your quick reaction!

Just to clarify: saving does work as expected?

Yes, the saving itself did work as expected.

I will test a500b90 as soon as I get the patch into my docker container.

[nijel]

I will test a500b90 as soon as I get the patch into my docker container.

If you can rebuild it, simply dropping the patch to the patches folder should do it.

[SomeBdyElse]

Hi @nijel ! I finally managed to test the latest commit a500b90. Unfortunately, I still ran into two issues:

My setup

Docker container:
https://github.com/WeblateOrg/docker/tree/2872a9b937946de4cf84b12f8d9a529a61d69c29

Plus patches:
784830be66e042bdd944b873b789c2c7a6bed4d8.patch
a500b90bf738cdb3996b7d2e8a4bc58da73c90a8.patch

My configuration is still the same as above.

Edit source

When I try to edit source, the "translation" appears empty:

Once I enter a new string and save it, it is correctly saved as the new source.

--- a/app/web/typo3conf/ext/usersaccassite/Resources/Private/Language/locallang.xlf
+++ b/app/web/typo3conf/ext/usersaccassite/Resources/Private/Language/locallang.xlf
@@ -1,10 +1,10 @@
-<?xml version="1.0" encoding="UTF-8"?>
+<?xml version='1.0' encoding='UTF-8'?>
 <xliff version="1.0">
-       <file source-language="de" datatype="plaintext" original="messages" date="2018-06-18T09:36:24Z" product-name="usersaccassite">
-               <header></header>
+       <file source-language="de" datatype="plaintext" original="messages" date="2018-06-18T09:36:24Z" product-name="usersaccassite" target-language="de">
+               <header/>
                <body>
-                       <trans-unit id="page.title" approved="yes">
-                               <source>Schweizer Alpen-Club SAC</source>
+                       <trans-unit id="page.title" approved="no">
+                               <source>Page title source</source>
                        </trans-unit>
                        <trans-unit id="page.title.theAlps" approved="yes">
                                <source>Die Alpen</source>

Instead of an empty textfield I would expect the textfield to show the source from the monolingual source file: "Schweizer Alpen-Club SAC" (see diff above).

Edit translation language (de -> en)

Once I edit one of the bilingual translation files, the source of the translated translation unit is cut back to the first character.

--- a/app/web/typo3conf/ext/usersaccassite/Resources/Private/Language/en.locallang.xlf
+++ b/app/web/typo3conf/ext/usersaccassite/Resources/Private/Language/en.locallang.xlf
@@ -19,8 +19,7 @@
                        <trans-unit id="mod.web_layout.backendLayouts.page.shop.full.title">
                                <source>Seite Shop volle Breite</source> <target>Page Shop full width</target> </trans-unit>
                        <trans-unit id="mod.web_layout.backendLayouts.pageDetail.title">
-                               <source>Seite mit Detail</source> <target>Page With Detail View</target>
-                       </trans-unit>
+                               <source>S</source> <target>Page With Detail view</target> </trans-unit>
                        <trans-unit id="mod.web_layout.backendLayouts.pageRightColumn.title">
                                <source>Seite Rechte Spalte</source> <target>Page Right Column</target>
                        </trans-unit>

Instead of the truncated source, I would expect the edited source to match the source from the monolingual source file "Seite mit Detail".

If you are interested, we could also meet up for a screen sharing session.

[nijel]

I've tried to create fixes on https://github.com/WeblateOrg/weblate/pull/2758, feedback welcome.

[SomeBdyElse]

I've tried to create fixes on #2758, feedback welcome.

Please find my feedback within the comments of that PR

[MonsieurBon]

@nijel will there be a Version 3.6.2 with this fix?

[reloxx13]

Heya,

thank you for this work!
Does all those fixes fix this?

Source was German, now i translated the target to "Deutsch" and in the output translated file the Source Tag got translated/changed too.

xliff1.2 and v3.6.1

[reloxx13]

Heya,
i tested the dev Version now and the Bug (writing translation in Source Tag) is fixed.

Due date for 3.7 is 15.6.2019, i hope you can hold this date. The Milestone looks almost completed (one issue left).

Looking forward to update at 16/17. 6. and stopping translations for XLIFF files till the update arrives.

BR and Thanks!