Weblate: Some translations are not automatically committed

[surli]

Describe the bug
Some of the translations made on my components are not automatically committed, while using the "Commit" button in the repository management UI works properly to commit them.

Expected behavior
The components are configured to be committed every 1 hour, so I expect them to be committed 1 hour after the translation has been done.

Server configuration and status

 * Weblate 3.7.1
 * Python 2.7.9
 * Django 1.11.13
 * Celery 4.2.1
 * celery-batches 0.2
 * six 1.11.0
 * social-auth-core 3.2.0
 * social-auth-app-django 3.1.0
 * django-appconf 1.0.2
 * translate-toolkit 2.4.0
 * translation-finder 1.6
 * Whoosh 2.7.4
 * defusedxml 0.5.0
 * Git 2.11.0
 * Pillow 5.1.0
 * python-dateutil 2.7.2
 * lxml 4.2.1
 * django-crispy-forms 1.7.2
 * django_compressor 2.2
 * djangorestframework 3.8.2
 * user-agents 1.1.0
 * jellyfish 0.6.1
 * diff-match-patch 20121119
 * pytz 2018.4
 * pyuca 1.2
 * ruamel.yaml 0.15.100
 * hub 2.3.0-pre10-136-g1e5bb95
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
 * Celery: redis://localhost:6379, redis://localhost:6379, regular
 * Platform: Linux 4.9.0-9-amd64 (x86_64)

SystemCheckError: System check identified some issues:

ERRORS:
?: (weblate.W023) Please upgrade your installation to Python 3. Python 2 support will be dropped in Weblate 4.0 currently sheduled on April 2020.
    HINT: https://docs.weblate.org/en/weblate-3.7.1/admin/updare.html#py3

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (weblate.W025.ass) Failure in loading handler for ass file format: gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-3.7.1/admin/install.html#optional-deps
?: (weblate.W025.srt) Failure in loading handler for srt file format: gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-3.7.1/admin/install.html#optional-deps
?: (weblate.W025.ssa) Failure in loading handler for ssa file format: gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-3.7.1/admin/install.html#optional-deps
?: (weblate.W025.sub) Failure in loading handler for sub file format: gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-3.7.1/admin/install.html#optional-deps

INFOS:
?: (weblate.I021) Error collection is not configured, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-3.7.1/admin/install.html#collecting-errors



md5-99b0a820ddf3d310627c62c832ee5207



# Celery worker configuration for production
CELERY_TASK_ALWAYS_EAGER = False
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = CELERY_BROKER_URL

# Celery settings, it is not recommended to change these
CELERY_WORKER_PREFETCH_MULTIPLIER = 0
CELERY_BEAT_SCHEDULE_FILENAME = os.path.join(
    DATA_DIR, 'celery', 'beat-schedule'
)
CELERY_TASK_ROUTES = {
    'weblate.trans.search.*': {'queue': 'search'},
    'weblate.trans.tasks.optimize_fulltext': {'queue': 'search'},
    'weblate.trans.tasks.cleanup_fulltext': {'queue': 'search'},
    'weblate.memory.tasks.*': {'queue': 'memory'},
}

# We want by default to commit after 1 hour
COMMIT_PENDING_HOURS=1

I checked the log of Celery on my server, I don't see any error, and I even can see some:

Received task: weblate.trans.tasks.perform_commit[35ed4072-7c0e-4b4c-afe3-4ae3b5d00763]

Moreover, if I perform locally weblate commit_pending --all (which is performed every hour by a cron), I can see the following line:

Committing XWiki Contrib/Mocca Calendar Macro

However, when I check the translation file of this exact component on the disk (in data/vcs), it does not reflect what has been done on the weblate interface: the translation have not been written to the file, and the repository management still tell me that I have uncommitted changes.

On the other hand, if I click on the "Commit" button on the repository management UI, I do get the translations written on the disk and I do see the status changing in the repository management UI.

So maybe my setup is somewhere wrong for Celery? For the record, we upgraded from Weblate 2 to Weblate 3 at the beginning of the year, and apparently it started to not work properly at this time. But back then, we did not set the COMMIT_PENDING_HOURS property in the settings of each component.

[nijel]

Do you happen to use linked components? There were some issues in handling those recently addressed see #2813. That issue also includes improvements to logging to log errors in Celery tasks even if error collection is not configured.

PS: The commit in your case can take up to two hours - the pending changes are committed hourly and it looks for changes older than hour, so in worst case it can be up to two hours.

[surli]

Do you happen to use linked components?

we are using heavily linked components but it looks like the problem also reproduce on single components.

[nijel]

Can you please try if patches mentioned in https://github.com/WeblateOrg/weblate/issues/2813#issuecomment-515640258 will help you?

[surli]

It's a bit complicated since I'm in a prod environment: I'm performing the upgrades on release only, so I'd prefer to wait 3.7.8.

Now I realized that when upgrading from 2.x.x to 3.x.x, we kept a cron job that were performing the following commands every hours:

weblate updategit --all
weblate commit_pending --all --age=1
weblate pushgit --all

is it possible that it interfered with the lazy commit mechanism and Celery?
For now I performed the following actions:

• disabled this cron
• activated push_on_commit on all my components
• performed manually a weblate commitgit --all && weblate pushgit --all which worked properly

I'll now check regularly what happens, if it's ok I leave this issue open until the situation is more clear.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[surli]

So I just upgraded to Weblate 3.8 and apparently the bug is indeed fixed: we got the automatic push again.

[surli]

It is possible to reopen this issue? I spoke too fast last time: apparently it looks like it did work just after restarting weblate and celery after the upgrade, but now it's again not committing automatically the translations.

I checked celery logs and apparently the translations are found as "pending commits" but they are never committed. I also tried again to run manually weblate commit_pending --all in that case I see the components that needs to be committed, but they are not. OTOH when I do weblate commitgit --all everything is properly committed.

Here the problem is about linked components, I don't know if it happens for other ones.

All my components are configured with push on commits and 1 hour for commit_pending.

[nijel]

@surli I've created new issue for this: https://github.com/WeblateOrg/weblate/issues/3005