Weblate: Out of memory

[bilalcorbacioglu]

Describe the bug

Hello,

We've been getting an "Out of Memory" error periodically since we update to Weblate 4.2.2. Especially when we edit some components, For example;

Source: And that is the end of our story.
Target: Voilà, c'est la fin de l'histoire. -> Voilà, c’est la fin de l’histoire.

Source: Touch a button!
Target: Touche un bouton ! -> Touche un bouton/u202f!

• I'm using a "Customized rule" but when I deactivate it, the error(Out of memory) continues.
• No records are forwarded to the error file during/after the error.

Screenshots

Before click "Save"

After Click "Save"

Server configuration and status

 * Weblate: 4.2.2
 * Django: 3.1.1
 * siphashc: 1.3
 * Whoosh: 2.7.4
 * translate-toolkit: 3.0.0
 * lxml: 4.5.2
 * Pillow: 7.2.0
 * bleach: 3.1.5
 * python-dateutil: 2.8.1
 * social-auth-core: 3.3.3
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.2
 * oauthlib: 3.1.0
 * django-compressor: 2.4
 * djangorestframework: 3.11.1
 * django-filter: 2.3.0
 * django-appconf: 1.0.4
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 40.8.0
 * jellyfish: 0.8.2
 * openpyxl: 3.0.5
 * celery: 4.4.7
 * kombu: 4.6.11
 * translation-finder: 2.1
 * html2text: 2020.1.16
 * pycairo: 1.16.2
 * pygobject: 3.30.4
 * diff-match-patch: 20200713
 * requests: 2.24.0
 * django-redis: 4.12.1
 * hiredis: 1.1.0
 * sentry_sdk: 0.16.5
 * Cython: 0.29.21
 * misaka: 2.1.1
 * GitPython: 3.1.7
 * borgbackup: 1.1.13
 * pyparsing: 2.4.7
 * Python: 3.7.3
 * Git: 2.20.1
 * psycopg2: 2.8.5
 * psycopg2-binary: 2.8.5
 * phply: 1.2.5
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.10
 * tesserocr: 2.5.1
 * akismet: 1.1
 * boto3: 1.14.53
 * zeep: 3.4.0
 * aeidon: 1.7.0
 * iniparse: 0.5
 * mysqlclient: 2.0.1
 * Mercurial: 5.5.1
 * git-svn: 2.20.1
 * git-review: 1.28.0
 * hub: 2.13.0
 * lab: 0.16
 * Redis server: 5.0.9
 * PostgreSQL server: 11.7
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.gmail.com
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://cache:6379/1, redis://cache:6379/1, regular
 * Platform: Linux 4.15.0-117-generic (x86_64)

[nijel]

You might want to reduce number of processes running inside the container, see https://docs.weblate.org/en/latest/admin/install/docker.html#container-settings

[github-actions[bot]]

This issue looks like a support question. We try to answer these reasonably fast, but in case you are looking for faster resolution, please consider purchasing support subscription and make Weblate stronger.

[bilalcorbacioglu]

This method did a little help with Weblate's freezing problem. However, Weblate continues to freezes after a few requests because CPU usage is always high(After trying to translate the translation). I also share a few other error logs I can catch below.

weblate_1   | celery-main stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,058: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,104: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:35:48,105: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:35:48,481: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[f6356314-2794-4a9f-a3e6-a9d3e26b37d1]  
weblate_1   | celery-backup stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from translate@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:07,442: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from memory@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:10,318: INFO/MainProcess] missed heartbeat from main@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:13,885: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-backup stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from notify@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-memory stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-notify stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-translate stderr | [2020-09-25 06:36:23,351: INFO/MainProcess] missed heartbeat from backup@cec1c82ad77a
weblate_1   | celery-main stderr | [2020-09-25 06:36:53,270: WARNING/MainProcess] Substantial drift from translate@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-main stderr | 45 seconds.  [orig: 2020-09-25 06:36:46.970975 recv: 2020-09-25 06:36:01.972813]
weblate_1   | celery-main stderr | 
weblate_1   | celery-notify stderr | [2020-09-25 06:36:46,899: WARNING/MainProcess] Substantial drift from translate@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-notify stderr | 44 seconds.  [orig: 2020-09-25 06:36:45.495046 recv: 2020-09-25 06:36:01.972813]
weblate_1   | celery-notify stderr | 
weblate_1   | celery-translate stderr | [2020-09-25 06:36:46,899: WARNING/MainProcess] Substantial drift from notify@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-translate stderr | 44 seconds.  [orig: 2020-09-25 06:36:45.495150 recv: 2020-09-25 06:36:01.972873]
weblate_1   | celery-translate stderr | 
weblate_1   | celery-backup stderr | [2020-09-25 06:36:53,825: WARNING/MainProcess] Substantial drift from translate@cec1c82ad77a may mean clocks are out of sync.  Current drift is
weblate_1   | celery-backup stderr | 52 seconds.  [orig: 2020-09-25 06:36:53.275485 recv: 2020-09-25 06:36:01.972813]
weblate_1   | celery-backup stderr 

celery-notify stderr | [2020-09-18 09:33:37,127: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-notify stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:33:37,127: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-main stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-memory stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-memory stderr | [2020-09-18 09:33:37,128: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:33:37,942: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[ea475680-12b2-457e-b173-69794d161dd7]  
uwsgi stderr | [pid: 703|app: 0|req: 497/3082] 178.62.34.228 () {56 vars in 1323 bytes} [Fri Sep 18 09:29:45 2020] GET /translate/onec/r2-x-b/fr_CD/?checksum=450fc80ce224094d => generated 0 bytes in 232671 msecs (HTTP/1.0 500) 6 headers in 0 bytes (0 switches on core 0)
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/fields/related_descriptors.py", line 904, in get_queryset
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/manager.py", line 147, in get_queryset
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 191, in __init__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/query.py", line 227, in __init__
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 942, in render
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 192, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 172, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 942, in render
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/accounts/middleware.py", line 73, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 192, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 172, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 942, in render
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 168, in render
uwsgi stderr |   File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/context.py", line 244, in bind_template
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/context_processors.py", line 160, in weblate_context
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/templatetags/compress.py", line 143, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/templatetags/compress.py", line 97, in render_compressed
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/templatetags/compress.py", line 67, in render_offline
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/cache.py", line 61, in get_offline_hexdigest
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/compressor/cache.py", line 22, in get_hexdigest
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/encoding.py", line 79, in smart_bytes
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/encoding.py", line 99, in force_bytes
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 168, in render
uwsgi stderr |   File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/context.py", line 244, in bind_template
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/context_processors.py", line 160, in weblate_context
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 312, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 446, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/urls/base.py", line 87, in reverse
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/urls/resolvers.py", line 654, in _reverse_with_prefix
uwsgi stderr |   File "/usr/lib/python3.7/re.py", line 183, in search
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 76, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/util.py", line 224, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/shortcuts.py", line 19, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader.py", line 62, in render_to_string
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 168, in render
uwsgi stderr |   File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/context.py", line 244, in bind_template
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/context_processors.py", line 160, in weblate_context
uwsgi stderr | MemoryError
uwsgi stderr | 
uwsgi stderr | During handling of the above exception, another exception occurred:
uwsgi stderr | 
uwsgi stderr | Traceback (most recent call last):
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 47, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/deprecation.py", line 114, in __call__
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 49, in inner
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 103, in response_for_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/core/handlers/exception.py", line 142, in handle_uncaught_exception
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/trans/views/error.py", line 79, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/decorators.py", line 130, in _wrapped_view
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/views/defaults.py", line 97, in server_error
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/backends/django.py", line 61, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 170, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/loader_tags.py", line 150, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 162, in _render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 312, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 938, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/base.py", line 905, in render_annotated
2020-09-18 09:34:04,528 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140476684648632 for <Subprocess at 140476684916888 with name uwsgi in state RUNNING> (stderr)> (<class 'MemoryError'>: [/usr/local/lib/python3.7/dist-packages/supervisor/supervisord.py|runforever|220] [/usr/local/lib/python3.7/dist-packages/supervisor/dispatchers.py|handle_read_event|268] [/usr/local/lib/python3.7/dist-packages/supervisor/options.py|readfd|1564])
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/template/defaulttags.py", line 446, in render
uwsgi stderr |   File "/usr/local/lib/python3.7/dist-packages/django/urls/base.py", line 87, in reverse
celery-main stderr | [2020-09-18 09:34:59,639: INFO/MainProcess] Received task: weblate.wladmin.tasks.configuration_health_check[ed81a207-35a7-4b5c-9c18-30f87f960908]  
celery-main stderr | [2020-09-18 09:34:59,668: INFO/MainProcess] Received task: weblate.accounts.tasks.cleanup_social_auth[72f44bba-17b3-40ab-8314-7008afeaf68e]  
celery-main stderr | [2020-09-18 09:34:59,673: INFO/MainProcess] Received task: weblate.trans.tasks.commit_pending[a699fe20-02c4-4930-88c4-8f9161f99901]  
celery-main stderr | [2020-09-18 09:34:59,727: INFO/MainProcess] Received task: weblate.accounts.tasks.cleanup_auditlog[0591408c-effb-450b-b3c1-345e30ba47e1]  
celery-main stderr | [2020-09-18 09:35:00,894: INFO/MainProcess] Received task: weblate.utils.tasks.heartbeat[78aae120-1d86-4982-864a-6a08b72855ff]  
celery-backup stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:11,790: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:12,376: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:12,741: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:12,934: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,025: INFO/MainProcess] missed heartbeat from memory@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,303: INFO/MainProcess] missed heartbeat from backup@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from main@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,447: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-backup stderr | [2020-09-18 09:35:19,989: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-main stderr | [2020-09-18 09:35:19,989: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-memory stderr | [2020-09-18 09:35:19,990: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-notify stderr | [2020-09-18 09:35:19,989: INFO/MainProcess] missed heartbeat from translate@8f73840ea4df
celery-translate stderr | [2020-09-18 09:35:19,990: INFO/MainProcess] missed heartbeat from notify@8f73840ea4df
celery-main stderr | ERROR Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | [2020-09-18 09:35:23,363: ERROR/ForkPoolWorker-3] Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | ERROR Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | ERROR Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.accounts.tasks.cleanup_social_auth of weblate at 0x7ffae35337b8>, 'task_id': '72f44bba-17b3-40ab-8314-7008afeaf68e', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad9596308>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | [2020-09-18 09:35:24,065: ERROR/ForkPoolWorker-3] Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.accounts.tasks.cleanup_social_auth of weblate at 0x7ffae35337b8>, 'task_id': '72f44bba-17b3-40ab-8314-7008afeaf68e', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad9596308>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | [2020-09-18 09:35:23,630: ERROR/ForkPoolWorker-2] Failure while executing task: OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | ERROR Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.trans.tasks.commit_pending of weblate at 0x7ffae35337b8>, 'task_id': 'a699fe20-02c4-4930-88c4-8f9161f99901', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad8ef1a48>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | [2020-09-18 09:35:24,994: ERROR/ForkPoolWorker-2] Failure while executing task: OperationalError: {'signal': <Signal: task_failure providing_args={'task_id', 'traceback', 'args', 'kwargs', 'exception', 'einfo'}>, 'sender': <@task: weblate.trans.tasks.commit_pending of weblate at 0x7ffae35337b8>, 'task_id': 'a699fe20-02c4-4930-88c4-8f9161f99901', 'args': [], 'kwargs': {}, 'traceback': <traceback object at 0x7ffad8ef1a48>, 'einfo': <ExceptionInfo: OperationalError('could not fork new process for connection: Out of memory\n\ncould not fork new process for connection: Out of memory\n')>}
celery-main stderr | ERROR Failure while executing task
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | psycopg2.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | 
celery-main stderr | The above exception was the direct cause of the following exception:
celery-main stderr | 
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
celery-main stderr |     R = retval = fun(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
celery-main stderr |     return self.run(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/accounts/tasks.py", line 40, in cleanup_social_auth
celery-main stderr |     for partial in Partial.objects.iterator():
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 360, in _iterator
celery-main stderr |     yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 53, in __iter__
celery-main stderr |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1152, in execute_sql
celery-main stderr |     cursor = self.connection.chunked_cursor()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 271, in chunked_cursor
celery-main stderr |     self._named_cursor_idx,
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 235, in _cursor
celery-main stderr |     self.ensure_connection()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
celery-main stderr |     raise dj_exc_value.with_traceback(traceback) from exc_value
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | django.db.utils.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
2020-09-18 09:35:25,600 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140476684580848 for <Subprocess at 140476684916216 with name celery-main in state RUNNING> (stderr)> (<class 'MemoryError'>: [/usr/local/lib/python3.7/dist-packages/supervisor/supervisord.py|runforever|220] [/usr/local/lib/python3.7/dist-packages/supervisor/dispatchers.py|handle_read_event|268] [/usr/local/lib/python3.7/dist-packages/supervisor/options.py|readfd|1564])
celery-main stderr | [2020-09-18 09:35:24,164: ERROR/ForkPoolWorker-3] Failure while executing task
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | psycopg2.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | 
celery-main stderr | The above exception was the direct cause of the following exception:
celery-main stderr | 
celery-main stderr | Traceback (most recent call last):
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
celery-main stderr |     R = retval = fun(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
celery-main stderr |     return self.run(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/weblate/accounts/tasks.py", line 40, in cleanup_social_auth
celery-main stderr |     for partial in Partial.objects.iterator():
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 360, in _iterator
celery-main stderr |     yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 53, in __iter__
celery-main stderr |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1152, in execute_sql
celery-main stderr |     cursor = self.connection.chunked_cursor()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 271, in chunked_cursor
celery-main stderr |     self._named_cursor_idx,
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 235, in _cursor
celery-main stderr |     self.ensure_connection()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__
celery-main stderr |     raise dj_exc_value.with_traceback(traceback) from exc_value
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 219, in ensure_connection
celery-main stderr |     self.connect()
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/base/base.py", line 200, in connect
celery-main stderr |     self.connection = self.get_new_connection(conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/utils/asyncio.py", line 26, in inner
celery-main stderr |     return func(*args, **kwargs)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
celery-main stderr |     connection = Database.connect(**conn_params)
celery-main stderr |   File "/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py", line 127, in connect
celery-main stderr |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
celery-main stderr | django.db.utils.OperationalError: could not fork new process for connection: Out of memory
celery-main stderr | 
celery-main stderr | could not fork new process for connection: Out of memory
celery-main stderr | 
2020-09-18 09:35:30,965 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140476684581576 for <Subprocess at 140476684916440 with name celery-notify in state RUNNING> (stderr)> (<class 'MemoryError'>: [/usr/local/lib/python3.7/dist-packages/supervisor/supervisord.py|runforever|220] [/usr/local/lib/python3.7/dist-packages/supervisor/dispatchers.py|handle_read_event|268] [/usr/local/lib/python3.7/dist-packages/supervisor/options.py|readfd|1564])

My settings-override.py,

REQUIRE_LOGIN = False
INSTALLED_APPS = [
    # "o_customization",
    # Weblate apps on top to override Django locales and templates
    "weblate.addons",
    "weblate.auth",
    "weblate.checks",
    "weblate.formats",
    "weblate.glossary",
    "weblate.machinery",
    "weblate.trans",
    "weblate.lang",
    "weblate.langdata",
    "weblate.memory",
    "weblate.screenshots",
    "weblate.fonts",
    "weblate.accounts",
    "weblate.utils",
    "weblate.vcs",
    "weblate.wladmin",
    "weblate",
    # Optional: Git exporter
    "weblate.gitexport",
    # Standard Django modules
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "django.contrib.admin.apps.SimpleAdminConfig",
    "django.contrib.admindocs",
    "django.contrib.sitemaps",
    "django.contrib.humanize",
    # Third party Django modules
    "social_django",
    "crispy_forms",
    "compressor",
    "rest_framework",
    "rest_framework.authtoken",
    "django_filters",
]
REST_FRAMEWORK = {
    # Use Django's standard `django.contrib.auth` permissions,
    # or allow read-only access for unauthenticated users.
    "DEFAULT_PERMISSION_CLASSES": [
        # Require authentication for login required sites
        "rest_framework.permissions.IsAuthenticated"
        if REQUIRE_LOGIN
        else "rest_framework.permissions.IsAuthenticatedOrReadOnly"
    ],
    "DEFAULT_AUTHENTICATION_CLASSES": (
        "rest_framework.authentication.TokenAuthentication",
        "weblate.api.authentication.BearerAuthentication",
        "rest_framework.authentication.SessionAuthentication",
    ),
    "DEFAULT_THROTTLE_CLASSES": (
        "weblate.api.throttling.UserRateThrottle",
        "weblate.api.throttling.AnonRateThrottle",
    ),
    "DEFAULT_THROTTLE_RATES": {"anon": "100/day", "user": "25000/hour"},
    "DEFAULT_PAGINATION_CLASS": ("rest_framework.pagination.PageNumberPagination"),
    "PAGE_SIZE": 500,
    "VIEW_DESCRIPTION_FUNCTION": "weblate.api.views.get_view_description",
    "UNAUTHENTICATED_USER": "weblate.auth.models.get_anonymous",
}
CHECK_LIST += (
  # "o_customization.checks.PunctuationSpacingCheckO",
)
AUTOFIX_LIST += (
  # "weblate.trans.autofixes.chars.ReplaceTrailingDotsWithEllipsis",
  # "o_customization.autofix.PunctuationSpacingAutoFixO"
)

[nijel]

It still seems to be out of memory issues. Can you look at individual processes how much of them is there and which consume the memory?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nijel]

I've identified memory issue in https://github.com/WeblateOrg/translation-finder/, it is fixed in the 2.4 release and Weblate 4.3.1 will require that.

[github-actions[bot]]

Thank you for your report, the issue you have reported has just been fixed.

• In case you see a problem with the fix, please comment on this issue.
• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, consider supporting Weblate by donating.

[mareksuscak]

EDIT: Updated to bleeding edge 4.3.1 and so far so good! Workers are not going down anymore.

Would this issue explain Weblate 4.3 going down after a while and all workers constantly restarting? That doesn't seem normal and this last release has been somewhat buggy.

[nijel]

Quite likely - there is a memory limit set for the workers and with the leak it is more likely it will reach it.