Synapse: Workers running under PyPy attempt to load psycopg2 (not psycopg2cffi) and crash

[intelfx]

Description

When attempting to run Synapse 0.99.3 (with PostgreSQL and workers) under PyPy, it somehow still tries to load regular psycopg2 (instead of psycopg2cffi) and crashes.

Steps to reproduce

• create a PyPy virtualenv
• install synapse, psycopg2cffi via pip
• start any worker

Logs

Apr 13 17:53:14 stratofortress systemd[1]: Started Synapse Matrix homeserver worker federation_reader-0.
Apr 13 17:53:23 stratofortress root[23166]: [None] ***** STARTING SERVER *****
Apr 13 17:53:23 stratofortress root[23166]: [None] Server /home/operator/venv/pypy/site-packages/synapse/app/federation_reader.py version 0.99.3 (b=master,7963d79c3)
Apr 13 17:53:23 stratofortress root[23166]: [None] Server hostname: intelfx.name
Apr 13 17:53:23 stratofortress twisted[23166]: [None] Redirected stdout/stderr to logs
Apr 13 17:53:23 stratofortress synapse.server[23166]: [None] Setting up.
Apr 13 17:53:23 stratofortress synapse.config.appservice[23166]: [None] Loaded application service: ApplicationService: {'token': '<redacted>', 'url': 'http://localhost:8409', 'hs_token': '<redacted>', 'sender': '@appservice-irc:intelfx.name', 'server_name': 'intelfx.name', 'namespaces': {'users': [{'exclusive': True, 'regex': re.compile('@fn_.*:.*')}, {'exclusive': True, 'regex': re.compile('@irc_.*_.*:.*')}], 'aliases': [{'exclusive': True, 'regex': re.compile('#fn_.*:.*')}, {'exclusive': True, 'regex': re.compile('#irc_.*_.*:.*')}], 'rooms': []}, 'id': '134313c1983d17ed2560eeff6ef95db173c5e42861f82d8f07f67054a7fd458d', 'ip_range_whitelist': None, 'protocols': set(), 'rate_limited': True}
Apr 13 17:53:23 stratofortress twisted[23166]: [] Traceback (most recent call last):
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/opt/pypy3/lib-python/3/runpy.py", line 193, in _run_module_as_main
Apr 13 17:53:23 stratofortress twisted[23166]: []     "__main__", mod_spec)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/opt/pypy3/lib-python/3/runpy.py", line 85, in _run_code
Apr 13 17:53:23 stratofortress twisted[23166]: []     exec(code, run_globals)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/app/federation_reader.py", line 186, in <module>
Apr 13 17:53:23 stratofortress twisted[23166]: []     start(sys.argv[1:])
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/app/federation_reader.py", line 178, in start
Apr 13 17:53:23 stratofortress twisted[23166]: []     ss.setup()
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/server.py", line 224, in setup
Apr 13 17:53:23 stratofortress twisted[23166]: []     self.datastore = self.DATASTORE_CLASS(conn, self)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/account_data.py", line 30, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedAccountDataStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/account_data.py", line 47, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(AccountDataWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/pushers.py", line 26, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedPusherStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/push_rule.py", line 28, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedPushRuleStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/receipts.py", line 40, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedReceiptsStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/events.py", line 61, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(SlavedEventStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/push_rule.py", line 74, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(PushRulesWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/appservice.py", line 59, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(ApplicationServiceWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/receipts.py", line 43, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(ReceiptsWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/event_push_actions.py", line 70, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(EventPushActionsWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/stream.py", line 164, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(StreamWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/state.py", line 375, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(StateGroupWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/registration.py", line 32, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(RegistrationWorkerStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/room.py", line 24, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(RoomStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/transactions.py", line 61, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(TransactionStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/replication/slave/storage/_base.py", line 37, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     super(BaseSlavedStore, self).__init__(db_conn, hs)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/storage/_base.py", line 203, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     self._db_pool = hs.get_db_pool()
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/server.py", line 540, in _get
Apr 13 17:53:23 stratofortress twisted[23166]: []     dep = builder()
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/synapse/server.py", line 404, in build_db_pool
Apr 13 17:53:23 stratofortress twisted[23166]: []     **self.db_config.get("args", {})
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/twisted/enterprise/adbapi.py", line 211, in __init__
Apr 13 17:53:23 stratofortress twisted[23166]: []     self.dbapi = reflect.namedModule(dbapiName)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/twisted/python/reflect.py", line 159, in namedModule
Apr 13 17:53:23 stratofortress twisted[23166]: []     topLevel = __import__(name)
Apr 13 17:53:23 stratofortress twisted[23166]: []   File "/home/operator/venv/pypy/site-packages/psycopg2/__init__.py", line 50, in <module>
Apr 13 17:53:23 stratofortress twisted[23166]: []     from psycopg2._psycopg import (                     # noqa
Apr 13 17:53:23 stratofortress twisted[23166]: [] ImportError: /home/operator/venv/pypy/site-packages/psycopg2/_psycopg.pypy3-71-x86_64-linux-gnu.so: undefined symbol: timeradd
Apr 13 17:53:23 stratofortress systemd[1]: synapse-worker@federation_reader-0.service: Main process exited, code=exited, status=1/FAILURE
Apr 13 17:53:23 stratofortress systemd[1]: synapse-worker@federation_reader-0.service: Failed with result 'exit-code'.

Version information

• Homeserver: intelfx.name
• Version: Synapse 0.99.3 / PyPy 7.1.0
• Install method: pip
• Platform: Arch GNU/Linux

[mtippmann]

what worked for me is to uninstall psycopg2 and install the psycopg2 compat shim from psycopg2cffi (just add a psycopg2.py in site-packages with the content below:

From: https://pypi.org/project/psycopg2cffi/#installation

To use this package with Django or SQLAlchemy invoke a compatibility hook (for example, from settings.py in case of Django, or from a psycopg2.py file in site-packages of your virtual env):

from psycopg2cffi import compat
compat.register()

This will map psycopg2cffi to psycopg2, so now any code that does import psycopg2 will use psycopg2cffi.

Additionally I got errors due to a missing function - patching psycopg2cffi with a pull request did help:

$ cd /opt/venvs/matrix-synapse-pypy3/site-packages (or where your pypy3 venv for synapse is) 
$ curl https://patch-diff.githubusercontent.com/raw/chtd/psycopg2cffi/pull/98.patch | patch -p1 

[richvdh]

we swap in psycopg2cffi in synapse.storage.engines.create_engine but not synapse.storage.database.make_pool :man_facepalming:

[richvdh]

incidentally, this is trivially fixed by anyone with the inclination...