Synapse: Internal server error when using List Room API with no filtering

[anthr76]

Description

When requesting all rooms in your home server via admin API the following response is sent.

{
    "errcode": "M_UNKNOWN",
    "error": "Internal server error"
}

Steps to reproduce

• Possibly federate rooms with emojis in room names
• Curl /_synapse/admin/v1/rooms

This seems like it's an issue having emojis in the room names. When using a search term to narrow down the results the issue doesn't show up.

A traceback in synpase logs show:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/synapse/http/server.py", line 233, in _async_render_wrapper
    self._send_response(request, code, response)
  File "/usr/local/lib/python3.7/site-packages/synapse/http/server.py", line 295, in _send_response
    canonical_json=self.canonical_json,
  File "/usr/local/lib/python3.7/site-packages/synapse/http/server.py", line 536, in respond_with_json
    json_bytes = encode_pretty_printed_json(json_object) + b"\n"
  File "/usr/local/lib/python3.7/site-packages/canonicaljson.py", line 96, in encode_pretty_printed_json
    return _pretty_encoder.encode(json_object).encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode characters in position 10427-10428: ordinal not in range(1
28)

Some rooms not in my server but are federated by my server have emojis in the name. Which show up in the logs:

...'name': '🔊 Audio Corner', 'canonical_alias': '#audiocorner:jupiterbroadcasting.com', 'joined_members': 39, 'joined_local_members': 1, 'version': '5', 'creator': '', 'encryption': None, 'federatable': True, 'public': ...

Version information

• Homeserver:

If not matrix.org:
matrix.rabbito.tech

• Version:

{
    "python_version": "3.7.8",
    "server_version": "1.19.0"
}

• Install method:
  
  matrixdotorg docker container

• Platform:
  
  Containerized on Ubuntu 20.04 LTS

[clokep]

This sounds like the same as matrix-org/python-canonicaljson#33 -- a workaround is to specify a user-agent which isn't curl.

[anthr76]

This workaround succeeded !

[clokep]

Glad that worked! We'll need to release a new version of canonicaljson at some point with the fix in it. 👍

[clokep]

FYI we released an update for python-canonicaljson which should fix this. You can install 1.4.0 and it should work fine!

[richvdh]

see also #8262 which adds a dependency on canonicaljson 1.4.0, so people upgrading to synapse 1.21.0 will automatically get this fix.