Synapse: GroupRoomServlet AttributeError: 'NoneType' object has no attribute 'content' - on group rooms query

[MurzNN]

My group work well with some users and rooms. But after adding some new rooms (via list of id's) - it stops working. On '/_matrix/client/r0/groups/+ru:ru-matrix.org/rooms it returns {"errcode":"M_UNKNOWN","error":"Internal server error"} and in Synapse logs I see those error log:

2019-08-06 10:07:33,578 - synapse.http.server - 108 - ERROR - GET-2439892- Failed handle request via 'GroupRoomServlet': <XForwardedForRequest at 0x7f9ac3dfcb70 method='GET' uri='/_matrix/client/r0/groups/+ru:ru-matrix.org/rooms?access_token=<redacted>' clientproto='HTTP
/1.0' site=8008>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/storage/_base.py", line 542, in runWithConnection
    defer.returnValue(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1362, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/storage/_base.py", line 502, in runInteraction
    defer.returnValue(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1362, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 76, in wrapped_request_handler
    await h(self, request)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 315, in _async_render
    callback_return = await callback_return
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/logging/opentracing.py", line 480, in _trace_servlet_inner
    result = yield defer.maybeDeferred(func, request, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/client/v2_alpha/groups.py", line 347, in on_GET
    group_id, requester_user_id
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/groups/groups_server.py", line 524, in get_rooms_in_group
    room_id, len(joined_users), with_alias=False, allow_private=True
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/handlers/room_list.py", line 427, in generate_room_entry
    result["m.federate"] = create_event.content.get("m.federate", True)
AttributeError: 'NoneType' object has no attribute 'content'

Others room queries (/summary and /users) for this group works well without errors. For other groups /rooms querues works well too.

Restarting Synapse don't helps. Synapse version is 1.2.1.

What could happen with my room? And does anybody have ideas how to recover it?

[MurzNN]

Via query SELECT * FROM group_rooms WHERE group_id = '+ru:ru-matrix.org' I see list of 66 valid rooms id, with is_public = true, and no visible errors.
Via SELECT * FROM group_users WHERE group_id = '+ru:ru-matrix.org' - list of valud users mxid without errors too.

Via SELECT * FROM group_attestations_renewals WHERE group_id = '+ru:ru-matrix.org' and SELECT * FROM group_attestations_remote WHERE group_id = '+ru:ru-matrix.org' I see list of valid mxid's too.

Maybe there are some problem with attestation_json field, but I can't find any spec how this must work. Can I try to clear this records and restart?

[MurzNN]

Seems this problem similar to #4786 and #4891?

[MurzNN]

My quick and ugly hack for workaround this problem is to add checking property, so in file synapse/handlers/room_list.py

        # Return whether this room is open to federation users or not
        create_event = current_state.get((EventTypes.Create, ""))
        result["m.federate"] = create_event.content.get("m.federate", True)

becomes:

        # Return whether this room is open to federation users or not
        create_event = current_state.get((EventTypes.Create, ""))
        if hasattr(create_event, 'content'):
          result["m.federate"] = create_event.content.get("m.federate", True)

After this my community starts working again, without error! But this is only workaround, right solution is understanding why create_event have no content property... Maybe this is because anybody try to add not valid room_id to community?
Or we checking room, that have no EventTypes.Create event in current state...

[MurzNN]

Here is screenshot of room list from problematic community:

Seems this is unknown rooms, that caused current issue.

[MurzNN]

Possible ways to reproduce:

1. Add to community room with not-exists id.
2. Add to community room with id, that not federated to current server.
3. Add to community exists room, and leave this room from all users on your homeserver.

[richvdh]

dup #4891 I think