Element-web: room directory is very very slow over federation as that API doesn't let you paginate

[richvdh]

It takes at least 90 seconds to get a room list

[ara4n]

@erikjohnston are you seeing this performance prob?

[richvdh]

(ftr, it has never been quick. I assume it pulls the entire room list from matrix.org before responding to the client)

[erikjohnston]

yes, it is quite slow

[ara4n]

...and it gets stuck issuing 50x's half the time :(

[ara4n]

I've split this into two synapse bugs: matrix-org/synapse#1980 (general perf problems) and matrix-org/synapse#1981 (client-reader gets entirely wedged).

[ara4n]

so @erikjohnston seems to have fixed the 'general perf problems' part of this, but the problem over federation still exists. Apparently this is because the publicRooms API simply doesn't paginate at all over federation.

[ara4n]

apparently it should be usable but slow over federation (i.e. it's as slow as if you weren't paginating).

[ara4n]

reprioritising appropriately

[martindale]

Current status?

[ara4n]

still waiting for someone to make the directory api paginate over federation, sadly :(

[ara4n]

This is going to get really bad with the advent of hosted homeservers.

[ara4n]

I'm also not sure whether the problem is really a question of pagination or caching problems. This might also improve if we ever land the matthew/stats branch, so that the public room directory can just be pulled straight out of a room_stats table rather than having to do more sophisticated shenanigans?

[martindale]

Any interest in a merkle tree implementation for federation of large lists?
🤷‍♂️

[turt2live]

Searching matrix.org now that the directory is paginated is pretty quick.