Matrix-doc: Room Versioning

[richvdh]

Documentation: https://docs.google.com/document/d/1urKgReoHqxX8R_XtySB17dPi-DZcKhqTEL2_s895Wz0
PRs: #1516

supercedes #1196.

[richvdh]

Last call for comments on this before we go ahead and start implementation next week.

[erikjohnston]

Are we using integers or strings for room versions? Benefits of strings is that it would make it a) clearer that you can't use ranges, and b) makes it easier to have versions like test-version-foo

OTOH, I'm not sure we want to be encouraging weird room versions in the wild

[turt2live]

The error schema implies integers, which I think is a good thing. It's much easier to determine if a room is outdated when it's an integer.

[non-Jedi]

The wording in this document makes it sound like the actual spec PR may have different "mechanics" than outlined in this document. Specifically, the phrase, "might look like". I just want to be clear that if the actual way that room versions work ends up being different than outlined in the mechanics section of this document, a new spec proposal will be made before a spec PR is merged, right?

Other than that, here's some points I'd like clarification on:

• Room versions are supposed to mark changes in mechanics for both clients and servers, right? But the way the proposal is worded, it seems to say that servers get to set the default version of rooms. Does this mean that outdated clients will no longer be able to create rooms that they can use on servers that set a default room version other than 1?
• Will clients be able to override the default room version when creating rooms (this only partially helps with previous point since many clients may not be aware of room version at all)?
• Will the spec continue to include sections about old room mechanics even as version 2 gets rolled out? It seems that it should since homeservers at least must implement both version 1 and version 2 room mechanics. Clients for a while probably ought to as well.
• In theory, could the version of a room affect how a client interacts with it before they've joined the room? For example if we implement different semantics for peeking into a room. If so, do we need to add an endpoint for checking room version or something that must be called before a client attempts to interact with the room?

For the record, I really like the general approach of this proposal. Worrying about room versioning before worrying about upgrade path seems like the right way to go about things.

[KitsuneRal]

There's not a word about exposing the room version to clients; exposing the join error is only mentioned. Given that the room version may (potentially greatly, in case of event id formats etc.) affect the clients, should this proposal mention where to find the room version in CS API? Or are we deliberately keeping it a server-server thing for now, given that there no _actually immediate_ changes that would need the clients to know about the room version?

[richvdh]

@erikjohnston / @turt2live : let's define them as integers.

[richvdh]

@non-Jedi:

The wording in this document makes it sound like the actual spec PR may have different "mechanics" than outlined in this document. Specifically, the phrase, "might look like"

Sorry, that phrasing was really just an artefact of the text being copied from an earlier proposal where things were less certain. Now updated.

I just want to be clear that if the actual way that room versions work ends up being different than outlined in the mechanics section of this document, a new spec proposal will be made before a spec PR is merged, right?

It's likely that, as I implement this, I'll discover things that don't work out as planned. If that happens, I'll update the doc and highlight the changes here.

Room versions are supposed to mark changes in mechanics for both clients and servers, right? But the way the proposal is worded, it seems to say that servers get to set the default version of rooms.

Correct.

Room versions are supposed to mark changes in mechanics for both clients and servers, right?

Yes, it's likely that some changes that might be supported by a room version bump will require client-side changes to support (an example would be: changing the way bans work so that bans are stored in a separate m.room.ban event rather than an m.room.member event. In order to figure out who has been banned from a room, a client would have to look at a different sort of event).

Does this mean that outdated clients will no longer be able to create rooms that they can use on servers that set a default room version other than 1?

Will clients be able to override the default room version when creating rooms (this only partially helps with previous point since many clients may not be aware of room version at all)?

These are interesting questions which I suspect we haven't thought about hard enough.

I think that, for testing future room versions if nothing else, the ability to override the default version is going to be important.

I'm hesitant to have an "I only support v1" flag on createRoom (or the inverse: "I do support v2"), for two reasons:

• I think "rooms they can use" overstates the case here. For many changes, there will be no need for clients to be changed at all. For many others, 90% of the functionality of a room will continue to work. That would lead to a lot of rooms being created on old room versions when there is no need.
• Obviously, it's not just the client that creates a room that needs to worry about the version of a room. Any other client that belongs to a user who has joined a v2 room needs to behave sanely in the presence of v2 rooms.

An extreme solution to the latter would be to extend the format of /sync so that v2 rooms are listed separately to v1 rooms, so that v2-unaware clients don't see the v2 rooms at all. We'd also need support on the /join API and others so that v2-unaware clients don't end up joining a v1 room by mistake.

Right now, I'm minded to hold fire on this, and make it more of a case-by-case judgement based on exactly what changes are happening: some changes will require significant changes to the C-S API, and others will be invisible to clients, and we should pick a solution accordingly. I'm certainly open to discussion here, though.

Will the spec continue to include sections about old room mechanics even as version 2 gets rolled out? It seems that it should since homeservers at least must implement both version 1 and version 2 room mechanics. Clients for a while probably ought to as well.

Absolutely. I think @turt2live is working on some refactoring of the spec to make this a more tractable thing to do.

In theory, could the version of a room affect how a client interacts with it before they've joined the room? ...

In theory, yes. In practice: it's just a particular case of the more general questions about client interaction above.

[richvdh]

@KitsuneRal: clients can observe the room version by looking at the m.room.create event, should they so wish.

[richvdh]

All: thanks for feedback so far. Having it in here where there is a record of it is useful, but I've also created a matrix room (#msc-1425:sw1v.org) if anyone has any /quick/ questions.

[KitsuneRal]

Will clients have to explicitly request m.room.create to find out how they should read events for that room already coming through sync to them? Doesn't sound client-friendly.

[turt2live]

Will the spec continue to include sections about old room mechanics even as version 2 gets rolled out? It seems that it should since homeservers at least must implement both version 1 and version 2 room mechanics. Clients for a while probably ought to as well.

Absolutely. I think @turt2live is working on some refactoring of the spec to make this a more tractable thing to do.

I'm not sure what the requirements are for this yet, although it is on the list (under r0 and above making it pretty).

[richvdh]

In discussion in #msc-1425:sw1v.org, I became convinced of the merits of making room_version a string. I've updated the doc accordingly, and added a section discussing the allowable grammar. Feedback welcome.

[I've also been thinking a bit more about client-side support. I'll update the doc accordingly in due course].

[richvdh]

I've now added some thoughts on client interaction at https://docs.google.com/document/d/1urKgReoHqxX8R_XtySB17dPi-DZcKhqTEL2_s895Wz0#heading=h.b398exru3iu5. It needs more discussion; feedback once again welcome.

[KitsuneRal]

I don't see much harm in trying to join an incompatible room - after all, I might still be able to read it on another client. On the other hand, filtering the response from /sync by supported versions looks useful for some cases. Adding the room version to list_public_rooms.yaml would certainly be nice but not a showstopper. Assuming these things are in place, the proposal looks good to me.

[richvdh]

Basic support for this has now landed in synapse, in https://github.com/matrix-org/synapse/pull/3654.

I think it was largely as originally specced, except for some minor tweaks:

• room_version is an opaque string, as per https://github.com/matrix-org/matrix-doc/issues/1425#issuecomment-407490149.
• /make_join now takes a series of separate ?ver=1&ver=2&ver=3 parameters, rather than a single ?room_versions=1,2,3 parameter. This is more consistent with what has been done elsewhere in Matrix.
• /make_join does not return the room_version (instead the joining server has to await the arrival of the m.room.create event in the send_join response)

The proposal doc has been updated accordingly.

[KitsuneRal]

/me still hopes for the room version in /sync response (maybe as a part of room summaries?)

[richvdh]

@KitsuneRal: why don't you just get the create event, if you want to know the room version?

(I certainly don't think the summary is the right place)

[KitsuneRal]

Well, basically it's yet another call for me, for EACH room in the roomlist - at best, I have to do something like that:

1. Do the first sync.
2. Collect the list of room ids from it.
3. Retrieve m.room.create for all rooms (by one more "service" /sync?).
4. Match up the response from this request with rooms from the first one.
5. Finally figure, which rooms I can read.

That looks a bit too involved for the initial sync when I really want to show things to the user as soon as possible. And there's never been a case so far when I needed to issue a network request right from processing the response from another network request. Not that libQMatrixClient couldn't do that, but I don't want to mess with special cases due to race conditions between the sync in step 1 and the sync in step 3 (what if the user left some room right before me sending a request for m.room.create events, e.g.?).

[richvdh]

1. Do the first sync.
2. Collect the list of room ids from it.
3. Retrieve m.room.create for all rooms (by one more "service" /sync?).

I don't quite understand why you wouldn't get the m.room.create in the first sync?

[KitsuneRal]

Because it would be beyond the latest N events I'm getting in the first sync?

[ara4n]

you always receive all state events in the first sync (ignoring lazy-loading members). the limit is only applied to the events in the timeline.

[KitsuneRal]

I suspected that I didn't notice an elephant in the room :) thanks, and sorry for the time taken.

[turt2live]

Merged 🎉

[richvdh]

the spec pr was #1516