Element-web: Let devices request keys to decrypt past history for a room.

[ara4n]

If I'm in a room as device A, and join it later as device B, i need a way to request the old room key data from device A.

Similarly if I'm invited into a room by someone and they want to share past history with me, they need a way of transferring key data.

[apexo]

Could a possible solution to this be:

• create a special (maybe hidden per default) per-user channel on the user's home server
• each new device sends an (encrypted) poll message to all other devices (via this room)
• all other devices reply with the encrypted e2e state (sans identity keypair), or just all keys required for decrypting room history

[ara4n]

@apexo this is sort of the plan, except we now have a dedicated "to device" store-and-forward API for sharing data like this, which is encrypted in turn by olm. Also, we'd probably request the historical ratchet state from a specific device rather than all of them, and require the target user to approve/deny the request. e.g. "Matthew has added a new device 'iPad' and is requesting room history since Sept 18th. Do you want to share history?" or something.

[0kto]

@ara4n So I am currently in a room listed with seven devices from three machines and as many browsers. well, I guess vector --> riot transition helped to increase the number. But with only one other person in the room, this is cumbersome.
I would prefer if a client-side certificate could be recognized as a user, and verification becomes an automated process that does not rely on devices or interaction of members in the room and I just see the history. Would work on private devices (i.e. not the library computer).
To understand the process, how are the IDs generated, and how long are they kept / stored? Do they change with version change of kernel, browser or if I purge cookies? I am pretty sure I am missing discussions on the same device I logged in at the time...

PS: I haven't seen another issue / discussion on this matter, so I hope this is the right place to put this :)

[madduck]

Maybe look at Bitcoin wallets, e.g. http://wallet.counterwallet.io, which ask a 12 dictionary word passphrase from the user. That passphrase is the private key.

Now when I log in to a device, I could be asked if I wanted to create a device-specific key, or if I wanted to instate the user's key from such a passphrase.

[csmart]

Can I check my understanding please, does the following represents current expected behaviour?

Say I'm in a room which has encryption turned on and we're all happily chatting away and everyone has verified keys. Then I sign out of riot and back in to see that all previous conversations are encrypted again and that there's no way to get the original key to you, so history remains unavailable?

Thanks!

[madduck]

It doesn't matter whether keys are verified. The point is that the encryption algorithm (megolm/ld double ratchet) encrypts to the current keys of all the people present.

Now if you log out of Riot, your key is discarded from local storage, and upon rejoining, a new key is made. Future messages will now be encrypted (also) to that key, but since you no longer have access to the previous key, the old messages aren't accessible any longer.

The solution is IMHO to allow one to export/safe/backup one's key and restore it upon rejoin.

[csmart]

Right, thanks @madduck

[kebertx]

I've been having major usability problems with end to end encryption in riot, and I didn't realize that it was because logging out deletes your session key! riot-android/#869 would be nice, because I had no idea!

Now that I understand why I've been losing my keys -- and being locked out of my chat history -- I have to say I'm really annoyed. I'm sure everyone agrees you shouldn't have to stay logged in to your chat program continuously in order to be able to use it reliably. It's one thing to hide past messages from newcomers to a channel, but this same issue means that I can't share decryption keys with myself! In my opinion, E2E should not have been rolled out as a feature before this problem was solved. It's been a trainwreck for everyone I know who's tried turning it on.

[csmart]

@kebertx Aside from this issue (which I now understand) e2e has been awesome for me and those I've been chatting with.

IIRC, it was advertised as beta and at least in the Android app it says it's that e2e is an "experimental labs feature which may break in unexpected ways." So I think the warning has been fair enough.

Having said that, I do think things can be improved (a simple warning about losing keys when signing out, for example would be handy for new users).

IMO, it's been great to have the feature early and be able to test it.

[madduck]

Hey @kebertx, sorry you've had a bad experience. It is true that e2e at the moment is highly beta, and things certainly need to be improved, which is, in part, why this issue exists.

FWIW, I see no need to ever log out. I do have riot.im exempted from my cookie munger, and so I don't have this issue. Similarly, I don't log out the Android app, but the problem will come when I need to switch to a new phone, and by then I hope they've got a fix.

[ara4n]

(fwiw the warning was added)

[irunonayran]

I don't see a need to log out either, if the app supports multiple identities, which riot seems to not do, so logging in and out would be a workaround for me. Alternatively I could use a second android user, but that drains my battery more.

[pik]

@madduck You could also go the other-way an export an existing private key as a mnemonic - e.g. BIP39 uses a 2048 word list (https://github.com/bitcoin/bips/blob/master/bip-0039/english.txt) which is sufficient to store 128-bit keys (2048^12). Although one issue is that ed25519 and curve25519 keys are I believe independently derived in Olm? So you would need mnemonics for both, or support deriving one from the other.

@ara4n

Similarly if I'm invited into a room by someone and they want to share past history with me, they need a way of transferring key data.

I'm not sure I understand the original issue very well, can't an existing member just key-share all of the session_keys they have received from other members in that room via. the same sendToDevice api using an Olm Session as is normally done when a client shares their outbound session key?

[ara4n]

having read through prior comments - most of them don't relate to this bug, but are general UISI concerns (#2996) or key management concerns (#3611); please upvote and/or comment on the right bugs rather than using this one as a dumping ground O:-)

As per https://github.com/vector-im/riot-web/issues/2996#issuecomment-299290453 we're moving on and implementing this asap, both to provide an escape hatch for cleaning up remaining UISIs as well as to provide the feature of syncing history to new devices without having to manually export/re-import megolm session keys. @richvdh, @NegativeMjark and I just went through the discussion of how this should work, resulting in the plan:

• If a client receives a message which it doesn't have megolm session keys for (i.e. a UISI), it should send to_device messages to all devices (i.e. "*") belonging to themselves and the sender, requesting the missing key.
• By default senders will automatically re-share megolm keys to devices they have verified (if they sent the message to that device, or it's their own device).
• If the device isn't verified, they have the option to verify it before sharing, or to ignore the request (and/or blacklist the device?)
• We'll also provide a setting called "Do not automatically share E2E keys with new devices" in settings, so that paranoid users can explicitly authorize this.

Separately, we will obviously need to improve the verification UX, e.g. by cross-signing devices to automatically trust them, QR codes or mnemonics for verifying fingerprints, etc. This is all #2142 however and orthogonal to the problem of actually sharing keys.

We also discussed whether we should store megolm keys on the server, to try to both improve clientside key storage capacity problems (#3660) or to provide an easier way of backing up keys and/or sharing them - this is #3661. We concluded that this is basically an optimisation for key storage and wouldn't change the semantics of the crypto at all (i.e. if you're running out of clientside storage, you're welcome to use an encrypted cloud storage service to store data, but that's not going to change our E2E behaviour).

Finally, we considered the edge case that Alice and Bob are both pretty paranoid, and only ever use Matrix by firing up an incognito window, logging in, sending a message, and closing the tab (not implausible). As a result, at any given point they probably wouldn't have any devices live. Therefore when Alice sends a message to Bob, she won't know how to encrypt for him, and when Bob gets a UISI, there won't be any devices of Alice he can query for the megolm keys (unless Alice happens to be logged in and having re-imported her megolm keys at the moment that Bob requests history, or perhaps later). We concluded that there were two possible solutions to this:

1. Introduce the concept of 'virtual devices' so that Alice & Bob can maintain a virtual device which is stored encrypted on the server, and is passed between physical logins, letting them pick up conversations when desired - this would almost act as an 'identity key'.
2. Introduce the concept of deliberately 'dehydrating' and 'rehydrating' devices, so that when Bob stops using his Incognito window, he could explicitly export that device intact onto a USB stick or whatever, and subsequently rehydrate it next time he opens an incognito tab - effectively deliberately backing up & unbacking up the device.

In the end, both of these are tantamount to the same idea, whether the device is stored encrypted on the server or on a USB stick. Both suffer from #3822 - that if two identical devices ever exist concurrently Olm will completely wedge and E2E will break. So the conclusion there was to educate users that they need to keep at least one device active if they expect E2E to work, unless we ever get as far as dehydrating/rehydrating devices.

@pik: hopefully this resolves your confusion a bit :)

[uhoreg]

Will there also be a button somewhere to request _all_ the keys from one of your own devices, in order to aid migration from one device to another? Without this, it sounds like when people want to replace one device with another, people would either need to manually export/import the keys (a bit less convenient than just clicking a button to request keys), or else go to each encrypted room and backfill to the beginning so that the device automatically requests keys (a lot less convenient).

[ara4n]

@uhoreg possibly. i'm a bit scared at how powerful such a button would be; at least an attacker needs to know the IDs of the megolm sessions they're trying to get keys for in the "send_missing_keys" to_device event outlined above. Whereas an "send_all_keys_ever" feels like a terrible thing to fall into the wrong hands...

[ara4n]

i've split out the 'device hydration/dehydration' tangent into #3825

[ara4n]

Thought: we have a UX problem in general in that the "send_missing_keys" event will only be actioned by the sender device if it happens to be online. So we may end up staring at the UISI for quite a while until one of the devices which has the key picks up the request and actions it. Perhaps we can mitigate this in the UI by renaming the tile "requesting keys..." when this scenario is happening (rather than UISI)?

[pik]

In the end, both of these are tantamount to the same idea, whether the device is stored encrypted on the server or on a USB stick. Both suffer from #3822 - that if two identical devices ever exist concurrently Olm will completely wedge and E2E will break. So the conclusion there was to educate users that they need to keep at least one device active if they expect E2E to work, unless we ever get as far as dehydrating/rehydrating devices.

So in this case Bob's keys stay the same but the device_id is subject to change (if I understand correctly). Currently it's possible to restore a deviceId on login (but typing a device ID doesn't seem like a user-friendly UI feature). Perhaps not using to_device but having a to_curve25519_key api would work but that probably raises new issues as well?

Alternatively what about generating the device_id deterministically from the curve_25519/ed_25519 public keys? Basically if your keys change, your device_id changes - if you import your keys, it's considered to be the same device.

[ara4n]

no, the idea of virtual or rehydrated devices is that they would keep the same device ID. agreed that generating device ID from its pubkey would be reasonable though. please let's use the split-out bug #3825 for further discussion on the rehydration idea. (god, i wish GH had threading)

[uhoreg]

@ara4n in the third bullet point of https://github.com/vector-im/riot-web/issues/2286#issuecomment-299532155, should it say

(if they sent the message to that user, or it's their own device).

instead of

(if they sent the message to that device, or it's their own device).

Or will it really only send to devices that should have already received the key?

[ara4n]

yes, this was a thinko. the idea is that the sender's device(s) will resend keys to any of the target user's devices if they have verified that device.

[richvdh]

https://github.com/matrix-org/matrix-js-sdk/pull/454 etc provided a partial fix for this bug; we now request keys from our own devices.

There remains work to be done here, though:

• make the management of key requests more reliable (once you close the dialog, there's no way to change your mind)
• better UX for handling the key requests in general (#4522)
• ask the original sending device for the keys too (since that is the only device which knows for certain if you were in the room at the time)

[richvdh]

the design doc for this was https://docs.google.com/document/d/1USqQVhEEiuBQF9MmECHEyN6zy6BolTJSdTfytbTT_sY/edit, ftr

[ara4n]

This is all done and live on all SDKs now, other than the ability to request keys from the original sending device.

[lhunath]

My apologies; this message seemed a little confusing to me, if the ability to request keys from the original sending device is not yet done and live, then what precisely is, and what's still missing? Also, many of these referenced issues are still Open; are they included in the "all"?

[ara4n]

the referenced issues which are open are still open.
the ability to request keys between a user's own devices is implemented and live on matrix-{js,ios,android}-sdk.
the ability to request them from the original sending device is missing, hence not yet closing this bug.

[lhunath]

Thanks! That really helps form a clear picture.

[ara4n]

Just discussed this with @richvdh in this morning's weekly sync, and conclusion was that we probably do want to implement the "please re-request keys for for this message" button, to mitigate scenarios where the keyshare request or response got lost. Scenarios include:

• Receiving device crashes or errors before sending the response
• Client's e2e store gets corrupted
• ...and basically any other UISI root cause.

We're very much agreed that it would be preferable to fix the root causes rather than paper over them like this - but given a choice between having and not having a last ditch rescue button, i think this is the least worse option right now.

[edef1c]

In my experience, the request seems to get lost basically every single time so far, which is a little unfortunate. Signing on a new device is the only way I have of recovering when the crypto randomly decides to break, and exporting/re-importing E2E keys manually involves too many error-prone steps to be something I want to repeatedly bet my opsec on.

[intelfx]

@ara4n The re-request button in Riot-web looks like it does not work for _your_ old messages.

the ability to request them from the original sending device is missing

still missing?

[ara4n]

I've written another proposal to finish this off (i.e. to request keys from the sending device, as well as the special case of DMs): https://docs.google.com/document/d/1_wDoDQ02JLZwYeVb-QVPylnJUXJVTZ9bNoGHd_Go6Zg

[ara4n]

There was a lot of discussion about this on #e2e-dev:matrix.org the other day (around https://matrix.to/#/!uewiilduiDRfPomIha:matrix.org/$1530880581166IkkUg:sw1v.org), where we realised that it's okay for the recipient to send key share requests to arbitrary room members (modulo niceness of UX), assuming that clients keep track of the membership of a room at the point that they first see a given megolm session. Because we rotate megolm sessions whenever anyone leaves, then any well-behaved client in the room can decide whether the requester should be allowed to see the message or not.

We probably don't ever want to pop up a KS req dialog box for those clients for unverified devices, but for verified devices perhaps we could just silently honour the request as a better way to recover from UISIs?

[richvdh]

assuming that clients keep track of the membership of a room at the point that they first see a given megolm session

this is a massive assumption, though. They don't, and it would be a large amount of data, and would fly in the face of the idea that clients don't need to store complete room history.

[ara4n]

(the variant of this where we share keys for a specific room OOB is https://github.com/vector-im/riot-web/issues/6454)

[thomasylee]

Is this still being looked into?

I recently set up my own Synapse server and discovered that restarting it sometimes causes some sessions to not be decryptable by some devices. There's also a pretty long list of other GitHub issues related to messages not decrypting when they should be decryptable.

It seems like enabling this feature could single-handedly make a bunch of encryption woes become much more palatable. I've added a $100 bounty to this issue to encourage its resolution. It expires Nov 20, but if it's still actively being worked on I can extend it.

Bounty details: https://www.bountysource.com/issues/38093648-let-devices-request-keys-to-decrypt-past-history-for-a-room

[uhoreg]

There have been some recent fixes in Synapse to solve some decryption issues. As well, @JorikSchellekens has added some code to help debug such issues.

[akontsevich]

People pointed that sometimes my messages are some kind inflammatory and drastic however in fact encryption in Riot still terrible, buggy and unusable. Reasons:

1. If I have few devices I need to Verify all of them. If other people in a room I need to verify all their devices and they should verify their devices and all my devices which is boring, wastes much time and very strange. There is no other messengers which use such complex and addled algorithms.
2. If I need encrypted room with tens or hundreds of people - Riot become totally unusable here cause negotiation number exceeds all reasonable limits and convenience.
3. I have a room with 2 people. 1st one (holding encryption keys and backups on server) can send messages which both 1st and 2nd can decrypt and view, however messages from the 2nd one is not decryptable by the 1st one. And such situation appears VERY often on relogin, reload, message sending - unpredictable. Re-request encryption keys from other devices does not work at all.

Think need to fix and radically change encryption idea to make it convenient and transparent to people: #9633.

[BrenBarn]

I just read through this whole thread and I still don't understand exactly what the consequences of E2E are or are supposed to be for users.

If it is really the case that simply logging out and logging back in again can cause you to lose access to messages you previously had, that seems absurd. Logging in and out on the same device shouldn't impact access to messages in any way. Also, the current messaging from the app on this is not at all clear. It has some message about "key backup" and then the alternative is "I don't want my encrypted messages". If it is the case that logging out loses access to messages, the app needs to say that explicitly, something like "WARNING: If you log out you will lose access to encrypted messages!"

Also it is suggested in #8751 that exporting keys and then importing them later won't give access to old messages. If so, then what is the purpose of exporting the keys? There needs to be much clearer documentation that tells users "If you do this, you will lose access to such-and-such messages". Right now I'm somewhat reluctant to log out on any device because nowhere can I find an explicit statement of how to determine what the consequences of that would be.

[aaronraimist]

8751 says they aren’t like GPG keys in that they don’t give you access to future messages, not that they don’t give you access to old messages (that’s the only thing they do). Will edit the issue assuming that explanation is clearer.

[gitvalds]

I have the same problem, on the latest Riot version 0.9.8 (F-b98798510).
Today I uninstalled it from F-droid, after discovering that Riot from F-droid does not show any new message unless I open it.
I installed it from the Google Play Store, which uses Google services to push the messages.

I could see the chat people and their messages but they were all not readable, and showing the message::
"* Unable to decrypt: The sender's device has not sent us the keys for this message. *
Re-request encryption keys from your other devices."

When I touched "Re-request encryption ..." on my own last message, it showed the message:
"Request sent
Please launch Riot on another device that can decrypt the message so it can send the keys to this device."

I think you see the irony of that message :)

Anyway I uninstalled Riot from Google Play Store.
Installed Riot from F-droid.
All messages were not readable.
After using my "Recovery key for the encryption key", I could see the first few messages encrypted and visible to me.
But the latest ones were encrypted and not readable by me, showing the message::
"** Unable to decrypt: The sender's device has not sent ..."

Even my own last message was not readable by me.
On the same phone, with the same Riot version from F-droid, with the same "backup key".
I touched again "Re-request encryption keys ..." but the result was the same.
It again showed "Key request sent" under the message, but I don't know to whom it sent it.
Not to me, although it was my message from this same phone.

I want to be constructive, so please let me know if I can help.
Thanks for your work :)

[gitvalds]

I uninstalled Riot from F-droid and installed Riot from the Google Play Store.
The result is the same.
The backup key never worked, on both Riot from F-droid and from Google.
I tried both passwords, also because I don't understand which is for what.
Instead, the "Recovery passphrase" gave the result said above.
Notice that the "Recovery passphrase" is called "Recovery Key for the encryption key" when installing Riot, and "Recovery passphrase" after having it installed.

[gitvalds]

I sent a message to the only person in the chat and I got the message (text not exact because it is from my memory):
"Could not send message because the chat contains an unverified device."

So I touched "devices" and I saw a list of nine devices, all unverified except the first one.
Their first line are as follows:
Mobile (this is the only one with a green lock)
https://riot.im/app via Safari on Linux
Mobile
Mobile
https://riot.im/app/ via Firefox on Linux
https://riot.im/app/ via Firefox on Linux
Mobile
(empty line)
Mobile

I might have used Riot once via Firefox, but certainly not with Safari and not with other phones.

[akontsevich]

Yes, situation still not fixed. One of my users can't read some messages in a room - see newest, can't decrypt older and I can't send him encryption keys automatically. His other device keys request did this some kind partially. Very buggy, user unfriendly and uncomfortable feature.

[andrewperry]

The latest app updates seem to have surfaced a bunch of these messages over recent days:

I like the security features that seem to have been made the default with the release of Element, except for the fact that they create usability issues that may make it very difficult to get people to adopt it and/or stick with it!!

[t3chguy]

That doesn't look like element-web @andrewperry

[andrewperry]

I had thought the message in this screenshot was sent from Mac desktop which I understand wraps element-web, but I freely admit I was focused more on the topic of this thread rather than the fact it is on the element-web project. Inspecting the source of the message I see it now came from the iOS app, my apologies for reporting in the wrong project.

[jeena]

@andrewperry just wanted to mention a workaround until it's fixed. I logged out of the app and in again which triggered the request for encryption keys and finally I could read the messages again.

[tim-seoss]

Another workaround (requires the user in possession of the keys to have significant relevant command line skills, but otherwise reasonably straightforward) is to use this:

https://github.com/cyphar/matrix-utils