Electrum: A better coordination with electrs?

[Kixunil]

I'm a contributor to electrs and I wonder if we could improve coordination so that accidents like #6772 don't happen. From experience I know that being unable to access money is pretty annoying. As a full node experience provider I would also like to have more confidence in Electrum.

I'm not sure what the best approach is but one idea that comes to my mind is that Electrum could make the release process a bit more formal. You could announce "protocol freeze" phase and give some time (two weeks?) for server developers to respond. During this period no protocol changes are accepted. If a protocol change must be implemented (due to a vulnerability or other critical problem) then the whole period would be restarted.

Is this approach useful/acceptable? Please let me know if I'm missing something.

The above idea is based on my understanding of the situation: nobody noticed that a new version of Electrum was released and so it wasn't tested in time. It should be possible to release a new version of electrs quickly but the time required is still not zero. Please correct me if I'm wrong.

Cc @romanz

[SomberNight]

You could announce "protocol freeze" phase and give some time (two weeks?) for server developers to respond. During this period no protocol changes are accepted. If a protocol change must be implemented (due to a vulnerability or other critical problem) then the whole period would be restarted.

In this case there was no protocol change. The client simply started enforcing a rule that was already part of the protocol specification, and other parts of the client code had already depended on it, it was just not explicitly checked before.
There were other similar cases in the past, e.g. https://github.com/spesmilo/electrum/issues/6289 and https://github.com/romanz/electrs/issues/314

I would like to think I try to notify at least @chris-belcher and @romanz when I know we are making a change that will affect their projects. Obviously also kyuupichan in the past.
I can promise we will continue to do that; but only if we know a change might affect them.

I did not know electrs did not conform to this rule - it presumably was due to a bug anyway if so (due to the nature of the change) - otherwise I would have pinged romanz.
I had been adding more checks client-side to server responses e.g. in https://github.com/spesmilo/electrum/commit/c5da22a9dd4459972a85d116fe281b8564c20834, this current change was related to that.

Regarding protocol changes, AFAICR we have never done them in point releases (so for version x.y.z, only if x or y changes).
Further, the protocol is versioned and negotiated, so changes mean a new version number there too. (but do note that historically we have been off-loading the complexity to the server: the client only ever supported a single protocol version, while the server usually supports a range)

I don't think we can promise a two week freeze before patch releases, frankly we often do them on a whim when we think master is stable and we have time, or when there is some criticial issue.
Also, as this case highlights, we might break compatibility unintentionally, so we would not know in general when we need to ping you. Still, usually we know at least a few days in advance that we would make a new a release, so if you think that would help we could ping you then, before each release. Recently releases have not been too frequent so maybe it's doable for you to test each, I don't know. Then again, I often connect to random public servers in hope of triggering some bug and I am sure I had connected to the blockstream one at least a few times since this client-change but I did not trigger the bug (but most of the time I connect to my own server that runs ElectrumX).

However, as above, most of the time we only do changes that we know might affect you in major/minor releases. We usually know weeks in advance we would release one soon. So in those cases, we could ping you more than a few days beforehand.
As an example, I pinged both belcher and romanz a few weeks ago about changes I am planning to introduce with protocol version 1.5.

Btw, it might help if you hanged around in IRC.

[Kixunil]

Ah, OK. I agree with adding checks for protocol rules. Maybe you could ping us (github mention should be enough) in those cases too? At least I don't mind being notified of that.

But maybe this is better solved with automated tests and CI. Does Electrum have some kind of tests that could detect protocol bugs? I could try to look at hooking them up with lnpbp-testkit. But even if there are no such tests, AFAIR Electrum has its own RPC, so creating a script that attempts to pay an address, check if Electrum sees the payment and then pay out could be sufficient.

[ecdsa]

@Kixunil see also #6778

[romanz]

First of all, I'd like to thank the Electrum wallet developers for their great project - it is a pleasure collaborating with you :)

In this case there was no protocol change. The client simply started enforcing a rule that was already part of the protocol specification, and other parts of the client code had already depended on it, it was just not explicitly checked before. There were other similar cases in the past, e.g. #6289 and romanz/electrs#314

Indeed - many thanks for reporting these issues!

However, as above, most of the time we only do changes that we know might affect you in major/minor releases. We usually know weeks in advance we would release one soon. So in those cases, we could ping you more than a few days beforehand.

Sounds great, much appreciated :)

Btw, it might help if you hanged around in IRC.

Joined as romanz.