Matrix-appservice-irc: Duplicate messages matrix-side (due to distributed IRCds)

[matrixbot]

Created by @ matthew:matrix.org.

e.g. roblabla on https://vector.im/beta/#/room/!cURbafjkfsMDVwdRDQ:matrix.org/$146192559593207XoTMY:matrix.org and half-shot a few days earlier

[kegsay]

The permalink says the commit sha: c3f8dbf6b5b5f33bebae09c2e92866e140e4601b

[kegsay]

Probably related: https://matrix.org/jira/browse/BOTS-142

[kegsay]

This SHA is for Synapse, not the IRC bridge and is therefore useless.

[kegsay]

Tried to repro this on v0.3.1 and I cannot. Used the demo federation with user IDs:

@fewd:localhost:8481
@fewd:localhost:8482

I surmise the title of this bug is wrong, or that particular issue has already been fixed. BOTS-142 still seems valid though, so clobbering this issue with that.

[kegsay]

This is the important bit:

2015-11-16 17:28:39 INFO:irc-client <[email protected]#bion5hs2eo848> (@intelfx:matrix.org) Connecting to IRC server irc.freenode.net as M-intelfx (user=intelfxmat)
2015-11-16 17:28:41 DEBUG:client-pool Connected with nick 'M-intelfx1' instead of desired nick 'M-intelfx'
2015-11-22 03:43:38 ERROR:client-connection [email protected]: {"command":"ERROR","rawCommand":"ERROR","commandType":"normal","args":["Closing Link: gateway/shell/matrix.org/x-ufiwesbeopqhxsaf (Ping timeout: 244 seconds)"]}
2015-11-22 03:43:38 INFO:client-connection disconnect()ing [email protected] - raw_error
2015-11-22 03:43:38 DEBUG:client-pool onClientDisconnected: <bion5hs2eo848> Reconnecting [email protected] in 10000ms
2015-11-22 03:43:48 INFO:irc-client <[email protected]#kakx3kvehj44c> (@intelfx:matrix.org) Connecting to IRC server irc.freenode.net as M-intelfx (user=intelfxmat)
2015-11-22 03:45:46 ERROR:client-connection [email protected] still not connected after 30000ms. Killing connection.
2015-11-22 03:45:46 INFO:client-connection disconnect()ing [email protected] - timeout
2015-11-22 03:46:24 DEBUG:irc-client <[email protected]#kakx3kvehj44c> (@intelfx:matrix.org) connected!
2015-11-22 03:46:24 DEBUG:irc-client <[email protected]#kakx3kvehj44c> (@intelfx:matrix.org) _keepAlive; Restarting 432000s idle timeout
2015-11-22 03:46:24 INFO:client-pool <kakx3kvehj44c> Reconnected [email protected]
2015-11-22 03:55:27 INFO:irc-client <[email protected]#kakx3kvehj44c> (@intelfx:matrix.org) NICK: Nick changed from 'M-intelfx' to '20WACI3YG'.

Netsplit and the single nick => Client mapping is wrong.

[kegsay]

This relates heavily to https://github.com/matrix-org/matrix-appservice-irc/issues/72 - We might as well do _that_ rather than faff with Nick => [Client] mappings (instead it'll be Username => Client which the bridge can ensure are unique :smile: )

[kegsay]

Punting this to the next release as #72 is effectively a blocker and I don't feel like doing that right now.

[kegsay]

This reared its ugly head again today, though the cause isn't a netsplit. If you connect with the same nick to 2 freenode servers at the same time, the IRCds don't handshake first before accepting the connection, so there's a brief period where both servers accept the connection for the same nick. This then gets corrected less than a second later, but by this point it's too late because it has clobbered the entry in the ClientPool. The entire sequence is:

// connect to one server, which accepts the proposed nick
clientPool["Alice"] = clientAlice
// connect at the same time to a different server, which also accepts the same nick. This clobbers `clientAlice` irreparably. 
clientPool["Alice"] = clientAlice2
// bridge gets rename
clientPool["425FE25"] = clientPool["Alice"]
clientPool["Alice"] = null;

But we keep another map of user_id -> client as well, so M->I is not affected by this. This means messages to IRC go out correctly, but means we don't recognise the nick when it comes back to us, so we send it on to Matrix again.

[kegsay]

Changed title because this is not netsplit specific anymore.

[copumpkin]

@Kegsay I'm seeing some duplicate messages on Matrix's IRC bridge to freenode in the ##nix-darwin channel. Not sure if that's a different cause or related to this one. I think the whole thing started with a netsplit a couple of weeks ago and hasn't fixed itself since.

[simoniz0r]

I just managed to do this yesterday by using NickServ from freenode to IDENTIFY and then REGAIN my nick.

[Perflyst]

Seems like I currently have this bug in any room which is bridged to Freenode.

[Half-Shot]

For reference, we are tracking this bug at the moment. The likely short term fix next week will be a bridge restart coupled with some related bug fixes. I suspect the very least we can do is some kind of debugAPI to refresh broken rooms.

In the shorter term, you are free to ban *[m] users from your rooms as it won't interrupt the bridge.

[Mikaela]

In the shorter term, you are free to ban *[m] users from your rooms as it won't interrupt the bridge.

I think this comment is easy to interpret as mode +b *[m]!*@* which bans all Matrix users using the default nick from the channels. I guess you meant that Matrix-side moderators should op their IRC ghosts and then ban the *[m] users on Matrix-side.

[Half-Shot]

In the shorter term, you are free to ban *[m] users from your rooms as it won't interrupt the bridge.

I think this comment is easy to interpret as mode +b *[m]!*@* which bans all Matrix users using the default nick from the channels. I guess you meant that Matrix-side moderators should op their IRC ghosts and then ban the *[m] users _on Matrix-side_.

Yes, I should be clear here. Please do NOT ban the matrix users from the IRC side -- that will break things. Ban @freenode_.*[m]:matrix.org users if they start echoing as it won't affect the bridge.

Although hopefully for the time being this is a moot point, as hopefully the restart just now fixed stuff.

[Mikaela]

I have reproduced this on Ergo IRCd which doesn't even support server linking.

1. join #libcasa:liberta.casa
2. say something to get invited to bridge admin room
3. !storepass Mikaela-@libertacasamatrix:password
4. say something
5. see how everything I say is duplicated.

Reconnecting to network...
Connecting to the IRC network 'irc.liberta.casa' as mikaela{m}...
Your connection to the IRC network 'irc.liberta.casa' has been lost. 
SASL authentication successful: You are now logged in as Mikaela-
You've been connected to the IRC network 'irc.liberta.casa' as mikaela{m}.
<me> !nick Mikaela-
The nickname Mikaela- is taken on irc.liberta.casa. Please pick a different nick.

• BridgeVersion: 0.27.0

On Matrix I am @mikaela:liberta.casa and my ghost who gets repeated is @irc_Mikaela-:liberta.casa. On IRC only messages from Mikaela- are visible.

I think this is matrix-appservice-irc not handling the integrated bouncer of Ergo which is enforcing my nickname into being my accountname (Mikaela-).

See also: https://github.com/ergochat/ergo/blob/v2.7.0/docs/USERGUIDE.md#how-ergo-is-different

In case you try to reproduce this on liberta.casa, https://github.com/matrix-org/matrix-appservice-irc/issues/371 or something similar is hitting it and you cannot contact @lunatic:liberta.casa unless you are on the same homeserver.

Network IRCv3 capabilities in case they are of any help: account-notify account-tag away-notify batch cap-notify chghost draft/channel-rename draft/chathistory draft/event-playback draft/languages=1,en draft/multiline=max-bytes=4096,max-lines=100 draft/register=before-connect draft/relaymsg=/ echo-message extended-join invite-notify labeled-response message-tags multi-prefix oragono.io/nope sasl=PLAIN,EXTERNAL server-time setname userhost-in-names znc.in/playback znc.in/self-message

[Mikaela]

I don't know whether this is a new issue or https://github.com/matrix-org/matrix-appservice-irc/issues/502#issuecomment-877644665, but additionally when I join the 13 missing rooms by hand, in the end I am in 14 rooms as one has duplicated and I cannot see anything sent from IRC while IRC can read me in some cases.

Possibly yet another issue is that some of the rooms have +i or +k where I am allowed to join right in due to having CS AMODE +h and being already in so not needing to enter key, but the matrixbot (lunatic) doesn't and thus cannot.

[Mikaela]

Update on reproducing on Ergo (https://github.com/matrix-org/matrix-appservice-irc/issues/27#issuecomment-877643707): I can !reconnect as much as I want and consistently reproduce this issue on every attempt.

Reconnecting to network...
Connecting to the IRC network 'irc.liberta.casa' as mikaela{m}...
Your connection to the IRC network 'irc.liberta.casa' has been lost. 
SASL authentication successful: You are now logged in as Mikaela-
You've been connected to the IRC network 'irc.liberta.casa' as mikaela{m}.

Maybe Ergo is a good IRCd to test against for fixing this bug in general if others don't allow reproducing this consistently?

[Mikaela]

@Half-Shot I am sorry to ping you on multiple issues, but this particular issue starting from https://github.com/matrix-org/matrix-appservice-irc/issues/27#issuecomment-877643707 (alongside https://github.com/matrix-org/matrix-appservice-irc/issues/1421 which I could bet our users will hit) is a major reason why Liberta.Casa (who would also be in need of Synapse assistance) and PirateIRC have suggested removing Matrix bridging entirely as it seems that the upstream doesn't even acknowledge our issue after 9 days.

We also have confusion over https://github.com/matrix-org/matrix-appservice-irc/issues/1411 while it hasn't yet been major to anyone actively using the bridges (I think majority of Liberta.Casa users are in a channel without RELAYMSG and PirateIRC is yet to move to RELAYMSG capable IRCd which has to happen sooner or later due to 9 months EOL IRCd).

[slingamn]

The diagnosis on https://github.com/matrix-org/node-irc/issues/74 looks correct to me. The issue is this: in general, an IRC server can assign a client a nickname that differs from the one the client requested with NICK. The client is expected to deduce this from the first parameter of the 001 RPL_WELCOME numeric.

To simplify reproducing the issue, I created an account on testnet.ergo.chat, account name nickchangetest, password _UqRdsxHQmbxWbUwq11APQ. Here's a sample transcript:

C: CAP REQ sasl
S: :testnet.ergo.chat CAP * ACK sasl
C: AUTHENTICATE PLAIN
S: AUTHENTICATE +
C: AUTHENTICATE bmlja2NoYW5nZXRlc3QAbmlja2NoYW5nZXRlc3QAX1VxUmRzeEhRbWJ4V2JVd3ExMUFQUQ==
S: :testnet.ergo.chat 900 * * nickchangetest :You are now logged in as nickchangetest
S: :testnet.ergo.chat 903 * :Authentication successful
C: NICK nickchangetest_
C: USER u s e r
C: CAP END
S: :testnet.ergo.chat 001 nickchangetest :Welcome to the ErgoTestnet IRC Network nickchangetest
S: :testnet.ergo.chat 002 nickchangetest :Your host is testnet.ergo.chat, running version ergo-2.7.0
S: :testnet.ergo.chat 003 nickchangetest :This server was created Tue, 08 Jun 2021 05:08:05 UTC

[Mikaela]

This issue is fixed in 0.29.0-rc2, thank you.

It does introduce a new issue I am opening soon.