Rocket.chat: Federation messages not working between two Rocketchat 3.4.2 instances

[col-panic]

Description:

I setup 2 rocketchat instances (A, B) to test federation. Each of the instances is configured via the DNS method.
I want to add users in B from A, and send direct messages from User@B to AnotherUser@A.

What works so far

I managed to work through the slightly wrong documentation (see #14728) in https://docs.rocket.chat/guides/administrator-guides/federation and now I can lookup the users on A from B and add them as external users on B

_HINT to other people trying to setup_

Raise the log level in _Administration / Logs / Log Level_ you can then see messages like ( this helped me to find the correct DNS settings)

rocketchat_1          | Federation ➔ dns.debug search: peerDomain=mydomain.ch
rocketchat_1          | Federation ➔ dns.debug search: peerDomain=mydomain.ch srv=_rocketchat._https.mydomain.ch
rocketchat_1          | Federation ➔ dns.debug search: peerDomain=mydomain.ch srv=_rocketchat._http.mydomain.ch
rocketchat_1          | Federation ➔ dns.debug search: peerDomain=mydomain.ch srv=_rocketchat._tcp.mydomain.ch
rocketchat_1          | Federation ➔ dns.debug search: could not find valid SRV entry peerDomain=mydomain.ch srvEntry=undefined protocol=
rocketchat_1          | Federation ➔ dns.debug searchHub: peerDomain=mydomain.ch
rocketchat_1          | Federation ➔ http.debug [GET] https://hub.rocket.chat/api/v1/peers?search=mydomain.ch
rocketchat_1          | server.js:204 Federation ➔ dns.error Error: failed [404] Not Found

What does not work

• The status of the remote user is not correctly shown, as documented in #17201
• Messages are seemingly sent in B but never appear in A, I can also not detect any incoming traffic
• The remote user info is incomplete and ?wrong? e.g. the avatar is not shown, and the mail-address is "obfuscated"? Mentioned in https://forums.rocket.chat/t/federation-and-user-info/6998

Steps to reproduce:

Perform a setup like documented above

Expected behavior:

Direct messages from User@B should arrive with AnotherUser@A

Actual behavior:

Nothing. AnotherUser@A does not see any incoming message, while User@B sees that the direct message was transmitted.

Server Setup Information:

• Version of Rocket.Chat Server: 3.4.2 on both sides
• Operating System: Linux
• Deployment Method: docker

[col-panic]

Related #17828

[col-panic]

Mentioned in https://forums.rocket.chat/t/federation-messages-sent-but-not-received/7991

[geekgonecrazy]

Just to be sure they are both using completely different domains from each other right? I’ve seen some cases where people had two different rocke.chat setups both with same domain.

[SEisenbach]

I can't speak for the Author, but I’m experiencing the same problem with version 3.5
i use two (sub)domains.
rc1.test.home -> IP - 192.168.0.233
rc2.test.home -> IP - 192.168.0.244

Should I try it with completely different domains?

[geekgonecrazy]

I don’t mean the domain rocket.chat is at. But the domain you have configured in the federation section. You need to make sure you aren’t attempting to both federate test.home for example as the federation domain in both. The federation domain should be different on both sides.

For example if you want to add user [email protected] from your server rc2.test.home it will then do a look up for the svc records on rc1.test.home and the txt record for the encryption key. But rc1.test.home needs to be the domain it sees it’s self as federating as. Same going the other direction rc2.test.home needs to be self aware at what address it will be federations at as rc2.test.home. With its dns configured so rc1.test.home can talk back

[tauame]

Just to be sure they are both using completely different domains from each other right? I’ve seen some cases where people had two different rocke.chat setups both with same domain.

Hi, I'm the author of the forum thread referenced by @col-panic , and we are not using the same domain. Each installation has its own domain.

rchat.domain1.com server has the "Domain" configuration in the Federation tab as "domain1.com"
rchat.domain2.com server has "domain2.com"

If I'm logged into rchat.domain1.com and want to speak to someone in domain2, I can find them on the "Directory" tab search for [email protected]. I can even message them, but they won't receive any message.

we are experiencing all the symptoms described in the "What does not work" section of this issue. More details in the Forum thread

[col-panic]

@geekgonecrazy yes, they are separate domains.
Both with respect to the dns and the rocketchat setting.

[SEisenbach]

Yes, in my case they are also different.

rc1.test.home-server has the Domain "rc1.test.home" in the Federation Tab
rc2.test.home-server has the Domain "rc2.test.home" in the Federation Tab

I can find all users at both servers, but the messages dont receive between the federated servers.

The users of federated Servers are always displayed as "online" btw. --> This seems to be buggy too?

[geekgonecrazy]

@alansikora would be best to help here 🙈. This is beyond me.

The only things I can think of are:

• use https
• make sure using different domains
• make sure dns are set
• make sure to test from both machines to see if they get the dns you set
• make sure they can communicate with each other

Beyond that I think it’s over my head

[col-panic]

Related https://github.com/RocketChat/Rocket.Chat/issues/17862

[qwertiko]

I am experiencing the same. Federation is working with Version 3.2.0 after setting it up with the help of the documentation and looking in the source code for the proper dns entries. But any version after no messages are sent between the instances

[col-panic]

The following image shows my test setup - on the right hand domain B sends a direct message to domain A - one can see in the log that the message is being generated (highest log levels on both sides) but never appears on the other side (uses are connected and found via federation search however). The same happens with a message from B to A

Here the log from B

rocketchat_1          | API ➔ debug POST: /api/v1/method.call/readMessages
rocketchat_1          | Meteor ➔ method readMessages -> userId: Psfs5PRtvn96i8ryR, arguments: ["EMd2fsZdPrmw9PorEPsfs5PRtvn96i8ryR"]
rocketchat_1          | API ➔ debug POST: /api/v1/method.call/sendMessage
rocketchat_1          | API ➔ debug Success {
rocketchat_1          |   statusCode: 200,
rocketchat_1          |   body: { message: '{"msg":"result","id":"39"}', success: true }
rocketchat_1          | }
rocketchat_1          | API ➔ debug Success {
rocketchat_1          |   statusCode: 200,
rocketchat_1          |   body: {
rocketchat_1          |     message: '{"msg":"result","id":"40","result":{"_id":"xivRSajoSwqboS2mC","rid":"EMd2fsZdPrmw9PorEPsfs5PRtvn96i8ryR","msg":"hello?","ts":{"$date":1597921653107},"u":{"_id":"Psfs5PRtvn96i8ryR","username":"RocketChatAdmin","name":"Administrator"},"_updatedAt":{"$date":1597921653196},"mentions":[],"channels":[]}}',
rocketchat_1          |     success: true
rocketchat_1          |   }
rocketchat_1          | }
rocketchat_1          | Integrations ➔ Outgoing WebHook.debug Execute Trigger: sendMessage
rocketchat_1          | Integrations ➔ Outgoing WebHook.debug Got the event arguments for the event: sendMessage {
rocketchat_1          |   event: 'sendMessage',
rocketchat_1          |   message: {
rocketchat_1          |     _id: 'xivRSajoSwqboS2mC',
rocketchat_1          |     rid: 'EMd2fsZdPrmw9PorEPsfs5PRtvn96i8ryR',
rocketchat_1          |     msg: 'hello?',
rocketchat_1          |     ts: 2020-08-20T11:07:33.107Z,
rocketchat_1          |     u: {
rocketchat_1          |       _id: 'Psfs5PRtvn96i8ryR',
rocketchat_1          |       username: 'RocketChatAdmin',
rocketchat_1          |       name: 'Administrator'
rocketchat_1          |     },
rocketchat_1          |     tmid: undefined,
rocketchat_1          |     _updatedAt: 2020-08-20T11:07:33.196Z,
rocketchat_1          |     editedBy: undefined,
rocketchat_1          |     editedAt: undefined,
rocketchat_1          |     emoji: undefined,
rocketchat_1          |     avatar: undefined,
rocketchat_1          |     alias: undefined,
rocketchat_1          |     customFields: undefined,
rocketchat_1          |     groupable: undefined,
rocketchat_1          |     attachments: undefined,
rocketchat_1          |     reactions: undefined,
rocketchat_1          |     parseUrls: undefined,
rocketchat_1          |     blocks: undefined,
rocketchat_1          |     token: undefined,
rocketchat_1          |     mentions: [],
rocketchat_1          |     channels: []
rocketchat_1          |   },
rocketchat_1          |   room: {
rocketchat_1          |     _id: 'EMd2fsZdPrmw9PorEPsfs5PRtvn96i8ryR',
rocketchat_1          |     t: 'd',
rocketchat_1          |     usernames: [ 'RocketChatAdmin', 'mdescher@domainA' ],
rocketchat_1          |     usersCount: 2,
rocketchat_1          |     msgs: 27,
rocketchat_1          |     ts: 2020-07-28T11:50:02.981Z,
rocketchat_1          |     uids: [ 'EMd2fsZdPrmw9PorE', 'Psfs5PRtvn96i8ryR' ],
rocketchat_1          |     default: false,
rocketchat_1          |     ro: false,
rocketchat_1          |     sysMes: true,
rocketchat_1          |     _updatedAt: 2020-08-20T11:06:33.618Z,
rocketchat_1          |     federation: { origin: 'domainB', domains: [Array] },
rocketchat_1          |     lastMessage: {
rocketchat_1          |       _id: '4LpmdWZNdbtPC4qCL',
rocketchat_1          |       rid: 'EMd2fsZdPrmw9PorEPsfs5PRtvn96i8ryR',
rocketchat_1          |       msg: 'hello?',
rocketchat_1          |       ts: 2020-08-20T11:06:33.534Z,
rocketchat_1          |       u: [Object],
rocketchat_1          |       _updatedAt: 2020-08-20T11:06:33.606Z,
rocketchat_1          |       mentions: [],
rocketchat_1          |       channels: []
rocketchat_1          |     },
rocketchat_1          |     lm: 2020-08-20T11:06:33.534Z
rocketchat_1          |   }
rocketchat_1          | }
rocketchat_1          | Integrations ➔ Outgoing WebHook.debug Starting search for triggers for the room: EMd2fsZdPrmw9PorEPsfs5PRtvn96i8ryR
rocketchat_1          | Integrations ➔ Outgoing WebHook.debug Found 0 to iterate over and see if the match the event.
rocketchat_1          | Push ➔ debug Sending notification {
rocketchat_1          |   createdAt: 2020-08-20T11:07:33.349Z,
rocketchat_1          |   createdBy: '<SERVER>',
rocketchat_1          |   sent: false,
rocketchat_1          |   sending: 0,
rocketchat_1          |   from: 'push',
rocketchat_1          |   title: 'RocketChatAdmin',
rocketchat_1          |   text: 'hello?',
rocketchat_1          |   userId: 'EMd2fsZdPrmw9PorE',
rocketchat_1          |   payload: {
rocketchat_1          |     host: 'https://ee.domainB/chat/',
rocketchat_1          |     rid: 'EMd2fsZdPrmw9PorEPsfs5PRtvn96i8ryR',
rocketchat_1          |     messageId: 'xivRSajoSwqboS2mC',
rocketchat_1          |     sender: {
rocketchat_1          |       _id: 'Psfs5PRtvn96i8ryR',
rocketchat_1          |       username: 'RocketChatAdmin',
rocketchat_1          |       name: 'Administrator'
rocketchat_1          |     },
rocketchat_1          |     type: 'd'
rocketchat_1          |   },
rocketchat_1          |   badge: 34,
rocketchat_1          |   sound: 'default',
rocketchat_1          |   notId: -1319342091,
rocketchat_1          |   apn: { category: 'MESSAGE' },
rocketchat_1          |   gcm: {
rocketchat_1          |     image: 'https://ee.domainB/chat/chat/images/logo/android-chrome-192x192.png',
rocketchat_1          |     style: 'inbox'
rocketchat_1          |   }
rocketchat_1          | }
rocketchat_1          | Push ➔ debug send message "RocketChatAdmin" to userId EMd2fsZdPrmw9PorE
rocketchat_1          | Push ➔ debug Sent message "RocketChatAdmin" to 0 ios apps 0 android apps
rocketchat_1          | Push ➔ debug GUIDE: The "appTokensCollection" is empty - No clients have registered on the server yet...
rocketchat_1          | SyncedCron ➔ info Starting "Generate download files for user data".
rocketchat_1          | SyncedCron ➔ info Finished "Generate download files for user data".

Is there any way to see what happens to this message? There got to be an error somewhere which seems to be simply swallowed? Or is there a way to see the result?

[SEisenbach]

I am experiencing the same. Federation is working with Version 3.2.0 after setting it up with the help of the documentation and looking in the source code for the proper dns entries. But any version after no messages are sent between the instances

So i tried version 3.2 with the same config i used in version 3.5 and it is actually working.
So it looks like the problem exists since version 3.2

Edit: It seems very very unreliable. Sometimes the messages do not receive.

[cbasca]

Here we are with the same problem. We have 4 internal servers that must communicate via federation to retire the current system in xmpp.

All domains are independent and in separate company networks, but accessible to each other.

We made an effort to upgrade the Rocket.Chat servers to version 3.5.2, but the problem remains.

It is possible to search for users from other domains in Directory - External Users, however when sending a message nothing happens. Users from other domains are always online in the client tool (any version).

It is a critical bug because it prevents the final implementation of the solution, as communication between domains is a minimum requirement.

[col-panic]

@geekgonecrazy is there any way we can escalate this problem and/or fund work on it?

[geekgonecrazy]

@sampaiodiego @rodrigok @alansikora seems it might still be broken. We have more and more reports.

I know it’s not production ready, but would be good to confirm this is an issue and see if it’s something we can fix

[alansikora]

@col-panic @cbasca @SEisenbach @qwertiko @tauame

Can you guys please update to version 3.6.3 and circle back?

[col-panic]

@alansikora waiting for the 3.6.3 docker release to become ready!

[ankar84]

@alansikora waiting for the 3.6.3 docker release to become ready!

It's ready for a while now.
I updated my test environment 4 or 5 hours ago.

[col-panic]

Which image do you use? The official one is stil with 3.6.1. https://hub.docker.com/_/rocket-chat?tab=tags

[sampaiodiego]

@col-panic you can use Docker image rocketchat/rocket.chat:3.6.3 .. official Docker image is waiting to be reviewed https://github.com/docker-library/official-images/pull/8776

[rajsardhara]

Confirmed working Federation on 3.6.3.

[qwertiko]

Works again with version 3.6.3

[col-panic]

Thank you very much for the patch - I can confirm that federated messages now work!

Still not working, however, are (for the sake of completeness, please don't get this wrong)

• #17201 and
• The remote user info is incomplete and ?wrong? e.g. the avatar is not shown, and the mail-address is "obfuscated"? - I will create a separate bug for this!

Thanks again for the willingness to tackle this problem!

[tauame]

Yes, the federation is working now (messages are being delivered)! Thank you for that, but for some reason, now the "Federation Dashboard" in Admin page hangs when opened.
From what I gathered, the browser loops sending requests to the server. I don't know if this should be registered as a new issue, or just reopen this one...

[sampaiodiego]

thanks @tauame for letting us know.. please open a new issue =)