Mattermost-server: Not all messages show up on all devices

[shochdoerfer]

We upgraded a few days ago to 5.12.2 and have a problem now that sometimes messages do not appear properly on different devices. E.g. messages show up on Mattermost on my mobile phone but not on the Desktop client. Refreshing the Desktop client does not help. For different user different channels are affected and some users also have the problem with alternative clients (e.g. Rambox).

[amyblais]

Hi @shochdoerfer, would you be open to sharing reproduction steps, environment details, and server log errors?

[collegenetjacob]

We have this(?) issue all over the place. Lots of reports of "mattermoist is exploding". I see it on the web client right now; all the messages in one or more channels since last time I used the web client are missing. I can see new messages since I came back. And everything is still there on my phone. So if you really do need more info, we can probably provide it (but I hear through backchannels the MM team can already reproduce).

[shochdoerfer]

We have upgraded to 5.12.3 this afternoon and the problem still persists. For me personally things are working fine on mobile (standard Mattermost app, nothing custom), on the web but fails von Rambox. So far just one specific channel is the problem, other channels seem to work fine. Rambox has two options "Clear Cache" & "Clear Local Storage". After using both and logging out & in again, things are working again.

Maybe this is some frontend caching problem and not something on server side?

[collegenetjacob]

Users report they can refresh by force quitting the phone app or restarting the desktop app, but forced refresh doesn't seem to do it for the web app. Perhaps it's stored on the server with the user's session.

[shochdoerfer]

Good catch, yes forcing a refresh did also not work in the web app & in Rambox for me.

[amyblais]

@shochdoerfer @collegenetjacob Does this describe your issue:

A channels gets marked unread, you switch to that channel, but there are no new messages. After refresh or clearing cache, the new messages in that channel appear.

[collegenetjacob]

I haven't seen it when a channel is unread (but from reports we've received that's probably true). I resumed an existing web client session that had not been loaded for some days and all the messages from that time were missing from the channel I had up when I closed the browser.

Shift refreshing the browser window didn't fix it. Closing and restarting the browser didn't fix it. Logging out and back in again did (so far). I think a fresh session has to be created to get back the missing messages.

We've also had other odd reports like notifications not getting sent, but they may just be outliers. People are more sensitive to everything when they know there's a problem.

[amyblais]

We are currently investigating this report and it might be related: https://github.com/mattermost/mattermost-server/issues/11079.

[shochdoerfer]

11079 is not an issue in our case, the channel is newly created and contains maybe 100 - 200 messages.

The channel also does not get marked unread in my case. It looks like I have read all messages in the channel. Click on it and I can see the messages I read last but not the newer ones. I can't recall if I was just missing the last message or multiple ones.

[tarabanton]

@shochdoerfer @collegenetjacob Does this describe your issue:

A channels gets marked unread, you switch to that channel, but there are no new messages. After refresh or clearing cache, the new messages in that channel appear.

In my case this description also correct. Appears even in PM.

[collegenetjacob]

We had a user report if you know about a missing message and can find it in search, then jumping to it in the channel via the search results will force the channel to refresh and gain back the missing messages.

[amyblais]

Are there any log error messages that any of you are seeing?

[collegenetjacob]

Are there any log error messages that any of you are seeing?

Nothing at all in ours. Well, there's tons of the usual stuff but nothing new/unusual.

[shochdoerfer]

No log error messages here that would point to the source of the problem. That's why I am thinking it might be a frontend issue, maybe some caching problem.

[collegenetjacob]

I can't say for certain but it seems like having multiple clients logged in at one time might be involved in the problem. I asked our userbase and nobody who always uses just a single client reported the problem (but the majority case is probably at least two clients).

[trustedsec]

I've been running into the same thing at our organization.

[amyblais]

FYI that our devs are currently on top of investigating issues around missing messages.

[furkanmustafa]

Experiencing same issue. It started happening after upgrading to 5.12.x (we're currently on 5.12.3).

[shochdoerfer]

@amyblais any progress? Any help needed? Is it possible to downgrade to 5.12.1 from 5.12.3 easily or where there changes made to the database?

[amyblais]

Our devs are currently on top of investigating issues around missing messages and I have more updates in the next few days. Here are instructions for downgrading: https://docs.mattermost.com/administration/downgrade.html#downgrading-mattermost-server. There were no database changes between 5.12.1 and 5.12.3.

[amyblais]

Can people help with more information about their environment?

• OS and version, device, browser
• Any notable Mattermost configurations (such as HA, Elasticsearch, image proxy, SSO)

[collegenetjacob]

Reports here are so broad I'm pretty sure it's roughly any client. I've personally seen it on the web client in Firefox 60.7.0esr on Linux (Debian 9).
Our Mattermost server is not HA, Elasticsearch, image proxy, or SSO.

[amyblais]

Fixes submitted for v5.13 and master:

• https://github.com/mattermost/mattermost-server/pull/11618
• https://github.com/mattermost/mattermost-server/pull/11619

[shochdoerfer]

Will there also be a fix for the 5.12.x series or will 5.13 come out soonish?

[arjitc]

@shochdoerfer if I recall correctly a new release comes out on 16th of every month (~3 days)

[amyblais]

We are also aiming to cut a v5.12.4 dot release for this today (or by tomorrow latest).

[amyblais]

v5.12.4 is now released and available to download.

[shochdoerfer]

@amyblais thx! Updating our instance now. Will let you know if the problem still persists. Should I leave the issue open for now?

[amyblais]

Thank you, we can leave this issue open for now,

[shochdoerfer]

No issues so far. Seems the problem is solved.

[amyblais]

Good to hear it, I'll close this issue for now but if anyone still sees the issue after upgrading, please post here and re-open this.