Mattermost-server: Slow text entry

[hjek]

Summary

Text entry is slow when entering messages on Mattermost

Steps to reproduce

• Start typing a message on Mattermost
• Wait for text box to catch up with what was typed, sometimes 5+ seconds

Software used:

• Firefox 78.0.2 (64-bit)
• Mattermost 5.23.1

Expected behavior

I'm expecting the message box to catch up with my typing with a delay less than 200 ms. The CPU is maxing out whenever I start typing. I can confirm this happens on Chromium as well, and both on Debian and Fedora on different devices.

Observed behavior (that appears unintentional)

Possible fixes

Sorry, I'm not sure how to fix this, but text boxes work fine on other websites, such as Github.

[agnivade]

/cc @sudheerDev

[amyblais]

@hjek Are there any console errors you could help share?

[hjek]

Not getting any console errors. :neutral_face:

[hjek]

Console log from when I've opened Mattermost and start typing in a message:

This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. off-topic
Loading bigbluebutton plugin index.js:107:16
Loading gabay.mattermost-rtl-plugin plugin index.js:107:16
Loaded gabay.mattermost-rtl-plugin plugin index.js:93:20
Loaded bigbluebutton plugin index.js:93:20
websocket connecting to wss://*****.***/api/v4/websocket websocket_client.jsx:35:20
computer woke up - fetching latest 2 needs_team.tsx:92:24

[sudheerDev]

@hjek, I don't see any logged issue so far for this issue, Can you send me a chrome performance profile of the issue?.

To record a performance profile.

Right-click on the tab.
Click on inspect
Click on performance tab
Click on record
Start typing in the textbox
Stop recording the profile.

[hjek]

Can you send me a chrome performance profile of the issue?

@sudheerDev, Profile-20200715T154011.json.zip

[hjek]

Was the performance profile of any use? Is there more information I can provide to aid debugging, or are there any settings I can twiddle to narrow down the issue?

[sudheerDev]

@hjek Thanks for sending in the profile.

Could you try few things and let me know how typing feels in the scenarios

1. Chrome incognito
2. Another browser(Like firefox)
3. i see that you have rtl plugin enabled. Can you ask your admin to disable and then try the app in incognito

Let me know if any of the following steps result in better typing experience

[mm--]

I'm also getting this issue, with both Firefox 77 and Chromium on GNU/Linux, with Mattermost version 5.22.1. I don't see the RTL plugin enabled on the instance I'm using.

I notice that the typing lag is slightly decreased when Markdown formatting is turned off. I only have access to a minified JS source, so I noticed that an event handler named Cn was taking the most time. Searching within the minified source, I saw there was something like Cn.MARKDOWN_PREVIEW.

Intuitively, it seems like there's some keydown event handler that's doing too much. Like, maybe the keydown event is causing a React state change which causes every dependent thing to be recalculated on every single keystroke. It seems like some of these events (like Markdown previewing or name/channel suggestions) should be debounced.

[Schleuse]

I've had performance issues in a channel which contained highlighted code written in a specific way.
After some profiling I've figured out that the Highlighting seems to be the Problem (in my case anyway).

Deleting the highlighted code fixed the performance issues for me. I've opened an Issue at hightlight.js for this.

[jeffrysleddens]

We ran into the same problem after going from 5.19.1 to 5.25.5. Very sluggish text input. Enabled plugins: Jira, Matterpoll and Memese. Used browser: Firefox 80.0.1.

[ChristianCiach]

Same issue since recent update to Mattermost 5.26.2 and using Firefox 81.0.1. Text input is also sluggish when using Chrome, but not as bad as with Firefox. Unfortunately I don't know which version of Mattermost we were using before the update that introduced the issue.

For me, it probably has nothing to do with highlighted code, because this issue presents itself in every chat, even those without any code.

[amyblais]

Hi all,

1. Do you have any specific tools on your end that we can use to measure the database performance of the instance? If yes, can you please share the results and the documentation for reference?
2. Are there any server log errors or console errors that you can help share?
3. Can you help test if you see same results in different browsers?
4. Can you help disable any plugins enabled on your server and see if it changes the behaviour?

[jeffrysleddens]

Interesting fact, when I perform a search in a channel and leave the search results open the typing slowness in the chat input is greatly intensified.

Answering the questions:

1. Not really. Can do any measurements that are needed if you tell me how/which.
2. No errors in the log or console errors that I can relate to this issue.
3. For me it only happens in Firefox. I have also tested Chrome and Edge browsers.
4. I have disabled all plugins on our server, but that did not improve the situation.

[RalfJung]

For me, it probably has nothing to do with highlighted code, because this issue presents itself in every chat, even those without any code.

I have the same problem. When I do not do anything else on my laptop it is okay, but once there is anything going on in background (even if just some of my 4 cores are used), Mattermost becomes basically unusable. Other websites (GitHub or Zulip) still have very responsive text input, but Mattermost can have multi-second input latency.

This is using Firefox 82. I even reported this in Firefox but actually it affects only Mattermost, not other websites.

Let me know if gathering more profiles or so would be useful.

[ahmaddanialmohd]

Hi, everyone.

I tested this on a fresh vanilla instance on the following environment:

• Mattermost Server 5.26.2
• Firefox 82.0 (64-bit)
• Chrome 86.0.4240.111 (Official Build) (x86_64)
• Mac OS 10.15.7 (19H2)

While comparing the text input delay between Chrome and Firefox, I do not see any significant difference between the browsers.

Putting the possibility of addons contributing to the issue into perspective, can you confirm if the problem is observed in the Incognito mode of Firefox?

When the issue occurs, how is the resource consumption like for Firefox? (using top or htop to verify this). For example:

Additionally, can you also reproduce the issue on https://community.mattermost.com/ if you have an account?

[jeffrysleddens]

For me the problem occurs on busy channels or channels with a lot of history. In new empty channels the typing is fast as normal.

[RalfJung]

Putting the possibility of addons contributing to the issue into perspective, can you confirm if the problem is observed in the Incognito mode of Firefox?

I created a fresh profile for Firefox. To reproduce the problem in its extreme form, I then also had to enable WebRender. Then I see an extreme slowdown in Mattermost once there is background load on my CPU, but not on other websites.

Without WebRender, the slowdown is much less pronounced, and does not seem to be much worse on Mattermost than it is elsewhere (without having objective measurements, this is hard to tell).

[RalfJung]

Okay I have to correct that, even without WebRender the latency is still notably higher in Mattermost than on other websites.

[amyblais]

Could be related: There was a forum post about this earlier and we submitted a PR for the Go project: https://go-review.googlesource.com/c/net/+/237957 (still under review).

[hjek]

Putting the possibility of addons contributing to the issue into perspective, can you confirm if the problem is observed in the _Incognito_ mode of Firefox?

Yes, the issue happens in incognito mode in Firefox, too.

Additionally, can you also reproduce the issue on https://community.mattermost.com/ if you have an account?

I can reproduce on community.mattermost.com as well.

[RalfJung]

A firefox dev analyzed a perf profile that I uploaded and noted

The profile that you did include has very large input lag, but it looks like all of that time is being spent running JS in Mattermost.

So it looks like Mattermost is doing some JS on the critical path between entering a character and showing it on the screen. If possible, it would be great if you could switch to some other API that does not block processing the input until JS is done -- APIs which seem to be used by applications such as GitHub or Zulip, which both provide "enhanced input mode" (emoji, auto-completion of @-mentions) without causing enormous input delays.

[amyblais]

Opened a ticket: https://mattermost.atlassian.net/browse/MM-30407

[hjek]

When the issue occurs, how is the resource consumption like for Firefox? (using top or htop to verify this).

Here are two screenshots showing resource consumption while typing and not typing in community.mattermost.com in Firefox.

[roelandschoukens]

I have the same issue both on the desktop app and in a browser, but it only happens if the extra panel (recent mentions or search) is open.

[kongr45gpen]

Can reproduce this on desktop Firefox. It is especially pronounced on the RHS with long threads open (eg >300 messages sent).
A quick performance analysis shows too many DOM Events caused by onkeydown events on the text input.

[Susticle]

A user of an instance maintained by me notified me about this a while ago. Now I was able to see the problem myself with Chromium/Firefox (Linux) in a longer discussion thread.

CPU load is very high when it happens and seems to be worse with Firefox.

So for me it appears similar as described here:
https://github.com/mattermost/mattermost-server/issues/15012#issuecomment-714669237

[hmhealey]

Is anyone on Mattermost 5.30 or newer able to still reproduce this issue? We made some decent performance fixes, particularly for users with a large number of channels and when typing in the RHS, in that release, so this problem might be resolved now.

[ChristianCiach]

@hmhealey The issue has been resolved for me in the meantime. Unfortunately, I am not the one in charge of our mattermost installation, so I cannot tell you which updates were applied by our IT department or when these updates happened - or if something else has been changed that resolved this issue.

[hmhealey]

As long as something fixed them, that's good at least. Hopefully it's improved for others as well.

[kongr45gpen]

For me, unfortunately the issue still exists, albeit a bit less pronounced (Mattermost 5.31.6, Firefox 84.0.2 on 64-bit Linux). I have about 100 channels in the LHS, and this issue only shows up when the RHS sidebar is full of messages (e.g. long thread opened, or many returned search results)