Desktop: Client constantly hangs in random moments

[jpVm5jYYRE1VIKL]

It is not possible to predict when it will hang next time. But look like mattermost client have dead thread lock somewhere . Have same behavior at least on 3 Different computers (1 PC Windows 10 PRO and 2 PC Windows 10 home). Possible fix only over kill mattermost client processes in Task Mananger and restart. In most cases it happens in moment click trey icon during conversation.

[jasonblais]

Thanks @jpVm5jYYRE1VIKL for the report, few questions that come to mind,

1. Which Mattermost server version are you using?
2. Which Mattermost desktop version are you using?
3. Does it reproduce after accessing a particular channel/team?
4. Do you see a CPU spike during this time?

[jpVm5jYYRE1VIKL]

Sorry that forced to ask extra questions.

1. Server was 3.6.2
2. Client was 3.5.0
3. Nope completely random. Sometimes i just click on window and understand that it hanged.
4. It is hard to detect because all issues happens on powerfull computers (core i7) i not feel cpu spike.
   Now i upgraded server to version 3.7.3 and client to version 3.6.0 to see that happens. And if it happens i ll try to attach debugger to see dead lock.

[jasonblais]

Awesome, thanks! Let us know if the issue reproduces on the latest server/client version

[jpVm5jYYRE1VIKL]

Small update to issue. Just now happened again. Issue is not real hanging. It is freeze for 1 - 2 minutes. During this time application GUI completely not respond. Cpu not have any spikes. In task manager application not marked as not responding and not eat CPU.
After 1 - 2 minutes application unfreeze itself and continue to work. Detected such behavior because started to search debugger to attach and suddenly found that application not hang anymore. Issue happened with 3.7.3 server and 3.6.0 client.
Did following actions before issue happened (not think that it help to recreate but for notice):
1) I sent to another person Link(web url)
2) Person sent to me answer . I got an answer and see in windows that person typing something else and my window already hang. In such position. I cannot even minimize or close window.
Most of freezes happens in such situations.

Is it possible to enable extra debugging somewhere ?

[jasonblais]

Thanks @jpVm5jYYRE1VIKL - how did you set up your Mattermost environment?

Are you using any proxies?

[jpVm5jYYRE1VIKL]

Yes using. https://nginx -> http://mattermost:8065
Settings standard :
location /api/v3/users/websocket {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 50M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_pass http://backend;

location / {
client_max_body_size 50M;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_pass http://backend;
}

[jasonblais]

@yuya-oc is there any debugging available for the desktop app?

Someone else has the client freezing in Windows 10: https://github.com/mattermost/desktop/issues/347#issuecomment-291153639

[yuya-oc]

It is freeze for 1 - 2 minutes.

As far as reading the behavior, it sounds like a performance issue. (but it's also weird because CPU doesn't have spike...) The application consists of many processes like Chrome, so the process which @jpVm5jYYRE1VIKL had attached might not be appropriate.

I feel "Timeline" and "Profiles" tabs of developer tools (Ctrl+Shift+I) might make sense to measure the problem.
By executing document.getElementById('mattermostView0').openDevTools() in the console, it opens another console which is attached to a webview. It can inspect the inside of the webview.

If the problem happened in the main process, we need to follow Electron's instruction for debugging.

[jasonblais]

Thanks @yuya-oc

@jpVm5jYYRE1VIKL Thank you for all your reports thus far, it's really helpful. Would you be open to help provide the information described above next time the issue occurs?

I'm wondering if it could be a memory issue on the app

[jpVm5jYYRE1VIKL]

Yes of course when next time it happen all try to provide as more as possible information.
But it is really hard to catch because of random nature.

I feel "Timeline" and "Profiles" tabs of developer tools (Ctrl+Shift+I) might make sense to measure the problem.

Not sure that this is possible because mattermost window completely not respond. Not possible to use menu at all to switch on dev tools. Even windows cursor not changing when you cross over mattermost window. May be this is extra property which can help to detect issue. And i also like idea about posible javascript issue on client itself. In one of freeze moments i looked on nginx proxy and not found anything strange (cpu load, network errors, logs etc - everything was ok). Maximum what i can try to do is to connect to console after it anfreeze and copy logs for 1 - 2 minutes.

[jasonblais]

Thank you @jpVm5jYYRE1VIKL, really appreciate it. Would getting the logs for 1-2 minutes after the app is accessible again be helpful @yuya-oc?

[yuya-oc]

mattermost window completely not respond

Does "window" include the title bar? Or only content area (webview) of the window? Once I had met a case webview freezes on Mac, but its window frame could respond.

Would getting the logs for 1-2 minutes after the app is accessible again be helpful @yuya-oc?

Yeah, for now I'm not sure what's actually happening in the app. So need to know what appears.

[jpVm5jYYRE1VIKL]

Basically exist 2 different types of freeze. One person complained that she see black mattermost window . I personally seen just simple freeze with title bar. But not responsible title bar. I cannot use any controls on it during time of freeze. Last week was without freezes. I continue monitoring.

[jpVm5jYYRE1VIKL]

Update: Hm look like nothing to copy from console log because look like it empty.
Added extra sympthom.
Application freezed this time in following conditions.
Application was in tray. Appeared pop up incoming message , I clicked immediately on pop up . Appeared application window which and it freeze. After some time application unfreeze i immedeately started developers tools and looked on console last messages i see was 2 hours ago .

[jasonblais]

Thanks @jpVm5jYYRE1VIKL for the continued follow-up. Really appreciate all the information.

For the "black screen" issue, there are a few troubleshooting steps in our docs: https://docs.mattermost.com/install/desktop.html#troubleshooting. In particular,

• First try to clear cache and reload the app from View > Clear Cache and Reload or by pressing CTRL/CMD+SHIFT+R.
• Next, quit the app and restart it to see if the issue clears.

For the client freeze issue --> I just tried clicking on a desktop notification while the app was minimized to the tray, but I wasn't able to reproduce it so far. I was on a light load though (not much else on my machine)

Did the post you receive contain any attachments (files, GIFs) or link previews? Does the channel itself have mostly text-based messages, or does it have GIFs / link previews to generate? We've heard performance issues before about channels containing lots of GIFs (see https://github.com/mattermost/desktop/issues/215 for an example).

anything else @yuya-oc that could come to mind?

[jpVm5jYYRE1VIKL]

Channels not have attached files at all. Have some links and pictures links (i guess they cached) in channel line.
Channels mostly text . 98 % text , 1.5 % links on pictures and 0.5 % just web links . These days not have a lot of messaging so cannot add extra data yet. I continue keep an eye on issue.
Can it be binded with Meteor itself ? O may be collaboration meteor and windows internals ? may be embedded browser ?
Update : One of the acccount where freeze happens quite often registered from 2 computers . Freeze sometimes happens during conversations on both computers. For example i see on one PC using remote terminal and on another PC physically.

[yuya-oc]

While a window is closed into the tray icon, CPU usage is limited to reduce power. But I'm not sure whether it impacts to app's stability.

Can it be binded with Meteor itself ? O may be collaboration meteor and windows internals ? may be embedded browser ?

Sorry, I'm not sure what you mean. The application is build with Electron, and its webview is used to load webapp of Mattermost server. So it works like embedded web browser.

[yuya-oc]

On Windows, "not responding" means that the native message loop is blocked due to some reason (like too heavy load). So I feel the matter is not webapp. But...I'm not sure the reason. I feel such a heavy load is not existing excepting GC.

[jpVm5jYYRE1VIKL]

Sorry my mistake. Of course i meant Electron.
Is it possible somehow to disable this feature "CPU usage is limited to reduce power" somehow ? Just to check. May be it is really issue between switch from limited CPU usage mode to normal.

[yuya-oc]

Please use --disable-renderer-backgrounding from command line. Otherwise, need to use webPreferences.backgroundThrottling: false by modifying source code.

[jasonblais]

@jpVm5jYYRE1VIKL @yuya-oc

We got a couple of reports about the app crashing due to high memory usage.

Could this be the reason why the app is constantly crashing for you @jpVm5jYYRE1VIKL?

Anything we can do to reduce the high memory use issue @yuya-oc?

[jasonblais]

^@jnugh also pinging you in case you had any thoughts about the high memory usage, I think we discussed earlier about it a little.

[jpVm5jYYRE1VIKL]

@jasonblais
Nope not think so. All affected PCs have a lot of RAM. Pc which most affected have 32G ram ,
For now i starting to think that issue binded with usage same credentials on 2 or more workplaces.
@yuya-oc
for sure --disable-renderer-backgrounding not solved issue. But feels freeze time bit decreased. Not 100 % sure. Still testing.

PS: I suggest to do following test to exclude. But it is quite complex and may be contain not nessesary steps
1) login using account1 on one PC(PC1) using mattermost desktop client (client window have to be open)
2) login using account1 on another PC(PC2) using mattermost desktop client(client have to be in tray)
3) login using account2 on another PC(PC3) using mattermost desktop client
4) Do some messaging
5) leave it without activity for 1 hour (PC1 must not sleep), PC2 can sleep, PC3 not important is it sleep or not.
6) Wake up PC2. And not give it to sleep 15 mins.
7) Send message from account2 to account1
8) In my case client on PC2 quite often temporary freeze.

[yuya-oc]

@jpVm5jYYRE1VIKL Thanks! Assuming that, I feel there are two possible cases.

A. When fetching latest messages after PC is woken from sleep.
B. When there are two clients and they have too different states.

I didn't notice the checklist doesn't exist in your first post. To make sure, did you also test the problem with Chrome?

[jpVm5jYYRE1VIKL]

@yuya-oc
Nope of course i not tested with browsers. I only use standalone desktop clients everywhere. And this is the main target for me. To avoid browser usage. But i ll try to make extra connection with browser for test. When i did my first post i was not so deep involved in process of fixing this issue.

[jpVm5jYYRE1VIKL]

Update: I tested with opened Chrome and loggen on from Chrome to mattermost server and Mattermost desktop client on same PC . Messaging in Chrome working fine , messagin in mattermost freeze as usual.

[jasonblais]

Thanks @jpVm5jYYRE1VIKL, looks like there's something in the desktop app causing it

Even though it was mentioned that memory usage shouldn't be an issue, I've been following both CPU and memory usage of the app in the last couple of days and have noticed the following (recorded via Task Manager):

• when starting the desktop app, the memory usage is generally low: 25MB for the app, < 200MB if you add all the background processes from the Task Manager that are related to the Mattermost app (it's kind of high in my opinion, but it shouldn't be causing issues)
• after a few days of usage: 37MB for the app (still okay), almost 1GB if you add all the background processes

So the memory usage does accumulate quite significantly over time, and that could cause general slowness, perhaps even a potential freeze if an action spikes up the memory usage a lot. For instance, I'm seeing large swings in memory usage (up to 300MB) when I switch a channel or perform other simple actions.

With regards to system info: Running Windows 10 on two Mattermost instances, where only one of them is active (the other has a few messages per day)

[jpVm5jYYRE1VIKL]

Extra update. I started sysinternals process monitor and found one strange issue

19:05:06.8628355    Mattermost.exe  8596    UnlockFileSingle    C:\Users\user1\AppData\Roaming\Mattermost\Cookies   SUCCESS Offset: 1,073,741,826, Length: 510
19:05:06.8629183    Mattermost.exe  8596    LockFile    C:\Users\user1\AppData\Roaming\Mattermost\Cookies   SUCCESS Exclusive: False, Offset: 1,073,741,826, Length: 510, Fail Immediately: True
19:05:06.8629834    Mattermost.exe  8596    UnlockFileSingle    C:\Users\user1\AppData\Roaming\Mattermost\Cookies   SUCCESS Offset: 1,073,741,825, Length: 1
19:05:06.8630391    Mattermost.exe  8596    UnlockFileSingle    C:\Users\user1\AppData\Roaming\Mattermost\Cookies   SUCCESS Offset: 1,073,741,824, Length: 1
19:05:06.8630974    Mattermost.exe  8596    UnlockFileSingle    C:\Users\user1\AppData\Roaming\Mattermost\Cookies   SUCCESS Offset: 1,073,741,826, Length: 510

Offset numbers like : 1,073,741,824 looks bit strange . The most strange that file Cookies is only 7kb size in reality.
PS: Nevermind .such locks i see even when app working fine.

[jasonblais]

@yuya-oc @jpVm5jYYRE1VIKL

there's a report where a user is able to consistently reproduce the issue when the application is minimized and they receive notifications

https://github.com/mattermost/desktop/issues/520

The issue no longer occurs after disabling desktop notifications.

[jnugh]

Something similar occurred to me, I'm not sure if it is 100% related but it kinda looks like it is.
I have an STRACE showing a deadlock. I'm building electron right now to get some debug symbols (and maybe a better stack to identify the root cause...)
mattermost-strace.txt

threads-bt.txt

UPDATE: I'm pretty sure my issue was different to the original problem.
I traced my issue back to something very low-level, (I'd guess it was an graphics driver issue in my case as I noticed some problems with chrome and fullscreen videos as well). Multiple threads were hanging in S32A_Blend_BlitRow32_SSE2 while syncing a register load (_mm_loadu_si128) which is nothing we can fix here.

[jpVm5jYYRE1VIKL]

I think issue can be merged. For now disable notifications solved issue. i e it is not real solution. I still testing.

[jasonblais]

@yuya-oc is there anything we can do about the notifications issue? It appears to be the root cause for freezing the Windows 10 client in #494 and #520

[yuya-oc]

Assuming that, we need to know what part is wrong in the notifications. i.e. Electron itself or our injection code for notifications. So I prepared two versions to check that. But unfortunately I can't reproduce the problem yet.

• Plain Electron's notification: https://circleci.com/gh/yuya-oc/desktop/247#artifacts
  
  
  
  • Note: When clicking notifications, the window is not activated due to Electron's spec.
  
  
• Refactoring with class syntax: https://circleci.com/gh/yuya-oc/desktop/246#artifacts

@jpVm5jYYRE1VIKL @mdementev Wondering if you could test with these builds. You can download them from CircleCI by expanding circle-artifacts.xxxxxxx/.

[jasonblais]

@jpVm5jYYRE1VIKL @mdementev Would either of you like to try if the app keeps freezing on the above test builds linked by @yuya-oc above?

[jpVm5jYYRE1VIKL]

For now i use workaround "disable notifications at all". Found build on page you provided. Windows defender prevent execution of it. If you add this change to next release of course i ll report is change fixed or not.

[yuya-oc]

I see. The installer was not code-signed and downloaded from internet, so Windows defender would prevent it at the first time. It's understandable.

@jasonblais To get @jpVm5jYYRE1VIKL's feedback and refactoring, I will create a PR. Would you check it?

[jasonblais]

@yuya-oc I'm happy to test it, but I haven't been able to reproduce it either.. I actually downloaded the test build from https://circleci.com/gh/yuya-oc/desktop/246#artifacts and haven't hit any issues.

[yuya-oc]

Okay, so may I push to master to add to next release?

[jasonblais]

Yeah, I'm okay with that.

How confident are you about the fix @yuya-oc? If we have to cut a 3.7.1 dot release for https://github.com/mattermost/desktop/issues/537 before Mattermost 3.10 (depends if it's a bug on the platform repo or desktop repo), we could consider including this fix in the dot release as well.

[yuya-oc]

Hmm...I'm not too confidence... But if there is 3.7.1, I agree to include it.

[yuya-oc]

@jasonblais For now, we don't release v3.7.1, so I would like to merge https://circleci.com/gh/yuya-oc/desktop/246#artifacts (Refactoring with class syntax) into master. Any thoughts?

[jasonblais]

Yes, that sounds good. I used that artifact for some time and didn't notice any issues.

[yuya-oc]

Merged at 0839e71

[jasonblais]

@yuya-oc My machine has been slower after the most recent Windows update and I've noticed the app freezing for a few seconds when I get a desktop notification (only sometimes).

Is this still the most recent artifact to test the fix you merged? If so, I'll give it a try and see if it helps. https://circleci.com/gh/yuya-oc/desktop/246#artifacts

[yuya-oc]

@jasonblais Yes, please use it. The master branch updated electron though, I think it's not related to this issue.

[jasonblais]

Sounds good, I'll test it out next week to see if I see a difference.

[jasonblais]

@yuya-oc FYI, I don't think the fix helped too much, unfortunately... I still experienced slowness with the test build but when I disabled desktop notifications today, I haven't had any issues. The freezing was worse on v3.7 though (I think, at least when I tested this morning).

I'm able to reproduce it after using the app for a few hours, and it starts to happen around mid-day during my working hours.

When you're back, I'd be happy to provide any info if needed..haven't really noticed any patterns and the console shows no errors.

EDIT: Additional notes:

• The issue seems to get progressively worse as more time goes on
• The freezing happens between when a message is received (and a channel gets marked unread) and when the desktop notification appears. During this time the app is frozen.

[jasonblais]

^update on this: It doesn't seem to be related to desktop notifications for me.

It seems to be happening while I'm receiving an unread -- or it's something else. Will investigate further..

[jasonblais]

Closed via #563

[jasonblais]

@jpVm5jYYRE1VIKL We released v3.7.1 of the desktop app which included a fix for this issue.

It would be great if you could help download the app and let us know if you're still experiencing problems with the client freezing https://about.mattermost.com/download/#mattermostApps

[jpVm5jYYRE1VIKL]

Problem Gone for me.

[jasonblais]

Sweet, glad to hear :)