Meshcentral: Windows 7 32bit - CPU at 100%

[unguzov]

I have a problem with Windows 7 32bit agent. In about two hours CPU goes to 100%. A restart of the service solves the problem, but then CPU goes back to 100%.

Tested on 2 VMs and one laptop with Windows 7 32bit.

[Ylianst]

Are you doing any operations on the MeshAgent (remtoe desktop, files, terminal?) or doing nothing at all? Thanks.

[ryanblenis]

It is also useful to know if you are utilizing plugins. If so, does the same issue occur with plugins disabled? Just trying to limit the liability of MeshCentral since plugins can do work in the background, and it may be unrelated to the MeshAgent itself.

[unguzov]

Two plugins are active: Event Log and ScriptTask. Event Log does not show any data from Windows 7 (works fine on Windows 10).

Bug is not limited to 32bit - today two desktops with Windows 7 64 bit has the same problem.

I will test now with all plugins disabled.

[ryanblenis]

With those two plugins, EventLog needs PowerShell v3+ in order to collect events (otherwise it won't work), and it really depends on what scripts you're sending in ScriptTask- I don't imagine either causing a notable CPU bump unless a script you're sending it doing a good amount of work.

Thanks for the update and keep us posted on the results!

[unguzov]

After more than 3 hours with Windows 7 and disabled plugins - it works fine. I will continue monitoring these workstations.

I do not use ScriptTask - plugin was just activated, but not used.

[ryanblenis]

Next test up would be to enable plugins one at a time (and choose the option to refresh your cores when enabling / disabling) to make sure and narrow it down to the plugin.

FYI I just updated EventLog with PowerShell version detection to disable the plugin on systems where PowerShell version less than 3 is detected. This should keep it from doing anything on those endpoints unless you have PowerShell 3 installed.

Please keep us updated on your findings. Thank you!

[unguzov]

Let me confirm that Windows 7 32/64bit works fine with disabled plugins.

[ryanblenis]

Thanks for the update. Looking forward to seeing which plugin is / was causing the issue with plugins enabled, and enabling the plugins one at a time.

[Ylianst]

Thanks @ryanblenis for taking care of this! Much appreciated.

[ryanblenis]

No problem. I don't know if there's a post / section where you indicate the best method to report issues / bugs, but if there is there should probably be a section on disabling plugins to see if the issue persists before posting to MeshCentral's issues. If there isn't any guidelines on posting, I'll just be keeping an eye on the issues and interject where it appears it may be needed. I haven't seen too many people look into developing plugins yet, but I've got some good metrics on my GitHub Insights section that says people are visiting the project / using them. I'm still hopeful they take off and we can get a few more devs on board!

[Ylianst]

Personally, I got no problem having plug-in bugs filed on this GitHub. I will gladly assign them to you :) I think plug-in are really valuable and the capabilities are so far, impressive.

[Ylianst]

@ryanblenis FYI. I enabled plugins on my dev server and it looks like moving to MeshCentral v0.5.0-x broke the plugin minimal version check for the server. I just pushed a fix for it. Don't hesitate to let me know and change the code if did not not do it as you like. Look for checkMeshCentralVersion() in pluginHandler.js.

[ryanblenis]

Hi @unguzov ,

Any luck narrowing down the offending plugin in this situation? Or did my update resolve it for you?

[ryanblenis]

FYI it looks like EventLog was confirmed here as the cause of Win7 CPU issues. It may have already been fixed in an update, but I am looking for confirmation from someone who uses it with Win7. (Win7 was removed from my test machines when it went EOL, but I'll see if I can find a test box). Anyone else who can comment on their findings would be helpful. Thank you.

[mjakovina]

This issue is still happening to us on Windows 64 and Linux systems (not sure about 32bit)
We do not use plugins

[krayon007]

Can you be more specific? Windows 7 32 bit and which Linux? 64 bit in which distro, and which version? When does it go to 100% On startup? On idle? After specific usage?

[krayon007]

And which server version are you using?

[mjakovina]

Our server version is 0.5.8
I can confirm this problem on following systems:
Linux Mint 19.2 64bit with kernel 4.15.0-91 (running in VM)
Windows Server 2008 R2 64bit (running in VM)
Windows 7 Pro SP1 64bit (running in VM)

It never appears on startup.
We have not find a way to repeat a problem - we do not know when it starts.
It happened on desktops/laptops too but we have not document it (people turn them off each day so it probably disappears after reboot)

We will upgrade our server to new stable version and report future ocuurencies here

[unguzov]

Sorry for the delay @ryanblenis - it is a busy month! When plugins are disabled it works fine, but I still see 100% CPU usage on random Windows 7 workstation - on one or two in a week. Just updated to the latest stable version (0.5.11) - I will watch about CPU usage and if it works OK I will enable Event Log plugin first.

[mjakovina]

I see it right now on on of our Win2008R2 64bit servers.
I see 2 Meshagent processes, one of them is running at 100% of 1 CPU.
It started running at 100% just after I connected to it via MashCentral2

@krayon007 I will leave it running so I can give you additional info if you need it

[krayon007]

When you say just after you connected to it, you mean with Terminal or Desktop?

[krayon007]

Ok, I happen to have a Win2008 R2 64 bit test setup... I was able to reproduce the situation. It seems to only occur after i disconnect a KVM session... I'll see if I can track it down... It looks like the child KVM process doesn't exit, it goes into a busy loop for some reason...

[krayon007]

Looks like it's not even the KVM, it's the connection bar... When I close the connection bar this happens.. I can only reproduce this on Windows Server 2008 R2 and Windows Server 2016 right now... So I'm looking into it...

[krayon007]

Ok, I fixed the issue. It was caused by a windows specific scenario where there could be a dispatch loop trying to unwind a thread apc. The fix will be picked up by the next agent update.

[ryanblenis]

@unguzov let's save that test until @krayon007's fix goes out to make sure it isn't something in the vanilla agent. Thanks for following up!

[krayon007]

Changes are live in MeshCentral_v0.5.15

[mjakovina]

When you say just after you connected to it, you mean with Terminal or Desktop?

@krayon007 Desktop

[krayon007]

@mjakovina , did you have the notification bar enabled? The bug I just fixed could have caused an endless dispatch loop to occur when the notification bar was closed. It wasn't specific to the connection bar per-se, as it was actually in the child_process module, when it was trying to dispatch the event when the process exited.

If your 100% CPU was caused by this issue, it should be resolved in v0.5.15

[unguzov]

@krayon007 - I'm using notification bar too for all workstations.

[mjakovina]

@mjakovina , did you have the notification bar enabled?

@krayon007 I don't think so. Where is the setting for enabling it? (I do not know what is "notification bar :-) )

[mjakovina]

@krayon007 we will now update to v0.5.15 and see will it happen again

[mjakovina]

@krayon007 I just noticed the problem again on Windows Server 2008 R2 64bit

I notice that Meshcentral Agent is still version 0.2.1.3 - Is that OK?

[mjakovina]

Problem noticed today again on another server (also Windows Server 2008 R2 64bit)