Meshcentral: Can't see a remote pointer (cursor) of remote user

[GusevVictor]

Hi all!

Step to reproduce:

1. Use 0.4.9-k version
2. Connect to a remote user and help him (Win 7/10)
3. User will be view how you move the cursor
4. When you not move and remote user wants to show you something you can't view a remote mouse pointer (cursor) just see a result of a click, like open or close windows and so on.

That is not good in sutiation where you want to understand which steps remote user do.

Best regards, Viktor.

[krayon007]

Am I right to assume the remote computer is windows?

I ask because on Linux I tied into an X event to be notified of cursor changes... But on windows I haven't found such an event, so rather than polling for it, I update it whenever I remote inject a cursor movement.

I'll see if I can try to find a win32 event...

[GusevVictor]

Am I right to assume the remote computer is windows?

Oh yes..! Sorry.

I'll see if I can try to find a win32 event...

It will be good if you can fix that.

[MailYouLater]

You indicated in another issue a while back that you hide the cursor when capturing the screen to reduce the amount of image data that needs to be transferred. Couldn't we have an option to not hide the cursor when capturing the screen?

[krayon007]

Yeah, I was pondering that last night. I can't move the pointer on the browser side, so the only real solution is to enable rendering the cursor as part of the bitmap. I meant to add a feature to allow a flag to enable that. Once I get that working, I can make it so if the local mouse moves independent of the remote input, to render the cursor in the bitmap, which would solve the issue in this thread.

[ryanblenis]

Just to add to this while you're working on it, as it is likely related, there is still some "flickering" of the mouse pointer on the system being controlled. (I actually just tested it, one connection looked like there was no flickering; disconnecting and reconnecting showed the flickering again, though mostly when moving the mouse or performing a click-drag. I figured I'd mention it as it's likely closely related to the work you're doing here, but is distracting enough to the local users to have them say something about it every time!

Thanks Brian!

[MailYouLater]

In the other issue I was referring Brian mentioned that the reason it flickers is because it's hiding the cursor to take a screenshot without the cursor, assuming my understanding of that is correct, then not hiding the cursor would fix the flickering as well as making the cursor visible at the cost of a bit of extra data being transferred for every frame where the mouse moves. Personally, I think that's a good trade, but like many other features, it might be a good idea to make it optional, and in this particular case, it might be good to make it something that can be changed on the fly, similar to how there's a checkbox for Input, perhaps there could be a checkbox to show the remote cursor.

[krayon007]

I'm not actually doing anything explicit to hide the cursor. The API I'm using to capture the screen does not contain the mouse cursor, because that is rendered on a different plane.

The flickering (at least before) was caused when the child process that was scraping the screen didn't die, so that when a KVM was connected, there were multiple processes scraping the display. If you are still experiencing flickering can you verify how many processes are running? You should see 3, if you have the connection bar enabled. One for the main agent, one for the kvm child, and one for the connection bar, if it's active. I haven't noticed any flicking on my system, but i will certainly take a close look if you are seeing lingering processes...

[ryanblenis]

Thanks for the explanation of how it's working!

I'm not using the connection bar, but have 2 MeshAgent processes running (as SYSTEM), and can see flickering. It's a lot more subtle and less annoying than it was many releases ago, but it is still there.

(Win10 Pro 1909 client)

[MailYouLater]

I'm not actually doing anything explicit to hide the cursor.

Oh, well I guess there was miscommunication in that other thread, because it sounded like you were saying that you were. That's too bad, I was hoping it would be as simple as changing an API call or two. 🙁

The API I'm using to capture the screen does not contain the mouse cursor, because that is rendered on a different plane.

Is it possible to capture the 'plane' the cursor is on too? if so, would it be a separate image that must be combined, or can you capture the combined image? ... or is it more reasonable to capture data about the cursor's location and appearance and add it to the bitmap?

[krayon007]

I already capture the cursor bitmap, in that other plane, so I have the bitmap all by itself separate from the desktop. I just got code working, to get events from Windows when the cursor location changes, and when the bitmap changes. I had to spin up a message pump on my KVM process, and register some hooks, but I got it working. I just have to piece it together, becuase right now the events I get are on a different thread from the KVM processing loop, so I need to write more code to manage all that correctly...

So the first step, will be to modify the KVM code, so it will only grab the cursor bitmap, when I get this event, rather then every time the mouse moves (This will hopefully ease the flickering). The second step will be to modify it some more so that I can use the location event, to render the cursor bitmap directly onto the KVM bitmaps sent to the browser. If this works, then I can modify it some more, to make it so that the cursor is only rendered onto the bitmap if a flag is set to enable that mode of operation. Then when this works, I can modify it further, so if the mouse cursor location changes without any remote input, it will set this flag, render the cursor, then clear the flag, allowing the browser to see remote mouse movements.

[MailYouLater]

Awesome! Am I blind? or have you not pushed these changes to the MeshAgent repo yet? Either way, I'm really looking forward to seeing these changes. 😄

[krayon007]

No I haven't pushed the changes to the repo yet, because I'm not done writing the code yet ;)

[GusevVictor]

No I haven't pushed the changes to the repo yet, because I'm not done writing the code yet ;)

Thank a lot. We waiting you.

[krayon007]

OK, I pushed an update to the git repo that supports remote mouse cursor rendering on Windows. (as well as event based cursor updates). Ylian is back from vacation tomorrow, so we'll do some extensive testing. In the coming days, I'll work on adding this functionality to linux and freebsd. MacOS will take a while, because the mac KVM code is completely different from linux/freebsd, so I'll have to investigate how to get cursor events and such.

[krayon007]

I made the logic, such that it will always remote render the mouse cursor if it has been more than 500ms since a mouse move injection when a mouse move event is received. That way, if you move the mouse on the browser, it will not render the mouse cursor in the bitmap... But if the user on the local machine moves the mouse, it will render the mouse cursor on the bitmap.

[krayon007]

By the way, the current behavior, is that if you locally move the mouse it will continue rendering the cursor icon until you stop moving the mouse, then the cursor disappears from the KVM display on the browser, until either the remote or local user moves the mouse. If this is an issue, I can modify the code. Right now, I'm overlaying the cursor on the bitmap when a mouse move occurs. If there is no mouse movement, in order to render the cursor, I'll have to continually paint the cursor on every frame grab, which I'm not doing currently.

[Ylianst]

Just published MeshCentral v0.4.9-m with Bryan's all new MeshAgents including on Windows his work on displaying the remote cursor. I give it a try and it works nicely.

[PathfinderNetworks]

Thank you for this! I just updated my server 0.4.9-o. Being able to see the remote cursor is a huge benefit when trying to do remote support with an EU. They just assume we can see where they are moving the mouse. Now I actually can. So far it seems to be working correctly from my limited testing.
Thanks!

[GusevVictor]

Works!

[MailYouLater]

@krayon007: Thanks so much for this. It'll prove very helpful.
I do have a bit of an issue with the current implementation because the cursor disappears basically instantaneously when it stops moving. I wish it would stay, even if only for a second or so.

[PathfinderNetworks]

@krayon007: Thanks so much for this. It'll prove very helpful.
I do have a bit of an issue with the current implementation because the cursor disappears basically instantaneously when it stops moving. I wish it would stay, even if only for a second or so.

I've noticed the same thing with the mouse disappearing like that as well. I wasn't sure if I was seeing things or what was going on.

[MailYouLater]

Yeah. Seems weird, but Bryan did state that this was expected behavior with the current implementation:

By the way, the current behavior, is that if you locally move the mouse it will continue rendering the cursor icon until you stop moving the mouse, then the cursor disappears from the KVM display on the browser, until either the remote or local user moves the mouse. If this is an issue, I can modify the code. Right now, I'm overlaying the cursor on the bitmap when a mouse move occurs. If there is no mouse movement, in order to render the cursor, I'll have to continually paint the cursor on every frame grab, which I'm not doing currently.

Personally, I'd prefer that it work the way TeamViewer does, where it either always shows, or it never shows, depending on whether the "Show remote cursor" option is enabled or disabled.

[krayon007]

When I get into the office tomorrow, I'll tweak the behavior. There already is a setting to always enable remote cursor rendering. But that flag currently only renders the cursor on mouse movement (remote or local). I'll tweak it, to always render the cursor regardless of movement...

I'll also try to tweak the normal behavior so that it'll keep rendering the cursor until the mouse movement is remote injected. Meaning, I'll modify the behavior so the remote cursor begins rendering when the local user moves the mouse, and it'll keep rendering until the remote user moves the mouse.

[MailYouLater]

That sounds great!