Element-web: Jitsi screensharing doesn't work on desktop

[t3chguy]

given option to share everything or an application window, the latter sees no application windows and hitting share on either does nothing, video stream remains from webcam

[t3chguy]

maybe needs something like https://github.com/jitsi/jitsi-meet-electron/

[lampholder]

Not strictly speaking voip but it's the tag that's most likely to see this issue picked up by the right people.

[t3chguy]

Seems to still not be working even though PR https://github.com/matrix-org/matrix-react-sdk/pull/1355 has landed :(

[t3chguy]

[xmeng1]

+1

[t3chguy]

Now that I have an NVL hat and can see both sides of the issue I'll take a look at it

[t3chguy]

This seems to have even further regressed as this never fires: https://github.com/matrix-org/matrix-react-sdk/commit/53574541c3a480ee3fda37b642d2e8d1e82c5183#diff-0e2cb0e79e0d376b964d6d11e52c4c45R150

[MurzNN]

Seems that this is Chrome engine issue, because Jitsi Screen Sharing works well in Riot-web on Firefox, but in Chrome browser - needs separate extension.

[t3chguy]

That's not correct. It's a nested iframe issue as I have pointed out above. For electron the extension would not work as electron exposes screen sharing devices via a native api and not gUM, and you can run screensharing in chrome without extensions if you start it with a command line argument

[BloodyIron]

I'm trying to do desktop sharing, via jitsi channel widget, with the desktop riot client, and it never ends up presenting options to share. When I try to share whole screen, or per app, just spins circles. :(

[MurzNN]

Can this https://chrome.google.com/webstore/detail/riotim-screen-sharing/ehgcnaneidbjcmblghjepmamomchgahd extension been integrated in Electron app for solve this issue?

[t3chguy]

Nope. Electron has special screen sharing apis which that extension does not support.

[MurzNN]

Developers of Jitsi recommend use https://github.com/jitsi/jitsi-meet-electron-utils for solve this permission problem, did you try this way?

[t3chguy]

@MurzNN Yup that is already fitted but it doesn't work due to it being an iframe inside an iframe and it not getting passed through because of this.

[BloodyIron]

I really wish we could get this solved. Using riot for screen sharing would be seriously useful! :D

[MurzNN]

Seems permission problem can be fixed using new getDisplayMedia() API in Chromium v70 https://groups.google.com/forum/#!msg/discuss-webrtc/Uf0SrR4uxzk/uO8sLrWuEAAJ

[MurzNN]

Jisti Meet implement new Chrome API via this commit https://github.com/jitsi/jitsi-meet/issues/2233#issuecomment-441087489 so we can update Jitsi lib + Chromium in Riot electron build, and recheck permission problem.

[dbkr]

Just been looking at this after encountering the feature and how it's buggy as above. Agreed the way to go here is to let it use getDisplayMedia() once Electron is using a new enough chromium.

[ara4n]

@dbkr: do we magically get this for free after all the recent electron upgrades then?

[dbkr]

Chrome 70 is the magic milestone with getDisplayMedia. Electron 4.0 is chromium 69 - almost!

[MurzNN]

Seems update to Chrome 70 will not be done quickly - only in Electron 5.0 https://github.com/electron/electron/issues/15059 :(
Also maybe we need Chrome 72, as described in https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia:

Chrome 70 — 72
Notes: Available as a member of Navigator instead of MediaDevices in Chrome 70 and 71.

[MurzNN]

Together with screensharing, maybe we also can got Remote control feature to Riot, available in Jitsi Meet, too? https://github.com/jitsi/jitsi-meet/issues/1437

[BloodyIron]

Remote control as in during a conference? That could be super handy!

[akontsevich]

Yes, screensharing works in Chromium 70+. Whole screen sharing does not work in Firefox - only separate window (Jitsi has same problem). Why?

For Riot waiting for next Electron version with Chromium 70+?

[pinpox]

Just rebuild riot-web using the electron beta. No change, the screen-sharing window still loads for ever.
If you want to try you'll need this patch:

diff --git a/electron_app/src/electron-main.js b/electron_app/src/electron-main.js
index 99ddfbd1..c3aa4f5d 100644
--- a/electron_app/src/electron-main.js
+++ b/electron_app/src/electron-main.js
@@ -206,7 +206,10 @@ const launcher = new AutoLaunch({
 // work.
 // Also mark it as secure (ie. accessing resources from this
 // protocol and HTTPS won't trigger mixed content warnings).
-protocol.registerStandardSchemes(['vector'], {secure: true});
+//protocol.registerStandardSchemes(['vector'], {secure: true});
+protocol.registerSchemesAsPrivileged([{
+    scheme: 'vector', privileges: {standard: true, secure: true, supportFetchAPI: true},
+}]);

 app.on('ready', () => {
     if (argv['devtools']) {
diff --git a/electron_app/src/preload.js b/electron_app/src/preload.js
index 3a4f7c9a..f63e19d4 100644
--- a/electron_app/src/preload.js
+++ b/electron_app/src/preload.js
@@ -23,7 +23,7 @@ window.ipcRenderer = ipcRenderer;
 // protocol: this is necessary to load olm.wasm.
 // (Also mark it a secure although we've already
 // done this in the main process).
-webFrame.registerURLSchemeAsPrivileged('vector', {
+/*webFrame.registerURLSchemeAsPrivileged('vector', {
     secure: true,
     supportFetchAPI: true,
-});
+});*/

[t3chguy]

Jitsi would have to update to get rid of their electron special case most likely

[BloodyIron]

Since @akontsevich mentions it works in Chromium 70+ but not FIrefox, are we also testing for this as a desktop application? (Win/Mac/Lin). Namely because the web interface being able to do this is awesome, but I think it's also worth keeping in-mind the desktop app facet of it too.

I really hope we can get this sorted because this is something I and others around me would use very regularly!

I can't recall, are we just talking about this for rooms, or for 1on1's too?

[t3chguy]

Orr 1:1 the interface is different and you can't select what to share (unless the browser let's you like Firefox does) or less you fire up a jitsi widget but it does work. (shift click the video call button in a 1:1)

[pinpox]

To clarify: I was testing this with the desktop app on Linux (Arch).It works fine in Chromium (not in Firefox though)

[pinpox]

Are there any (working) alternatives for the jitsi video conference widget that support screen sharing?
I would consider using something else in matrix rooms as a workaround until this is fixed.

[MurzNN]

Electron 5.0 is released with Chromium 73 version, that have full support of getDisplayMedia! Can anybody try Riot with Electron 5.0 and check this issue?

[bytepoets-bkr]

i have tried Riot with Electron 5.0, but screen sharing still doesn't work. Has anybody else tried it?
Error log:

/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppleFSCompression/AppleFSCompression-96.200.3/Libraries/CompressData/CompressData.c:35: getxattr(/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist): Operation not permitted

[Bendodroid]

I am using the desktop riot app on Manjaro (Awesome wm) and screenshare does not work at all.

[ara4n]

@saghul do you know of anything which would prevent screensharing from working in Electron 5 on the jitsi side?

[saghul]

Last I checked, Electron 5 doesn't ship with a desktop picker UI. It does ship the getDisplayMedia API, but not a native desktop picker, that must be implemnted by the app for now, using Electron APIs, so from the usablity prespective, nothing has changed.

Thus, you should use jitsi-meet-electron-utils for enabling screen-sharing in the Jitsi iframe: https://github.com/jitsi/jitsi-meet-electron-utils#screen-sharing

[ChristianChiarulli]

I am using the desktop riot app on ubuntu and screensharing doesn't work for me.

[Mitch9378]

Last I checked, Electron 5 doesn't ship with a desktop picker UI. It does ship the getDisplayMedia API, but not a native desktop picker, that must be implemnted by the app for now, using Electron APIs, so from the usablity prespective, nothing has changed.

Thus, you should use jitsi-meet-electron-utils for enabling screen-sharing in the Jitsi iframe: https://github.com/jitsi/jitsi-meet-electron-utils#screen-sharing

@saghul see @t3chguy comment:

@MurzNN Yup that is already fitted but it doesn't work due to it being an iframe inside an iframe and it not getting passed through because of this.

Was getDisplayMedia supposed to help us work around this nested iframe issue?

I wan't to spend some time looking into this but the conversation seems a little fragmented and I am hoping to get us back on a path forward.

Thanks

[MurzNN]

As I understand we need to add something like this to Riot source:
```
const { RemoteControl } = require("jitsi-meet-electron-utils"); // iframe - the Jitsi Meet iframe
const remoteControl = new RemoteControl(iframe);

[t3chguy]

Remote control is entirely unrelated.

[BloodyIron]

Screensharing without remote input is half the functionality of screensharing by any other tool that does it. I disagree that remote control is unrelated.

[t3chguy]

What other instant messaging tool with screensharing offers remote control? I don't see it on slack, discord, Facebook, Skype, etc

[t3chguy]

This issue is not about remote control. I made the issue, I never mentioned remote control, don't hijack it for such.

[BloodyIron]

Skype does have remote control, as you can share control to other parties. I can't speak about Slack, Discord or Facebook, however I have used other screensharing tools that do allow "other user control" equivalent functionality.

I'm not hijacking it, I'm trying to point out that enabling others in a screen sharing situation to interactively control the system is actually commonplace and a relevant feature. Like many other GitHub Issue posts, not all contributions come from the original poster.

[t3chguy]

Of course, but riot is not primarily a screen sharing tool. This issue is about matching the functionality of the web client in the desktop client. Not adding new functionality

[Mitch9378]

Jitsi would have to update to get rid of their electron special case most likely

@t3chguy good catch

https://github.com/jitsi/lib-jitsi-meet/blob/9f123f34264bea52feb6efe25fa2f6a5876f1116/modules/RTC/ScreenObtainer.js#L163-L207

In Riots case it will fall into the electron code before every trying to do getDisplayMedia. They will be reluctant to remove this as many electron based based applications will not be on electron 5 for a while.

It seems though that if they just move the electron case before the getDisplayMedia case this would solve the issue and not break anyone.

Ie. If the browser supports getDisplayMedia use it regardless of if its electron or otherwise and then fall into the electron case if it doesn't support getDisplayMedia.

[Mitch9378]

Alright so so far I was able to build and run Riot on electron 5 using @binaryplease 's patch above and verified it reports that it supports getDisplayMedia.

I also modified and built jitsi-meet/lib-jits-meet and moved the electron case to a different spot. Were I am struggling a bit now is to get my Riot client to point to the local jitsi server but I am exploring that now.

It seems like the integrations_jitsi_widget_url in config.json should help with this

https://github.com/matrix-org/matrix-react-sdk/blob/2d6317fdd0303ec575e64588e5490606207a1d3f/src/CallHandler.js#L419-L425

but I haven't quite got that working yet.

[t3chguy]

You could try just make a custom widget and stick a url to your local jitsi in there, riot won't know its jitsi but it should still work

[Mitch9378]

That didn't quite work either as the custom widget didn't allow me to embed.

What I did do is the following:

I temporarily told the app to ignore cert errors (not great if connecting to outside matrix server):

app.commandLine.appendSwitch('ignore-certificate-errors', 'true');

I then pointed the src of the jitsi widget to my local server.

I was able to then start working with the following patch on lib-jitsi-meet:

diff --git a/modules/RTC/ScreenObtainer.js b/modules/RTC/ScreenObtainer.js
index 7a0dada3..1f730177 100644
--- a/modules/RTC/ScreenObtainer.js
+++ b/modules/RTC/ScreenObtainer.js
@@ -113,10 +113,15 @@ const ScreenObtainer = {
                     });
             };
         } else if (browser.isElectron()) {
+            if (browser.supportsGetDisplayMedia()) {
+                return this.obtainScreenFromGetDisplayMedia;
+            }
+
             return this.obtainScreenOnElectron;
+
         } else if (browser.isChrome() || browser.isOpera()) {
             if (browser.supportsGetDisplayMedia()
-                    && !options.desktopSharingChromeDisabled) {
+                && !options.desktopSharingChromeDisabled) {

                 return this.obtainScreenFromGetDisplayMedia;
             } else if (options.desktopSharingChromeDisabled
-- 
2.22.0

When I click the screen share the application is throwing a built in Jitsi exception saying I cancelled the screen picker window, although it never actually appeared:

https://github.com/jitsi/lib-jitsi-meet/blob/master/modules/RTC/ScreenObtainer.js#L305-L340

The jisi exception is masking the real error here so I will work on getting more information about that.

Initially I thought that it may be unhappy with the constraints that ScreenObtainer is trying to enforce so I removed them and simplified the code for debugging purposes and no change but things are still working in the browser directly.

The details are there but i'm too tired to take this further for now. Hopefully I will be able to provide some more information soon.

[saghul]

@Mitch9378 Thanks for your efforts here. As I said earlier, getDisplayMedia doesn't quite work on Electron 5 as you expect. You must pass the specific device (screen ID in this case) that you need to the getDisplayMedia call, but Jitsi won't do that. We rely on the fact that a native picker will show up and ask the user what they want to share. We "solved" this by implementing our own picker for Electron, using the Electron provided APIs for listing the available sources for desktop sharing.

I suggest you try to call navigator.mediaDevices.getDisplayMedia({ video: true }) in Electron Fiddle and see it for yourself.

[Mitch9378]

@saghul Thanks for the suggestion. It looks like at one point someone attempted to implement it and it was removed in a subsequent commit:

https://github.com/vector-im/riot-web/commit/19f1489c9284171e5ea3e69121864697a4e97e8a#diff-c62222a7b0c834636c8b72d6fcb782d5

by @dbkr in hopes that getDisplayMedia will solve this problem, but at present it does not.

To me it seems this violates the contract of getDisplayMedia at least as moz intended it and how its supported in chrome proper.

https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia

The MediaDevices interface's getDisplayMedia() method prompts the user to select and grant permission to capture the contents of a display or portion thereof (such as a window) as a MediaStream. The resulting stream can then be recorded using the MediaStream Recording API or transmitted as part of a WebRTC session.

For UI consistency I could see an argument why you wouldn't use chrome's picker in electron. Although electron has gained the support of getDisplayMedia in electron 5 by being built on newer chrome there is no electron specific documentation at the moment on how they intend it to work in electron if at all.

[saghul]

For UI consistency I could see an argument why you wouldn't use chrome's picker in electron. Although electron has gained the support of getDisplayMedia in electron 5 by being built on newer chrome there is no electron specific documentation at the moment on how they intend it to work in electron if at all.

While I don't disagree, that has been the status quo up until now. The way to do it is to use the desktopCapture Electron helper to build a picker yourself: https://electronjs.org/docs/api/desktop-capturer

With the above API you can build a picker because it provides you with all the necessary sources information.

getDisplayMedia doesn't allow for pasing a deviceId constraint as per: https://w3c.github.io/mediacapture-screen-share/#device-identifiers

So my conclusion at this point is that getDisplayMedia is unusable in Electron unless they either add a picker or a custom constraint so we can do the same thing as we used to with getUserMedia. But if they opt for the latter, we'd be in the same situation as we are today.

[MurzNN]

Can anybody test, maybe recently released Electron 6.0 have any fixes with screensharing using getDisplayMedia?

[saghul]

You can test it with Electron Fiddle and this code:

(async () => {
    let stream;
    try {
        stream = await navigator.mediaDevices.getDisplayMedia({ video: true });
        console.log(`g§ot a stream! ${stream}`);
    } catch (e) {
        console.error(`Oops! [${e.name}] ${e.message}`);
    }
})();

Spoiler: it doesn't work.

[MurzNN]

Does this problem described here https://github.com/electron/electron/issues/16513 or that is other issue? They provide this https://electronjs.org/docs/api/desktop-capturer as solution for screen sharing.

[saghul]

@MurzNN Yes. The "solution" is what jitsi-meet-electron-utils alredy does, which I mentioned before here: https://github.com/vector-im/riot-web/issues/4880#issuecomment-508365531

As per https://github.com/electron/electron/issues/16513 Electron doesn't seem to be keen on implementing the UI so I think waiting for it is a dead end.

IMHO this whole iframe within an iframe thing is the real source of the problem (and maybe future problems). I wonder if Riot on Electron could launch a separate window with the video call for example, avoiding the nested iframes thing. Note that you could be clever about it and hide this window when it goes out of focus, because we do provide an "always on top" mini-window.

[enannos]

Any possibility to have this issue solved? We are using the electron app for more than 200 people but we still have to run the browser version if someone wants to start f.i. a jitsi conference.

[nebulaw2081d]

I was referred to this issue from #10782. I am not sure if they are so different. It seems jitsi uses a different system from standard WebRTC.

[saghul]

@nebulaw2081d Absolutely not. Jitsi uses plain WebRTC. The problem is that it's loaded on an iframe within an iframe.

[biberino]

any updates on this issue?

[Mitch9378]

@biberino

I think @saghul has the right idea here:

@MurzNN Yes. The "solution" is what jitsi-meet-electron-utils alredy does, which I mentioned before here: #4880 (comment)

As per electron/electron#16513 Electron doesn't seem to be keen on implementing the UI so I think waiting for it is a dead end.

IMHO this whole iframe within an iframe thing is the real source of the problem (and maybe future problems). I wonder if Riot on Electron could launch a separate window with the video call for example, avoiding the nested iframes thing. Note that you could be clever about it and hide this window when it goes out of focus, because we do provide an "always on top" mini-window.

While this is likely the only long-term solution as it will reduce complexity it also calls in to question the long term sustainability and stability of the widget model in general in my opinion.

Potentially a short/medium term solution to this is to use bots like how jitsi integrates with slack:

https://jitsi.org/slack/

[t3chguy]

@Mitch9378 why not just use the pop-out button on the Jitsi widget if you want to instead of open it browser instead of forcing everyone to not use the integration and use a bot.

[Mitch9378]

@t3chguy I agree to an extent and for me that works fine. I am more concerned with general users having a bad experience and leaving the platform.

[akontsevich]

@t3chguy I agree to an extent and for me that works fine. I am more concerned with general users having a bad experience and leaving the platform.

We have bad experience for years.

[Mitch9378]

@akontsevich I agree but just because it has been bad doesn't mean it's ok for it to continue to be bad.

[RBNXI]

I have the same error in riot-desktop 1.5.12 (Manjaro), why this problem is not fixed already? is it that hard?

[t3chguy]

Yup, nested iframes make it hard to pass the native calls securely.

[MurzNN]

Maybe https://github.com/matrix-org/matrix-js-sdk/pull/1276 PR can help with solving this problem too?

[t3chguy]

@MurzNN nope, Jitsi doesn't use js-sdk. When Jitsi notices it is in Electron it tries to use electron's native API for getting the screens information which fails if you don't pass it through all the iframes using their native module.

[kebekus]

Same issue here. This is seriously a show-stopper for an otherwise phantastic app.

[kebekus]

Am 25.03.20 um 23:31 schrieb Networking Guy:
>

it looks like Jitsi team is working on it
https://community.jitsi.org/t/jitsi-meet-browser-warning-with-supported-browser/21379/2
since at least Jan. 20

This is very good news. Thank you for your wonderful work!

[saghul]

That work is not related to the problem at hand here.

[MurzNN]

When Jitsi notices it is in Electron it tries to use electron's native API for getting the screens information

As I understand, current Electron version miss native API for screen picker, so the solution is implementing own picker, here is example https://github.com/electron/electron/issues/16513#issuecomment-602070250 - can this be implemented from Riot side to solve current issue, or iframe-in-iframe blocks this way too?

[t3chguy]

This is not needed for Jitsi. Jitsi uses the native APIs and has its own picker for electron embedded already.

[MurzNN]

Jitsi uses the native APIs and has its own picker for electron embedded already.

Yes, Jitsi have own picker, but it shows empty list of screens&windows, so if we implement same picker on Riot side (outside iframe), maybe it will work right?

[t3chguy]

Nope, there is no way to tell Jitsi to use a specific media stream, it has to choose itself.

[MurzNN]

I have found https://github.com/wavebox/waveboxapp/issues/912#issuecomment-613882511 and check Riot-Web in Wavebox 10 app, and screen sharing in Jitsi works well! Can you lookup, is Wavebox work like regular Electron app like Riot-Electron, or based on other technology?

[t3chguy]

So nope, that behaves as an embedded chromium; its user agent is
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36
Which does not say electron like an electron app:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) RiotNightly/0.0.1-nightly.2020041401 Chrome/80.0.3987.134 Electron/8.0.3 Safari/537.36

[aral-matrix]

2020-05-05, freshly upgraded to Riot 1.6.0, both sandboxed (firejail) and non-sandboxed:
Shift-clicking the video-call button results in
Failed to get screen-sharing stream: NotFoundError: Requested device not found

[colalord]

I'm using the Riot.im desktop client on Windows.

Tried doing a shift+click on the video-call button and got this error:
Failed to get screen-sharing stream: NotAllowedError: Permission denied

Added the below switch to my Riot launcher.
--enable-usermedia-screen-capturing

Then I started getting the following error when trying to shift+click the video-call button:
Failed to get screen-sharing stream: NotFoundError: Requested device not found

Is there a fix for this?

Thanks everyone!

[servermancer]

Riot has been awesome, However while working with a Colleague we discovered that screen sharing is not functioning. We attempted to do the following on both of our systems and had identical experiences Windows for my Colleague and Ubuntu 20.04 for my self, both sessions riot-desktop version 1.6.0.

Tried initiating a Screen share call with the shift+ Click on the video call button and got this error:
Failed to get screen-sharing stream: NotAllowedError: Permission denied

Added the below switch to my Riot Launcher.
--enable-usermedia-screen-capturing

I then like my Colleague started getting the following error when trying to use the shift+click video call button:
Failed to get screen-sharing stream: NotFoundError: Requested device not found

Hopefully this will help in finding a solution.

Thanks for you hard work !