Cordova-plugin-iosrtc: Capturing a static image from the local MediaStream
Hi Iñaki, first I'd like to thank you for this great plugin. I'm currently building a Cordova-based video-chat solution using OpenTok.js, Crosswalk and your plugin and most of the things are working out great. I'll happily share the project once it's done.
Now to the issue at hand: I need to be able to capture static images from the MediaStream of the local client before the stream gets sent to the remote client and therefore gets compressed and loses quality. The stream needs to continue while capturing the static images.
The common WebRTC solution is to draw the content of the <video> element onto a <canvas> element and use <canvas>.toDataURL('image/png') to get the static image. However, due to the specifics of iosrtc, this is obviously not possible.
My guess is that this requires some sort of native implementation. While my Swift/Objective-C skills are limited, I'll happily give it a shot and create a pull-request if you can point me into the right direction. Or maybe you already have a solution at hand.
Many thanks!
oliverlukesch
All 51 comments
There is no easy way to do that, and given that this plugin mimics the official WebRTC API I don't want a custom API to capture images from the video. The way to go should be adding <canvas> support to the plugin with all the Canvas API, but that would be an horrible work.
I'm sorry, I cannot point you in the right direction :)
ibc
on 4 Jan 2016
Thanks for the update, I understand your approach. And yes, implementing the <canvas> API would be quite a different kind of beast :) I'll start writing on a patch for this functionality I can use in my own project.
oliverlukesch
on 4 Jan 2016
@ibc If you were to add an internal API to grab the image, I can work on integrating it into canvas so you can use the same code you would in a browser
contra
on 5 Jan 2016
@contra I'm so sorry, but currently I cannot get the time to work on that.
ibc
on 5 Jan 2016
@ibc Can you leave the issue open so somebody (ie me) can work on it?
contra
on 6 Jan 2016
Sure.
ibc
on 6 Jan 2016
@oliverlukesch I had to do the same in my application and I've extended the plugin on JS and Swift side.
What you can do: Take a snapshot of the UIView element. You need the ID of the mediastreamrenderer
In cordova-plugin.iosrtc.js do (for example) the following trick
var keys = Object.keys(mediaStreamRenderers);
id = mediaStreamRenderers[keys[0]].id;
The ID contains your specific view element. It's mapped to the Swift dictionary
which contains the specific UIView.
Do a local snapshot in Swift with this code:
// Create the UIImage
UIGraphicsBeginImageContextWithOptions(view.bounds.size, true, 0.0)
view.drawViewHierarchyInRect(view.bounds, afterScreenUpdates: true)
view.layer.renderInContext(UIGraphicsGetCurrentContext()!)
let image = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
var imageData = UIImagePNGRepresentation(image)
and return it to the function as base64.
This should do the trick
Regards (Grüße nach Berlin)
Rolf
derMani
on 13 Jan 2016
Hey Rolf, that's great, many thanks for sharing! Think I'll have time for implementing this solution next week, will keep you updated.
Grüße nach Osnabrück :)
oliverlukesch
on 14 Jan 2016
@derMani @oliverlukesch Awesome!
1N50MN14
on 14 Jan 2016
Guys, this is exactly what I am missing. The ability to write the current stream to a canvas element. I want to use the camera as motion detector (see this super simple and awesome example: https://github.com/ReallyGood/js-motion-detection). If you manage to get this to work that would be awesome!
Grüße aus Stuttgart
(Seems like the Germans are taking over – again)
gefangenimnetz
on 14 Jan 2016
@gefangenimnetz Pull Requests are welcome.
ibc
on 14 Jan 2016
@derMani I would be interested to capture the image, but the only "limit" I see on your method is that the resolution of the image will depend on the size of the UIView. So this means that you get a 320 or 640 HxW image... Am I wrong?
AntonioCurci
on 21 Jan 2016
@SirSeymour The resolution will depend on what you decide to use with the plugin, from there you can use the base64 image data to resize, crop etc.
Anyone on this thread _actually_ working on a PR? @oliverlukesch are you intending to send in a PR?
1N50MN14
on 22 Jan 2016
@1N50MN14 if the local video UIView is 240x240, you'll get a 240x240 image. Don't you? I'm saying if I want to capture the full resolution image from local camera, with the method stated above I can't (ex. the video source is set a 1280x960 but the UIView is at 640x320, the image I'll get is the 640x320)
AntonioCurci
on 22 Jan 2016
@SirSeymour You're actually right this won't serve say as a 12M full image resolution capture, but it's good for stuff such as "chose your profile photo"
1N50MN14
on 22 Jan 2016
Anyway I just need to know if anyone on this thread is working on a PR so I could decide whether to write it myself or not as "implementing the solution" is not clear if translates to submitting a PR
1N50MN14
on 22 Jan 2016
At least for me, I am not working on a PR.
Please note that the Screenshot-thing is kind of a heavy task (at least, that's what the documentation says)
For anything live / canvas related, it might be better to utilize somehow a AVCaptureConnection and AVCaptureStillImageOutput on the video connection.
derMani
on 22 Jan 2016
Ok cool thanks @derMani I'll squeeze this on my to-do list next Friday and submit a PR @SirSeymour I'll try to generate a full resolution image I'll have to see how it'll work out in relation to the constrains properties the plugin receives
1N50MN14
on 22 Jan 2016
@1N50MN14 Nope, haven't had any time to touch this issue so far. Great stuff you're taking a look!
oliverlukesch
on 22 Jan 2016
I am currently working with iosRTC plugin and i can test PR when available.
ikostic
on 23 Jan 2016
@1N50MN14 Hi, i am just wondering if you had any luck creating the solution. DId you made any progress? Many thanks.
ikostic
on 3 Feb 2016
@ikostic Apologies for not updating this, unfortunately I lost interest in this project / plugin, I'll no longer be participating here and will be focusing on my own implementation instead; I have some catching up to do but once I get there I can revisit the repo and send a PR if no one has implemented this until then... sorry about that.
1N50MN14
on 3 Feb 2016
@1N50MN14 No problem, i understand. I will try to find some solution.
ikostic
on 5 Feb 2016
Is there any chance that someone will make this possible with iosrtc?
scottmahr
on 13 Jun 2016
@scottmahr I'm sending a PR for this one on Friday
1N50MN14
on 13 Jun 2016
Wow, that is amazing timing. In your implementation what sort of format will the image be coming out from the library? Right now I use canvas and canvas.toDataURL to pull a jpeg of the current video output.
Let me know if you want help testing to see if it works on different devices.
Thanks!
Scott
scottmahr
on 13 Jun 2016
@scottmahr Similar to toDataURL you'll receive base64 data back, the only caveat is that you're receiving back a "screenshot" having whatever dimensions the device has, not an actual frame from the video
1N50MN14
on 13 Jun 2016
libwebrtc doesn't offer any capabilities for extracting frame, this is pretty much the best that can be accomplished atm unfortunately..
1N50MN14
on 13 Jun 2016
Does that mean that the video would need to be full screen on the device when you wanted to record an image from it? I guess it could be full screen but not displayed? My goal would be to capture video frames in the background with them being on the screen. It isn't a deal breaker though, I could make it work even if the video had to be shown.
Thanks
scottmahr
on 13 Jun 2016
You should be able to accomplish that if you patch PR #179 as well then you can place a tint above the full screen video view to "hide it" or even change the HTML's background opacity from 0 to 1
1N50MN14
on 13 Jun 2016
Hey @1N50MN14, any update on the PR you were going to submit on this? I am still excited to try it out.
Thanks!
Scott
scottmahr
on 5 Jul 2016
@scottmahr Apologies for the delay been extremely preoccupied haven't forgotten about this, I'll try to get to it before the weekend because I will it myself as well
1N50MN14
on 5 Jul 2016
@scottmahr I just created the patch https://gist.github.com/1N50MN14/80562300ec92eab4dfd9f65d819764e1 haven't had time to send a PR, give it a go and let me know
1N50MN14
on 8 Jul 2016
@1N50MN14 hum, that seems to take a screenshot of the webview, right?but what about the video elements that can be positioned on top or behind it? At any rate I guess the desire here is to take a screenshot of a given video stream, not the entire screen.
saghul
on 9 Jul 2016
@saghul Yes that only takes a screenshot if the video view (at the device's dimensions not the vide frame) - doesn't consider anything else. Ideally we should be able to forward the stream as an ArrayBuffer so we could use it in a Canvas element and drop the "video view" all together
1N50MN14
on 9 Jul 2016
I was just about to investigate the patch by @1N50MN14 but I see that there's just been a comment from @SomewhatCloudy saying it captures everything _except_ the video.
danielearwicker
on 17 Aug 2016
You might have more joy with it than me! I am sure the theory is sound. Let me know if you have any success on it.
SomewhatCloudy
on 17 Aug 2016
Unfortunately after I added the plugin to my project it won't build (compiler error, a missing type). But if I figure out why that is and get it working I'll let you know!
danielearwicker
on 17 Aug 2016
anyone has already got this working?
cometta
on 24 Jun 2017
No.
ibc
on 24 Jun 2017
hthetiot
on 13 Sep 2019
Done here:
Example:
var canvasEl;
function TestMediaRenderCatpure(videoEl) {
canvasEl = document.createElement('canvas');
var ctx = canvasEl.getContext("2d");
canvasEl.width = "100";
canvasEl.height = "100";
canvasEl.style.position = 'absolute';
canvasEl.style.left = 0;
canvasEl.style.bottom = 0;
appContainer.appendChild(canvasEl);
var image = new Image();
image.onload = function() {
ctx.drawImage(image, 0, 0);
};
videoEl.render.save(function (data) {
image.src = "data:image/jpg;base64," + data;
});
}
Test Source:
Implementation:
hthetiot
on 17 Sep 2019
Can the video element be hidden? I only want to show the frames into the canvas. Is this possible?
alexp25
on 30 Aug 2020
6.0.15 support CanvasRenderingContext2D.drawImage on VideoElement with iosrtc MediaStream
https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/03e4a0db4bfbb5de14b613f978620f79322a105d
// Apply CanvasRenderingContext2D.drawImage monkey patch
var drawImage = CanvasRenderingContext2D.prototype.drawImage;
CanvasRenderingContext2D.prototype.drawImage = function (arg) {
var args = Array.prototype.slice.call(arguments);
var context = this;
if (arg instanceof HTMLVideoElement && arg.render) {
arg.render.save(function (data) {
var img = new window.Image();
img.addEventListener("load", function () {
args.splice(0, 1, img);
drawImage.apply(context, args);
});
img.setAttribute("src", "data:image/jpg;base64," + data);
});
} else {
return drawImage.apply(context, args);
}
I just tried that but in Safari it gives me Type Error on "drawImage.apply(context, args);".
Any idea why?
In the debugger args seems correct: it's an array and the element 0 is a string "data:image...."
gabrielenosso
on 28 Oct 2020
@gefangenimnetz This is not made for Safari but for cordova, the img is not a string but an image tag loaded, also I have no idea if Safari support draw on video.
If you want really help with this issue please test with Cordova, you need to pass the proper argument (videoEl, 0, 0) but since you did not provided any details on what you exactly called it's impossible to know what to expect.
This script above is not made for Safari,
hthetiot
on 28 Oct 2020
I am using Cordova, debugging the WKWebView with Safari.
Sorry, my message was not clear.
This is my code:
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({ video: true, audio: false })
.then((stream) => {
const video = this.videoRef.current; // This is the <video> element, taken from the DOM via React
video.onloadedmetadata = (event) => {
this.canvasRef.current.width = event.srcElement.videoWidth;
this.canvasRef.current.height = event.srcElement.videoHeight;
};
video.srcObject = stream;
const onFrame = () => {
window.requestAnimationFrame(onFrame);
const canvasContext = this.canvasRef.current.getContext('2d');
canvasContext.drawImage(video, 0, 0);
};
onFrame();
})
.catch((error) => {
console.log(error);
});
}
@gabrielenosso looking into it one moment.
hthetiot
on 28 Oct 2020
To complete the information: I am using version 6.0.15.
Changing the line:
drawImage.apply(context, args);
To:
context.drawImage(img, 0 0);
seems to work fine.
EDIT:
I think I found the issue:
In your comment the line is:
args.splice(0, 1, img); // this works fine
But in the repository code the line is:
args.splice(0, 1, img.src); // error
gabrielenosso
on 28 Oct 2020
@gabrielenosso First, this issue is originaly closed.
Second, yes, I did pushed the fix on master already of img vs img.src.
Please comment on #582 not this issue.
Thank you.
Note: Regarding the background thread warning, I have a fix but it also create another error and also requestAnimationFrame will not be efficient with the current implementation.
hthetiot
on 28 Oct 2020
582 will be closed on 6.0.16 with the img fix. its on master already.
hthetiot
on 28 Oct 2020
Related issues
derMani
·
9Comments
smuddy
·
15Comments
ashish-mw
·
11Comments
derMani
·
15Comments
magestican
·
11Comments
Most helpful comment
Done here:
Example:
Test Source:
Implementation: