Puppeteer: startScreencast feature?

[ghost]

Would be great to screencast the page.

[Garbee]

Why screencast a headless display? If you want to see it, simply don't start it in headless mode.

[ghost]

Good point. In other words if there is a need to screencast page - one have to use Chromium without headless mode.

On the other hand puppeteer supports non-headless mode too. Would be great that Puppetier would support this: https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-startScreencast

[nigeon]

+1

[aslushnikov]

@Mykolas-Molis how do you want to consume it? Do you want to save a screencast as a video file?

[michaelshiel]

I've implemented this for myself over here: https://github.com/michaelshiel/puppeteer (Page emits an event for each frame) but I'm not sure what kind of tests would be appropriate, or how to describe/structure them, any suggestions?

[sul4bh]

@aslushnikov I would like to have this feature implemented too. My use case is recording the screen as a video.

[paulirish]

+1 to video capture start/stop/save API

[sul4bh]

Not sure where to ask this question but here it goes:
What does the document here(https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-startScreencast) mean by a frame? What is the maximum frame-rate that I might be able to get by using startScreencast()?

I experimented with startScreencast() but I was only able to get about 5fps (meaning 5 png images/frames in 1 second). On headless mode I got only 1 fps. This is a very low fps for video.

[michaelshiel]

I think it depends on how much the page is changing, in my experience it will stop sending frames if there are no changes on the page. I tried it on a youtube video, and I was getting at least 20 frames per second.

[sul4bh]

@michaelshiel you are right. I played a 25fps youtube video and got an average of 25 "frames" per second. EDIT: Can't verify if the frames are generated only when the content of the screen changes. EDIT: Verified

Also I got really high fps today. Not sure why it was low yesterday. Probably Chrome Canary got updated or something...

Also, it would be nice to have a Viewport option for startScreencast(). (not sure where to request for a feature)

[KeizerDev]

Check this issue: https://github.com/ChromeDevTools/devtools-protocol/issues/17

[imolorhe]

+1 for screencast video saving.

[aslushnikov]

Everybody: as a workaround, tracing could be recorded with screenshots embedded:

await page.tracing.start({path: 'trace.json', screenshots: true});
await page.goto('https://www.google.com');
await page.tracing.stop();

If the trace.json file is dropped into the devtools performance panel, there will be screenshots embedded into the timeline.

[matthiasg]

Related to this, I would love to start a WebRTC session from puppeteer but how do I opt-in allowing access to the webcam/video/microphone streams ? When starting with headless:false it prompts the user of course and then works fine.

My use case is sharing a connected capture card to a different device.

[mpseidel]

I've also experimented with the startScreencast feature to turn animations into a video. But I would need to record every frame (60fps). Is this realistic to expect from such an api or can you point me into another direction?

I also tried recording every frame by taking a screenshot but that takes forever...

[cretz]

@mpseidel - Not sure if this issue is a good place for general requests, but I have done some experiments around this. I used the screen capture ability for extensions and recorded my findings here: https://github.com/cretz/chrome-screen-rec-poc/tree/master/attempt1. It was too slow for me (I have another example in there that uses D3D on Windows)...I suspect the screen frame events will be too slow too but have not tried them.

[mpseidel]

@cretz thanks! I'll have a look

[XBeg9]

+1

[purch-FE]

Is it possible please to merge https://github.com/michaelshiel/puppeteer fork with his screencast.js ?
even if i did not understood why i get this issue using his code :

~$ node screencast.js
(node:13449) UnhandledPromiseRejectionWarning: TypeError: page.startScreencast is not a function
    at /home/bitnami/screencast.js:39:12
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:118:7)
(node:13449) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:13449) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[ebidel]

@purch-FE That solution produces image frames, not a video file. You can already get screenshot frames in a trace. https://github.com/GoogleChrome/puppeteer/issues/478#issuecomment-341623663

[jameskeane]

@ebidel @aslushnikov Tracing only supports capturing a maximum of 450 frames: https://github.com/chromium/chromium/blob/master/content/browser/devtools/devtools_frame_trace_recorder.h#L35

So far I've found tracing to be the most reliable, as it doesn't depend on how fast you can ack; but this is a serious limitation.

[ghost]

If this feature does end up getting implemented, I would very much like it to have the ability to specify a target framerate for the page, and to also specify that the page should never drop a frame.

In other words, I'd want to be able to opt-out of realtime rendering, and instead go into a consistent, deterministic rendering mode which will always render the same thing every time, even if that means slowing down to wait for longer frames.

I have two use cases for this feature:

1. I create broadcast (as in, television) graphics using web technologies, and I want to have full video regression testing of all my animations. This requires that the video rendering process be completely deterministic, and that it never drop a frame.
2. This would let me also create pre-rendered versions of some of my dynamic graphics, so that they could be loaded as video files into vision mixers (the big broadcast switchboards with all the fun buttons and levers on them), in the few rare cases where this is the preferred workflow for my clients.

EDIT: I'd also ideally like to have an alpha channel in the output video, though I know that dramatically increases filesize requirements.

[mpseidel]

@Lange That would be killer. Not sure if it should be done through a screencast api but if you can capture frames I would also would like to see an option for deterministic fixed framerate capturing as fast as possible to render a video file from html5 animations.

[matthiasg]

Starting a webcam session might be even more interesting. That way one could very simply set up an WebRTC Server broadcasting a Capture Device to another device (which would be non headless)

[muralikg]

Was able to use getUserMedia API and do

1. screen capture and export as a .webm video file
2. Specify bitrate

Currently have the following issues

1. Capture stops when page navigation occurs reloading the page.
2. How to get rid of the user permission dialog. Any flags to bypass the user dialog?

pardon me for the nested code as I am stuck with a an older version of nodejs

const puppeteer           = require('puppeteer');
var options = {
  headless: false,
  args: [
    // '--disable-extensions-except=/path/to/extension/',
    // '--load-extension=/path/to/extension/',
    '--enable-usermedia-screen-capturing',
    '--allow-http-screen-capture',
    '--no-sandbox',
    // '--start-fullscreen',
    // '--user-data-dir=/tmp/chromedata'
  ],
  executablePath: 'google-chrome-unstable',
}
puppeteer.launch(options).then(browser=>{
    return browser.pages().then(pages=>{
        var page = pages[0];
        return page.goto('https://news.ycombinator.com/', {waitUntil: 'networkidle2'}).then(_=>{
            page.evaluate(()=>{
                var session = {
                    audio: false,
                    video: {
                        mandatory: {
                            chromeMediaSource: 'screen',
                        },
                        optional: []
                    },
                };
                navigator.webkitGetUserMedia(session, (stream)=>{
                    var chunks=[], recorder = new MediaRecorder(stream, {
                        videoBitsPerSecond: 250000,
                        ignoreMutedMedia: true,
                        mimeType: 'video/webm'
                    });
                    recorder.ondataavailable = function (event) {
                        if (event.data.size > 0) {
                            chunks.push(event.data);
                        }
                    };

                    recorder.onstop = function () {
                        var superBuffer = new Blob(chunks, {
                            type: 'video/webm'
                        });

                        var url = URL.createObjectURL(superBuffer);
                        var a = document.createElement('a');
                        document.body.appendChild(a);
                        a.style = 'display: none';
                        a.href = url;
                        a.download = 'test.webm';
                        a.click();
                    }

                    recorder.start();

                    setTimeout(()=>{
                        recorder.stop()
                    }, 5000)
                }, (args)=>{
                    console.log(args)
                });

            })
        })
    }).then(_=>{
        // return browser.close()
    })
})

UPDATE:
Full working example using xvfb https://github.com/muralikg/puppetcam

[maxweber]

+1

[cretz]

@muralikg, see the link I pasted earlier: https://github.com/cretz/chrome-screen-rec-poc/tree/master/attempt1. It uses capture for a tab instead of a screen and uses --auto-select-desktop-capture-source with a certain value. Combined with an extension, this providesa no-interaction way to capture the tab's video.

[muralikg]

@cretz Thanks a lot for the pointers that solves all the problems I had with recording. I tested it on http://tobiasahlin.com/spinkit/ and all the css animations were captured well enough.

As an aside, were you trying to capture video elements in your plugin ? If so you could try captureStream

[matthiasg]

How can the permission dialog be circumvented ?

[ebidel]

@matthiasg you can follow https://github.com/GoogleChrome/puppeteer/issues/846

[ebidel]

Another request for this: https://twitter.com/ldenoue/status/994663644725202945

[matthiasg]

@ebidel thanks ... good input there. I had tried re-using a profile but that did not work at first , building that from standard chrome though I hadn't

[joelgriffith]

Might be nice to show a roadmap on the readme for this project, or at least mark these issues with milestones so folks can (roughly) know when to expect this to drop.

I'm not sure how much resources Google's throwing at this, but some visibility of where things are headed would be nice for consumers. I'd be more than happy to help in any way possible. Also probably avoid some of the +1's and BUMP's in these types of issues.

[sadaka]

@muralikg, I tried to avoid user permission for screen share as suggested by @cretz. I am still getting user permissions popup. As you are able to achieve it, please can you help me to resolve it?
Thanks!

[muralikg]

@sadaka here is a gist . Update the stop logic as necessary, currently it records for 10 seconds and then stops

[sadaka]

@muralikg Thanks for quick reply. It worked for me, document.title='pickme' is doing the magic using extension. My use case is, I am running a online class featuring screen sharing and video streaming and sharing. I need to save whole class session at backend server. So, I need to run it in headless mode. I am trying to run it in headless mode, please if you have a solution, suggest me.

[sadaka]

Chrome is not supporting extension in headless mode #659 . So, is it possible to record screen in head less mode in any other browser?

[HiroyukTamura]

@sadaka I think xvfb is currently the only way...

[joelgriffith]

Nah, it's baked into the devtools protocol: https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-startScreencast

You _could_ use puppeteer's https://github.com/GoogleChrome/puppeteer/blob/v1.4.0/docs/api.md#cdpsessionsendmethod-params to do this manually and, essentially, implement the functionality yourself. If I'm to guess I'd say puppeteer will eventually have an API for this, just not sure when and what that would look like.

[J-F-Liu]

If the page is playing animation using css or canvas, we may need to convert the animation (including audio) into a video file using Puppeteer.

[aimerib]

I had a very specific need to capture images of pages scrolling for documentation purposes, and generating an animated gif with the results. If anyone has a similar need, check out my repo: https://github.com/aimerib/puppeteer-gif-cast

[jasonellington]

Can someone clearly layout the options for a video recording of a script in headless... seems like there are a bunch of answers but no consensus or agreement

[jasonellington]

@sul4bh how did you implementstartScreencast()

[kristijanmatic]

@Garbee it's about being able to record whole transitions or screens to display the UX experience after testing, to always have up to date short videos of features. usages could vary, but automatically creating videos from a headless browser is a plus.

[DuBistKomisch]

I've brought @michaelshiel's changes up to date here: https://github.com/viviedu/puppeteer/tree/screencast

seems to work pretty well

[henriquebol]

I tried to run the example (screencast.js), but I'm getting the error:

(node: 20382) UnhandledPromiseRejectionWarning: TypeError: page.startScreencast is not a function

I have a checkout of the Puppeteer repo (branch screencast) and have run npm i.

Am I doing something wrong? Do I need to implement startScreencast()?

[DuBistKomisch]

The examples use require('puppeteer');, you'll have to edit that to point to your checkout of it, I guess it's picking up a global install by default or something like that. require('../index'); should do the trick.

[tnolet]

@DuBistKomisch I got your solution working on a local Puppeteer fork. Seems to do the trick! Reconstituting the separate images into a .gif or mp4 is really expensive though.

Using Image Magick on a c5.2xlarge AWS instance (8 cores) takes about 30 seconds for a fairly short Puppeteer session (i.e. around 12 seconds) that generates roughly 150 jpegs. Of course the lower the resolution, the faster it goes.

# running in a directory with ~150 jpegs with an average size of 140kb
time convert -resize 640x -delay 10 -loop 0 `ls -1 *.jpg | sort -n` myimage.gif

real    0m28.608s
user    1m42.944s
sys 0m5.960s

Unless I'm missing something obvious, I guess this is where the bottleneck is for using this method at any type of scale.

[DuBistKomisch]

Yes, the flaw with this "screencast" is that chrome encoding it into JPEGs is a massive waste of time if you want to do anything besides store it as is. The WebRTC solution mentioned earlier might be more suitable since they encode to a video directly. As for the re-encoding efficiency, you might have more luck with ffmpeg or gstreamer, which could even get hardware acceleration, however much that helps.

[muralikg]

Here is an example to automate the process of exporting videos using MediaRecorder API and xvfb
https://github.com/muralikg/puppetcam

Thanks to @cretz

[sadaka]

@muralikg ,

I have setup a complete automated solution to record 'live classes' on my servers. I have used xvfb, headless chrome and node.
Everything works fine and session gets recorded. I have two problems I hope you can provide me some solution for that

1. webm files generated takes too long to convert in mp4 using ffmpeg
2. some times recoding do not starts and fails without any errors
   Please suggest me what should I do, I have used basic concept from your code as discussed earlier.
   Thanks!

[muralikg]

@sadaka ,

1. If you need the output in mp4 you could try capturing from x11grab device and export as mp4 and avoid the conversion. https://trac.ffmpeg.org/wiki/Capture/Desktop

1. I am yet to use my method in production and will update here if I find similar observations. Make sure there are no errors in the background script of the chrome plugin too as that will go unnoticed.

[sadaka]

@muralikg ,
Thanks for prompt reply!!
I have to complete this process without human intervention at backend on servers. So, I think I can not use ffmpeg for it. I have placed error logs in background script and just waiting for this error to come again.

Thanks!

[jsilvia721]

+1 on this. I would really benefit from having a screencast API where you can set the captured FPS so that I can later encode the frames into a video with a set framerate.

[clonn]

+1 that is very helpful, we can record continue status from client view and review again.

[fcsonline]

I was really interested in this feature for my pupperter session and I was able to record it with this workaround:

const page = await browser.newPage();

let frameNum = 0;

const interval = setInterval(async () => {
  await page.screenshot({path: `frame${String(frameNum).padStart(5, '0')}.png`});
  frameNum++;
}, 100);

... your session goes here

clearInterval(interval);
await browser.close();

And then:

ffmpeg -r 4 -i frame%5d.png -pix_fmt yuv420p -r 10 output.mp4

If this is interesting for someone, we can transform it to an API.

[sanbornhilland]

@fcsonline This is really interesting. I was trying to do something similar based on this PhantomJS example here. Is it possible to pipe this straight into ffmpeg? I tried simply writing to /dev/stdout as in the above example but I am getting UnhandledPromiseRejectionWarning: Error: ESPIPE: invalid seek, write

[fcsonline]

@sanbornhilland I know that ffmpeg supports to pipe in videos but I not sure about frames.

[mpseidel]

I had some success recording 60fps video using https://github.com/muralikg/puppetcam by @muralikg running inside a Docker container. Still having issues running it in Azure but looks promising!

[Bkucera]

@mpseidel I believe that will not work for headless, since there are no chrome extensions in headless mode

[mpseidel]

@Bkucera I'm not running in headless mode but using xvfb https://github.com/muralikg/puppetcam/blob/master/export.js#L7

[jimkang]

I'd like to be able to capture just audio for generative sound projects written for WebAudio. Is there a separate issue for that?

[joelgriffith]

Was able to get this integrated into browserless -- it's definitely tricky getting this up and working inside of docker if you go with route of writing an extension (as @Bkucera mentioned). You'll also need to be running docker using the overlay2 engine storage driver (otherwise the built-in CryptoToken extension has issues running). Sandboxing can throw a wrench into things as well.

There are docs here on to accomplish this, and an example cURL call:

curl -X POST \
  https://chrome.browserless.io/screencast \
  -H 'Content-Type: application/javascript' \
  -o video.webm \
  -d 'module.exports = async ({ page }) => {
    await page.goto('\''https://codepen.io/davidkpiano/embed/BGxgLa/?height=275&theme-id=0&default-tab=result'\'');
    await page.waitFor(5000);
};'

[EitanGoldfrad]

Timesnap and Timecut solved it for me.

Not the fastest way but good frame rate results.

[PhoenixHe-msft]

I've figured out some way to capture video. The solution is something like https://github.com/michaelshiel/puppeteer , but you can steal page._client as devtools client, then send Page.startScreencast, then listen to Page.screencastFrame, then pipe the output to ffmpeg with

const ffmpegArgs = fps => [
  '-y',
  '-f', 'image2pipe',
  '-r', `${+fps}`,
  '-i', '-',
  '-c:v', 'libx265',
  '-preset', 'fast',
  '-tune', 'zerolatency',
  '-movflags', '+faststart',
  '-crf', '20'
];

The only problem is that devtools may not report each frame at the desired frame rate. I want to make some subtitles to match the video so I need to store the time of each frame and recalculate the mapped time point in the video.

[skortchmark9]

Has anybody had any success with Chrome's getDisplayMedia api? It seems like the only trick would be running puppeteer with the experimental flags and programmatically passing through the "what to look at" dialogue.

[quartexNOR]

This feature is vital. It allows us to run X webapps and stream the results back, thus taking advantage of more cores on embedded devices -- while the result is presented on a final page.
Probably one of the most important additions to Chrome in years!

[auchenberg]

Adding another use-case for a more performant screecasting API, as I've built vscode-browser-preview that enables a browser instance inside VS Code using Chrome Headless. Perf is the biggest feedback item from users, and I'd love to be able to have a WebRTC stream or alike.

Demo:

https://github.com/auchenberg/vscode-browser-preview/

[jokin]

I had some success recording 60fps video using https://github.com/muralikg/puppetcam by @muralikg running inside a Docker container. Still having issues running it in Azure but looks promising!

could you share your dockerfile?

[mpseidel]

@jokin I dropped this a while ago because I had issues with xvfb - I'll see if I still have a partly working version to share.

[TomasHubelbauer]

This thread is pretty long, so maybe I've missed some ways, but I've compared the three ways I know of: tracing, screenshot loop and screencast using CDP and the code is available here for anyone who wants to take a look:
https://github.com/TomasHubelbauer/node-puppeteer-apng/tree/master/test
I generate animated PNGs from the captured frames with accurate frame timings and in my experiment the best solution is to use Puppeteer to get a CDP session and use startScreencast. Both screenshot loop and tracing are much laggier.

[c9fe]

If anyone's still interested in this feature, I have a popular project that does just this (albeit using MJPEG, not MPEG-4):

https://github.com/dosycorp/Joystick

You can hook into your headless browser and view and control it like a regular browser.

The headless browser can be local or remote, and you can tweak things like when and how often frames are sent.

[skortchmark9]

I took a stab at this with puppeteer at latest and got stuck. Posting my progress since I had to do a fair bit of spelunking.

1.) Page.startScreencast tops out around 10fps, ignoring the everyNthFrame parameter. I saw some stuff around responding with session.send('Page.screencastFrameAck', { sessionId }); but it didn't make a difference. The API is obviously still experimental, so hopefully these are just bugs that will get ironed out.

2.) navigator.mediaDevices.getDisplayMedia(); I can't find a way to get past the dialogue programmatically - I tried the '--auto-select-desktop-capture-source="record"' flag, but I think it only works with extension API. It seems like an oversight to me to only offer this workaround for the extension and not for the native API.

3.) Puppetcam (mentioned above https://github.com/puppeteer/puppeteer/issues/478#issuecomment-424958588 ) works decently but since it uses the extension API it won't work headlessly.

I didn't try the xvfb route yet - at first, it felt a bit unclean. Ultimately for regression testing, xvfb and a separate screen recorder might actually be more faithful than doing an all-chrome solution: the video recording won't interfere with chrome's processing, and we don't have to deal with headless chrome quirks.

[c9fe]

DevTools has a grant permission method. you could use that to give permission for WebRTC

https://chromedevtools.github.io/devtools-protocol/tot/Browser#method-grantPermissions

[auchenberg]

@skortchmark9 Interesting strategy of using navigator.mediaDevices.getDisplayMedia(). Is your thinking to use WebRTC between the browser and a Node "WebRTC client'?

[andreabisello]

recording the video of a test execution is vital to debug complex use cases when a test fails.

for now i wrapped every puppeteer api in a high level api that use puppeteer screenshot api to take a screenshot of the website when the test fails,

but only because i'm not able to record a video.

[remote-specialist]

please, add screencast feature - we really need it!

[taewookim]

Any updates?

[Kashuab]

+1. Need this in order to see screencasts of failed integration tests executed in GitLab CI.

[andreabisello]

+1. Need this in order to see screencasts of failed integration tests executed in GitLab CI.

True. now i wrapped every api call inside a function that make a page.screenshot in order to obtain a "movie" , but i'm also considering cypress.io that has the screencast working. maybe puppeteer developers can takes some hints from cypress.io

[tnolet]

For those interested, we are working on a screencast beta for Puppeteer (and also Playwright) over at https://checklyhq.com.

We are in the prototyping and design phase, but the goal is to roll it out to our SaaS first and then release the implementation To the community as an open source “plugin” for both frameworks.

Would love to learn your use cases. Mail me at Tim (at) checklyhq.com.

P.S. we are also behind Puppeteer Recorder. Some of you might know that one.

/highjack promo mode off

[pan-alexey]

I created PR for solve the problem https://github.com/chromium/chromium/pull/61

[card-b]

Looks like this feature is hard locked to 10fps. This file in the Chromium source: https://chromium.googlesource.com/chromium/src/+/refs/heads/master/content/browser/devtools/devtools_video_consumer.cc has the variable kDefaultMinCapturePeriod set to 100ms which is never overridden anywhere. It looks as though we might be able to set that capture period to something smaller (either 32 or 16) to get a faster framerate if somewhere in page_handler.cc we called:

video_consumer_->SetMinCapturePeriod(16)

or

video_consumer_->SetMinCapturePeriod(base::TimeDelta::FromMilliseconds(16))

but as far as I can see, there's nothing currently available to set the capture period to anything other than the default. The reason for this default isn't immediately apparent to me, it may be that the renderer itself has a hard cap at 10fps and this is intended to clamp the capture period down to the renderer's max frame rate in headless.

[Aaronphy]

I've figured out some way to capture video. The solution is something like michaelshiel/puppeteer , but you can steal page._client as devtools client, then send Page.startScreencast, then listen to Page.screencastFrame, then pipe the output to ffmpeg with

const ffmpegArgs = fps => [
  '-y',
  '-f', 'image2pipe',
  '-r', `${+fps}`,
  '-i', '-',
  '-c:v', 'libx265',
  '-preset', 'fast',
  '-tune', 'zerolatency',
  '-movflags', '+faststart',
  '-crf', '20'
];

The only problem is that devtools may not report each frame at the desired frame rate. I want to make some subtitles to match the video so I need to store the time of each frame and recalculate the mapped time point in the video.

good