Chrome-remote-interface: Can't avoid a clean slate on a page reload when a promise gets collected

[fracasula]

Hi there,

So this is a slightly complicated scenario where I do not know whether the website I'm crawling will trigger a page reload or not. This code is part of an API that gets a URL to crawl and some JS to execute on that page. If the target page triggers a reload the idea is to wait and then execute the given JS on the new page.

| Component | Version
|-|-
| Operating system | node:8 image
| Node.js | v8.1.0
| Chrome/Chromium/... | Google Chrome 59.0.3071.115
| chrome-remote-interface | 0.23.3

Is Chrome running in a container? YES

const CDP = require('chrome-remote-interface');

const evaluate = async (Runtime, wait, callerName) => {
    try {
        let data = await Runtime.evaluate({
            expression: `new Promise(resolve => {
                setTimeout(() => {
                    resolve(document.documentElement.outerHTML);
                }, ${wait});
            });`,
            awaitPromise: true
        });

        console.log(data);
    } catch (e) {
        console.log(callerName, e);
    }
};

CDP(async (client) => {
    const {Network, Page, Runtime} = client;

    try {
        await Page.enable();
        await Network.enable();

        const onReload = async () => {
            await evaluate(Runtime, 1000, 'RELOAD');
        };

        client.once('Page.loadEventFired', onReload);
        await Page.navigate({url: 'https://www.behance.net/stainmyth'});

        await evaluate(Runtime, 5000, 'MAIN');
    } catch (err) {
        console.error('Some other error:', err);
    }

    client.close();
}).on('error', (err) => {
    console.error('CDP error', err);
});

This basically only works if I don't restart Chrome before every execution of the above script.
I tried playing with the waiting times as well and only managed to get a "Promise was collected" both on 'MAIN' and 'RELOAD'.

To get it to work avoiding a Chrome restart and bringing the 'MAIN' waiting time to 0 is enough.

Anyway, to me it looks like it may not be possible because the "Promise was collected" of the first page maybe is forcing Chrome to lose whatever was set thus we start again on the reload.

Any thoughts? Thanks!

[cyrus-and]

You have to be careful about the following points in order to avoid race conditions and errors:

• after Page.navigate you have no guarantees about the state of the page so it makes little sense to inject JavaScript at that time;
• if a re/load happens when you have a pending Promise in the browser context, such Promise will fail (collected).

Said that, and assuming that you want to execute some (asynchronous; using setTimeout for the sake of the demo) JavaScript once the page is loaded regardless of whether it's the first, the second or whatever page reload you can do something like the following. Of course, if a page load happens before that the asynchronous computation completes the whole computation is thrown away and a new one is executed in the newly loaded page.

const CDP = require('chrome-remote-interface');

async function evaluate(Runtime) {
    const {result} = await Runtime.evaluate({
        expression: `new Promise(resolve => {
            setTimeout(() => {
                resolve(document.documentElement.outerHTML);
            }, 3000); // XXX simulate long running/async operation...
        });`,
        awaitPromise: true
    });
    return result.value;
};

CDP(async (client) => {
    const {Network, Page, Runtime} = client;

    try {
        await Page.enable();
        await Network.enable();

        Page.loadEventFired(async () => {
            try {
                const data = await evaluate(Runtime);
                console.log(`COMPLETED: ${data.slice(0, 50)}...`);
                client.close();
            } catch (err) {
                console.error('INTERRUPTED BY REALOD: waiting for page load event...');
            }
        });

        await Page.navigate({url: 'https://www.behance.net/stainmyth'});
    } catch (err) {
        console.error('Some other error:', err);
    }
}).on('error', (err) => {
    console.error('CDP error', err);
});

[fracasula]

I didn't even need different stages since everything in this case can be handled by the Page.loadEventFired.

It makes a lot of sense, thanks again for your help. 👍