Chrome-remote-interface: How to wait for all ajax requests to be completed then log page output?

[nddipiazza]

Looking for some help doing with Chrome-Remote-Interface what a Google Chrome browser extension could do easily:

I wrote a Google Chrome browser extension that can count pending ajax requests https://bugs.chromium.org/p/chromedriver/issues/attachment?aid=330118&signed_aid=RkWFIDa8dBw0LLG860Kldg==

This allows a selenium-based web crawler a way to wait for certain ajax to be done loading before saving the HTML from Selenium.

But because of https://bugs.chromium.org/p/chromium/issues/detail?id=706008 we cannot use Browser Extensions in headless Chrome. So I can not use this option.

So I'm trying to achieve this same thing with Chrome-Remote-Interface.

Looking for some help writing a simple chrome-remote-interface node application that does the following:

• Navigates to a page
• Immediately listens to ajax requests incoming and outgoing (XMLHttpRequest) and starts to count pending number of ajax requests.
• When the pending ajax request count goes to 0, print out the page contents to the console.

Is this even going to be possible?

I found this link https://github.com/cyrus-and/chrome-remote-interface/issues/76#issuecomment-271531236 which roughly states it is possible to get the javascript-evaluated HTML contents of a page using CRI. But I'm not sure about listening for ajax requests...

Any pointers would be awesome. Thanks!

[cyrus-and]

You can simply listen for Network.requestWillBeSent and Network.loadingFinished (plus error handling), the requestId field allows you to associate them. The type field in the former event will be set to XHR for AJAX requests.

Hope it helps.

[nddipiazza]

Here's what I've got with that advise:

const CDP = require('chrome-remote-interface');

var requestCounterMinWaitMs = 15000;
var requestCounterMaxWaitMs = 20000;
var theUrl = 'http://localhost:7001/page500.html';

var numSent = 0;
var numReceived = 0;
var startTime = new Date().getTime();

function minWaitTimeExceeded() {
  return new Date().getTime() - startTime > requestCounterMinWaitMs;
}

function maxWaitTimeExceeded() {
  return new Date().getTime() - startTime > requestCounterMaxWaitMs;
}

CDP((client) => {
    // extract domains
    const {Network, Page} = client;
    // setup handlers
    Network.requestWillBeSent((params) => {
      if (params.type == 'XHR') {
        console.log('Sent ' + params.type);  
        ++numSent;
      }
    });
    Network.responseReceived((params) => {
      if (params.type == 'XHR') {
        console.log('Recieved ' + params.type);
        ++numReceived;
      }
    });
    Page.loadEventFired(() => {
      var ajaxDoneInterval = setInterval(function() {
        if (numSent == numReceived && minWaitTimeExceeded()) {
          clearInterval(ajaxDoneInterval);
          var pageContents;
          client.Runtime.evaluate({expression: 'document.documentElement.outerHTML'}, (error, result) => {
              pageContents = result.result.value;
              console.log(pageContents);
              client.close();
          });
        } else if (maxWaitTimeExceeded()) {
          console.log('Timed out. Waited ' + (new Date().getTime() - startTime) + ' ms. Had been still waiting for ' + (numSent - numReceived) + ' ajax requests');
          client.Runtime.evaluate({expression: 'document.documentElement.outerHTML'}, (error, result) => {
              pageContents = result.result.value;
              console.log(pageContents);
              client.close();
          });
        } else {
          if (numSent == numReceived) {
            console.log('No pending ajax requests, but still waiting for minWaitTime of ' + requestCounterMinWaitMs + '. Current wait: ' + (new Date().getTime() - startTime));
          } else {
            console.log('Still waiting for ' + (numSent - numReceived) + ' ajax requests');
          }
        }
      }, 500);
    });
    // enable events then start!
    Promise.all([
        Network.enable(),
        Page.enable()
    ]).then(() => {
        return Page.navigate({url: theUrl});
    }).catch((err) => {
        console.error(err);
        client.close();
    });
}).on('error', (err) => {
    // cannot connect to the remote endpoint
    console.error(err);
});

see any issues with this?

[cyrus-and]

I'd simply use an auxiliary data structure: requestId -> AJAX info and avoid the 500ms setInterval polling. But I guess you're on the right track.

[nddipiazza]

Your suggestion was worked. thanks closing.

[hbakhtiyor]

@cyrus-and can't understand how to do without setInterval here

[nddipiazza]

I just ended up leaving this almost exactly as I have it. But I used requestId's instead of counters. works

[hbakhtiyor]

mine not work, even if requestId list is empty, maybe request will be sent

[cyrus-and]

@hbakhtiyor

can't understand how to do without setInterval here

That was simply a suggestion, anyway the idea is to set and clear a timeout in the Network.* events directly.

mine not work

Feel free to provide the code so I can take a look.

[hbakhtiyor]

@cyrus-and this snippet, even if requestld list is empty, for about 1-2seconds some requests will be fire

[cyrus-and]

@hbakhtiyor the program output would help. Anyway before requestCounterMinWaitMs the snippet just waits for requests even if there are no pending requests.