Chrome-remote-interface: Can't load specific urls

[hgfdd]

Hello, i'm trying to load the following urls:

1. https://nevnov.ru/487640-v-peterburge-otkrylas-vystavka-illyustracii-k-romanu-evgeniya-vodolazkina-lavr
2. https://nevnov.ru/488016-v-noch-alyh-parusov-v-peterburge-budut-kursirovat-avtobusy

and some other articles from that site. But loading via headless chrome is freezing, but in normal mode everything alright.

Sample code is:

const CDP = require('chrome-remote-interface');

CDP((client) => {
    // extract domains
    const {Network, Page} = client;
    // setup handlers
    Network.requestWillBeSent((params) => {
        console.log(params.request.url);
    });
    Page.loadEventFired(() => {
        console.log('Page loaded...');
        client.close();
    });
    // enable events then start!
    Promise.all([
        Network.enable(),
        Page.enable()
    ]).then(() => {
        return Page.navigate({url: 'https://nevnov.ru/488016-v-noch-alyh-parusov-v-peterburge-budut-kursirovat-avtobusy'});
    }).catch((err) => {
        console.error(`ERROR: ${err.message}`);
        client.close();
    });
}).on('error', (err) => {
    console.error('Cannot connect to remote endpoint:', err);
});

As i understand, some resources can't be loaded. How can i fix it?
Hope on your response.

[cyrus-and]

The freezing may happen if the Page.loadEventFired event is not fired, for example if some important resource never finishes loading, i.e., the server keeps the request pending, I'm not talking about errors. In this case a timeout is usually fired so the freezing should be temporary. In automated page loads it's a good idea though to explicitly set a custom timeout to avoid that a bougus site blocks the whole testbed, for example:

setTimeout(() => {
    client.close();
}, giveupSeconds * 1000);

Still it doesn't quite explain while it happens in headless mode only...

But I'm unable to reproduce your issue: I always get "Page loaded...". What version of Chrome are you using?

[hgfdd]

My chrome version 58.0.3029.110.

Sometimes it is working well, but most of the time it is not loading and i receive these errors:

[0628/165427.744930:ERROR:nss_ocsp.cc(591)] No URLRequestContext for NSS HTTP handler. host: ocsp2.globalsign.com
[0628/165427.744952:ERROR:nss_ocsp.cc(591)] No URLRequestContext for NSS HTTP handler. host: crl.globalsign.com
[0628/165427.760878:ERROR:nss_ocsp.cc(591)] No URLRequestContext for NSS HTTP handler. host: cert.int-x3.letsencrypt.org
[0628/165427.760973:ERROR:cert_verify_proc_nss.cc(918)] CERT_PKIXVerifyCert for ko6ka.ru failed err=-8179

Similar problems with these urls:

https://www.astrobl.ru/news/99343
http://www.fsvps.ru/fsvps/news/21741.html

Also, can i however forbid loading some types of resources, such as gifs?

[cyrus-and]

https://www.astrobl.ru/news/99343
http://www.fsvps.ru/fsvps/news/21741.html

OK, using these two URLs I can reproduce your issue with Chrome 59. Using version 60 instead it works as expected, still getting those errors, so probably they don't matter in this case:

[0628/161942.308782:ERROR:nss_ocsp.cc(613)] No URLRequestContext for NSS HTTP handler. host: cert.int-x3.letsencrypt.org
[0628/161942.309674:ERROR:cert_verify_proc_nss.cc(912)] CERT_PKIXVerifyCert for gosmonitor.ru failed err=-8179

Give Chrome 60 a try.

Also, can i however forbid loading some types of resources, such as gifs?

See #80, especially this comment.

[hgfdd]

When i am trying to launch the code from those comment, i get the following error:

(node:9878) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: 'Network.setBlockedURLs' wasn't found

node v6.10.2
chrome version 58.0.3029.110.

Code:

const CDP = require('chrome-remote-interface');
const fs = require('fs');
const blacklistURLs = [
  '*facebook*',
  '*fullstory*',
  '*getdrip*',
  '*olark*',
  '*optimizely.com*',
  'https://shift.com/images/*',
  '*segment.com*',
  // 'https://cdn.segment.com/analytics.js/v1/oD9PafeNRrJRbC3NL41R8DgV8SANLDZ9/analytics.min.js', - already skipped with wildcard '*segment.com*',
];
CDP.Version(function (err, info) {
  if (!err) {
    console.log(info);
  }
});
CDP(client => {
  // extract domains
  const { Network, Page } = client;
  // setup handlers

  Network.setBlockedURLs({ urls: blacklistURLs });

  Network.requestWillBeSent(params => {
    const url = params.request.url;
    console.log(`-> ${params.requestId} ${url.substring(0, 150)}`);
  });
  Network.loadingFailed(params => {
    console.log('*** loadingFailed: ', params.requestId);
  })
  Network.loadingFinished(params => {
    console.log('<-', params.requestId, params.encodedDataLength);
  })
  Page.loadEventFired(() => {
    console.log('loadEventFired!');

    Page.captureScreenshot().then(v => {
      let filename = `screenshot-${Date.now()}`;
      fs.writeFileSync(filename + '.png', v.data, 'base64');
      console.log(`Image saved as ${filename}.png`);
      client.close();
    });
  });
  // enable events then start!
  Promise.all([
    Network.enable(),
    Page.enable()
  ]).then(
    () => {
      return Page.navigate({ url: 'https://shift.com/cars/san-francisco' });
    },
    () => {
      console.log('REJECT');
      client.close();
    }
    );
}).on('error', (err) => {
  console.error('Cannot connect to remote endpoint:', err);
});

What can go wrong?

[cyrus-and]

I think Chrome 58 still uses Network.addBlockedURL instead of Network.setBlockedURLs. If you can't upgrade your Chrome version (recommended) you can obtain the documentation for Network.addBlockedURL with:

$ chrome-remote-interface inspect -r
>>> Network.addBlockedURL
{ [Function: override]
  category: 'command',
  [...],
  experimental: true }

[hgfdd]

When i do console.log(Network) i receive the list of functions and there is only setBlockedUrls and no addBlockedUrl. When i make chrome-remote-interface inspect -r there is only addBlockedUrl. When i use addBlockedUrl in my code i receive following error:

Network.addBlockedURL({url: 'https://yandex.ru'});
            ^
TypeError: Network.addBlockedURL is not a function

Hope on your response.

[cyrus-and]

That's because you either have to use the {remote: true} option or client.send (recommended) as described here.

[hgfdd]

Give Chrome 60 a try.

I have upgraded my Chrome to 60, but i am still have a lot of valid urls, that i can't load by headless mode. It's critical for my tasks. Is there any way to determine the reason of problem for specific url? For example, can i however watch the list of loading resources in the current moment? It will be usefull, when the program is closed by timeout.

Send you a short example list with problem urls:

• http://rcc.ru/article/kitayskie-proizvoditeli-shin-stolknulis-so-snizheniem-prodazh-60265#astart
• http://www.m24.ru/articles/147341
• http://www.m24.ru/articles/147342
• http://informing.ru/2017/07/25/samolet-an-24-ne-smog-vyletet-iz-yakutskogo-tiksi-iz-za-tehnicheskoy-nepoladki.html
• http://informing.ru/2017/07/25/den-vmf-2017-sankt-peterburg-sevastopol-i-vladivostok-vozglavili-reyting-samyh-populyarnyh-gorodov-v-den-voenno-morskogo-flota.html
• http://ru.delfi.lt/abroad/russia/memorial-soobschil-ob-ischeznuvshih-chechencah-kotoryh-predpolozhitelno-rasstrelyali.d?id=75307961&utm_source=rss&utm_medium=rss&utm_campaign=rss
• http://www.russland.news/irak-wuenscht-staerkere-zusammenarbeit-mit-russland/

Hope on your responce!

[cyrus-and]

Using HeadlessChrome/60.0.3112.78 I can load all of these URLs without problems. Are you running Chrome from a container?

For example, can i however watch the list of loading resources in the current moment?

You can listen for the following events:

• Network.requestWillBeSent
• Network.responseReceived
• Network.loadingFinished
• Network.loadingFailed

[hgfdd]

Using HeadlessChrome/60.0.3112.78 I can load all of these URLs without problems. Are you running > Chrome from a container?

Yes, i am using chrome inside lxc container. Is there any ways to improve/speed up the work of chrome?

[cyrus-and]

Oh that'd explain then, see this FAQ entry.