Wappalyzer: UnhandledPromiseRejectionWarning: Error: The website took too long to respond

[smokeyflavor]

Platform
NPM

Describe the bug
This is a warning printed to the console in debug mode

To Reproduce
Run Wappalyzer via the CLI

Expected behavior
No warnings

Additional context

(node:741) UnhandledPromiseRejectionWarning: Error: The website took too long to respond
    at Timeout._onTimeout (/build/lib/driver.js:167:20)
    at listOnTimeout (internal/timers.js:551:17)
    at processTimers (internal/timers.js:494:7)
(node:741) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 130)
(node:741) UnhandledPromiseRejectionWarning: Error: The website took too long to respond
    at Timeout._onTimeout (/build/lib/driver.js:167:20)
    at listOnTimeout (internal/timers.js:551:17)
    at processTimers (internal/timers.js:494:7)
(node:741) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 131)

[AliasIO]

Are you using the latest version?

[smokeyflavor]

Yes. @AliasIO
Also, your timeout code is not working properly, look at the following snippet from driver.js:

const links = await Promise.race([
        this.timeout(),
        (
          await page.evaluateHandle(() =>
            Array.from(document.getElementsByTagName('a')).map(
              ({ hash, hostname, href, pathname, protocol, rel }) => ({
                hash,
                hostname,
                href,
                pathname,
                protocol,
                rel,
              })
            )
          )
        ).jsonValue(),
      ])

You are calling await page.evaluateHandle before invoking Promise.race, hence the whole timeout function is operating on .jsonValue() only, and not on the evaluateHandle part, which is crucial as well.

[AliasIO]

You're right, I'll fix that!

[smokeyflavor]

@AliasIO Would you consider incorporating p-timeout to cover these timeouts? https://www.npmjs.com/package/p-timeout
This would save time rewriting this code

[AliasIO]

The order just needs to be changed, I prefer as few dependencies as possible. Fixed in 476083fb30e0b416ee5b34ab41ade26b96b9ea53.

[smokeyflavor]

The new code works incredibly faster with the properly working timeouts. Thanks! @AliasIO

[smokeyflavor]

@AliasIO Is it possible to add timeout to .jsonValue() too? Since it's a promise as well, and might hang if the element is too big?
I'm currently running it on a few URLs and I see my process hanging for 2 days already. strace shows a hanging epoll_wait call but I figure it might be puppeteer or something. Anyways I think it's good practice to have all promises covered.

[AliasIO]

It seems unlikely that it would hang on that but I added it anyway!