Wappalyzer: Best approach for analyzing multiple url's with the same wappalyzer instance?

[AdityaAnand1]

Right now I have to

const wappalyzer = await new Wappalyzer(website, options).analyze();

For each website. If I am iterating through thousands of websites, then that means I'm leaving behind thousands of chrome processes just lying around after each loop. Which eventually take over all memory and require killing all chrome processes.

Is there a way around this? I would like to create one wappalyzer instance with one underlying pupeteer instance and iterate through a list of websites being analyzed one at a time.

[AliasIO]

The memory should be released when the process ends, if that's not the case then there is a memory leak somewhere. I do want to look into switching to puppeteer-cluster to handle parallel jobs better.

[AdityaAnand1]

The memory should be released when the process ends

Yes, that happens, but while the process is still running, it keeps accumulating, going up and up until my ssh session freezes and I have to do a restart from the VPS provider. Since, I'm routinely scanning hundreds of thousands of websites, I actually don't want the process to ever end.

I do want to look into switching to puppeteer-cluster to handle parallel jobs better

That would actually be fantastic, a built in way to handle parallelization would make this much more powerful, allowing a single instance to get way more throughput. Although, instead of a cluster, why not just open a bunch of pages in the same browser instance (tabs) and manage their lifecycle?

[mzpqnxow]

I don't think this will help you since you are calling Wappalyzer directly from code, though others may be interested to know that I have very good results by using GNU parallel to do KISS parallelism on the command-line. On a 16 core machine I'm able to do 16 concurrent processes at once without issue (I haven't tried higher) though the load does get near 80-90% across all of the cores. I assume all of the regex and the matching engine in general is expensive in terms of CPU. The memory footprint is pretty light however, so RAM is less important.

[AliasIO]

I created a new driver that lets you run multiple instances of Wappalyzer using a single browser. This will replace the existing NPM driver. Please give it a try and let me know what you find, it should perform much better.

https://github.com/AliasIO/wappalyzer/tree/master/src/drivers/puppeteer

[AdityaAnand1]

Awesome. Are there some setup instructions somewhere or should I try via pulling the latest git repository?

Maybe we can also release it on npmjs with a beta version tag?

[AliasIO]

For now, just do a pull and take the example from the README. You can also run node ./src/drivers/puppeteer/cli.js https://example.com

[AdityaAnand1]

So far so good! Working much better, haven't run into any issues, churning through hundreds of URLs with several simultaneous wappalyzer processes. It's exiting gracefully as well, no ghost processes.

Green light for release from me 👍 This one seems way more stable.

[AliasIO]

This is now in v6.0.0. The README has an example on how to process multiple URLs in parallel.

https://www.npmjs.com/package/wappalyzer