Colly: Scraping a website that requires you to scroll down?

[Aracki]

Is there a way to force the colly.Colector to scroll down the page in order to get additional data?

[zvodd]

TL:DR - No, read some tutorials or books on webscraping and/or check out http://phantomjs.org / http://seleniumhq.org for automating a complete web browser.
But yes, if you know what you're doing.

Colly interacts with websites by way of sending HTTP requests for specific resources. Websites that scroll are usually(always?) implemented through JavaScript detecting the scroll interaction and making additional HTTP requests (aka AJAX, XHR) and finally translating received responses back into the page - by manipulating the original HTML document with the JavaScript DOM API inside the clients browser.

Colly doesn't "resolved"/render the HTML DOM to the same degree as a web browser, in a few ways, but mainly it doesn't interpret and run JavaScript. So there is no way for scroll events to happen or acted upon.

To use Colly to scrape these types of lazy loaded resources, you need to manually identify the pattern of those HTTP resources - the URLs and how they are embedded in / called from the initial page.

Its usually in practice quite easy to do, try opening the browser's inspector panel -> network tab and watching what resources are loaded. Often there's a simple pattern to the URLs that load additional content e.g. example.com/somepage?currentPartOfPage=2 . Other times you have to reverse engineer complicated JS and you're better off using headless browser automation i.e. Selenium or WebDriver

[Aracki]

Thanks for a very profound answer. 👍