Colly: Stop to collecting data

[dzakaammar]

Sorry, I'm a newbie in Go and want to build a web scraper based on it.
How did you do to stop the collecting process without exiting the application? I couldn't find any related function or method in the doc. Thanks.

[asciimoo]

Colly stops if you don't perform further requests and the ongoing requests are finished.
It can be achieved multiple ways. It is done automatically, if you have fixed number of requests, or you can introduce a variable which indicates stop.

The following example stops the execution if an error occurs:

c := colly.NewCollector()
stop := false

c.OnError(func(_ *colly.Response, _ error) {
    stop = true
})

for !stop {
    c.Visit("your url")
}

[saifat29]

I am using Async in Colly and want to stop my crawler after it satisfies a specific condition. I referred issue #26 and #109 but it is not working.

Following is my code -

c := colly.NewCollector(
    colly.MaxDepth(2),
    colly.Async(true),
)

c.Limit(&colly.LimitRule{
    DomainGlob:  "*",
    Parallelism: 10,
})

I iterate over a slice of base URLs and Visit them

for _, url := range targetURLs.URLs {
    if stopScan {
        return
    }
    c.Visit(helpers.GetValidURL(url))
    c.Wait()
}

And then I Visit URLs found on those base URLs

c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    if stopScan {
        return
    }
    link := e.Attr("href")
    e.Request.Visit(link)
})

How to stop Colly running in Async mode?

[karust]

Another way to exit crawl is to call panic and recover

func CrawlSite(urlSite string, saveto string, config CollyConfig) {
    defer func() {
        if r := recover(); r != nil {
            fmt.Println("exit crawl")
        }
    }()
        c := cly.NewCollector()

    // SOME CODE

    c.OnResponse(func(r *cly.Response) {
                // SOME CODE
        if (elapsed > waitTime) || (downloaded >= config.MaxAmount) {
            panic("Exit")
        }
                // SOME CODE
    })
        // SOME CODE
    c.Visit(url)
}