Colly: Is it possible to stop a queue manually after calling Run?

[Davidonium]

I am trying to implement a pausable/resumable crawler. From the implementation that I have seen I believe that currently it is not possible to stop a queue after calling Run until it is empty or has 0 active requests as can be seen in https://github.com/gocolly/colly/blob/v2.1.0/queue/queue.go#L150.

I could implement this at the collector level by avoiding to call AddURL when needed but the crawler would keep going if the queue got big.

Has someone been able to implement this? Maybe I am missing something.

If not, is this a desirable behavior? Maybe it could be worth implementing

[asciimoo]

@Davidonium It isn't possible currently to achieve what you want, but it sounds like a useful addition. Could you work on it?

[Davidonium]

Certainly, I will open a PR when I get something presentable.

Thanks a lot for your work, this is an awesome library.

[lox]

A workaround that seems to work for me is calling request.Abort() in an OnRequest handler that checks if there was an error previously.

[PalanQu]

@asciimoo @lox Hi, I really want this feature and I can create a PR for this, but can you give some idea about how to implement this?
My idea is that add a method for Queue like queue.Stop(), but what's next, Thx.

[Davidonium]

My idea for this was as you said, add a Stop method to queue. Further I tought about adding a runnning property that defaults to true and in the Stop method set it to false.

This requires modifying this line https://github.com/gocolly/colly/blob/v2.1.0/queue/queue.go#L150 to be:

if size == 0 && active == 0 || !q.running {

Adding that would make the Run method return and stop the queue.

What I am not sure about is how to handle this runnning property, maybe it needs a lock, but I lack the experience determine if it is really needed or if that is a decent implementation.

[PalanQu]

@Davidonium Thank you for your idea, I will try to create a PR for this.

[PalanQu]

Hi @Davidonium , @asciimoo I have created a PR for this, please help to review this PR thx.