Would be nice to have an OnFinally handler that will be executed after both OnError and OnScraped handlers
icamys
All 3 comments
@icamys isn't OnScraped already your OnFinally handler? For reference if you look at the order of things: https://github.com/vendasta/web-crawler/blob/786b1eb775942f5dd8432c6c3b6015b1e89a9d87/vendor/github.com/gocolly/colly/colly.go#L551
OnRequest is functionally equivalent to --> OnInit
OnScraped is functionally equivalent to --> OnFinally
jredl-va
on 12 Mar 2019
@jredl-va My bad. I was returning a custom error in RedirectHandler and this made colly to stop working. After inspecting sources I found out that OnScraped is reachable after OnError. Thank you for your time!
icamys
on 13 Mar 2019
@jredl-va I got a question though: when I get "no such host" error, the OnScraped handlers are not called at all. Colly just stops execution here:
https://github.com/gocolly/colly/blob/master/colly.go#L612
That was the reason why I asked about OnFinally handler.
I think it would be convenient to have an opportunity to make some actions both after successful scrape and after error on response.
icamys
on 17 Mar 2019
Related issues
yanshiyason
路
7Comments
zplzpl
路
5Comments
x0rzkov
路
3Comments
soanni
路
6Comments
CrazyMouse
路
3Comments
Most helpful comment
@jredl-va I got a question though: when I get "no such host" error, the OnScraped handlers are not called at all. Colly just stops execution here:
https://github.com/gocolly/colly/blob/master/colly.go#L612
That was the reason why I asked about OnFinally handler.
I think it would be convenient to have an opportunity to make some actions both after successful scrape and after error on response.