Colly: Get proxy URL for request

[ghost]

I'd like to know which proxy URL was used with which request so that on errors I can automatically remove proxies that return repeated errors or timeouts. Ideally it would be a property, like Request.ProxyURL. It doesn't look like there's any way to do that currently, though.

I'm happy to contribute a PR, but I'm not seeing an obvious way to get at that except by keeping a map[string]string ledger which is less than ideal. Thoughts?

[asciimoo]

@matthew-noken yes, this is definitely a valid use-case and currently it isn't possible to retrieve the proxy URL in callbacks.
I'd extend the Request struct with a Proxy() string member function, because Request is accessible from all the callbacks. What do you think about this solution?

[ghost]

I think that's good, but I'm trying to figure out how to get at the proxy URL for a particular request.

The call flow looks like this:

1. User passes a ProxyFunc (such as the return value of proxy.RoundRobinProxySwitcher()) to colly.Collector.SetProxyFunc()
2. colly.Collector.SetProxyFunc() assigns ProxyFunc to http.Transport.Proxy
3. http.Transport.RoundTrip() calls http.Transport.connectMethodForRequest()
4. http.Transport.connectMethodForRequest() calls http.Transport.Proxy() (our ProxyFunc).

The return value of connectMethodForRequest(), though—the proxy URL itself—is ephemeral. It lives and dies in http.Transport.RoundTrip().

Ultimately the proxy URL used needs to be assigned to a Request somewhere, so the question is where? As I see it, because the request is passed to the ProxyFunc, the best option is to embed http.Request in colly.Request and pass it to http.Client directly. Add a ProxyURL property to colly.Request. Modify proxy.roundRobinSwitcher.GetProxy to set colly.Request.ProxyURL (and change the name to something like SelectProxyURL, since it has side effects). This can retain public interface backwards compatibility.

I can create a PR if this sounds good.

[asciimoo]

@matthew-noken you are right, this solution seems valid to me. I'd appreciate a PR. Thanks!

[ghost]

(I've been super busy, but I do still plan to do this in the next 2-3 weeks.)

[webzak]

It is possible to set additional request header in ProxyFunc like:

func ProxyFunc(r *http.Request) (*url.URL, error) {
        proxy := "socks5://127.0.0.1:1777"
    r.Header["proxy"] = []string{proxy}
    return url.Parse(proxy)
}

Then it can be taken back in event handlers via Response: r.Request.Headers.
For sure it's not the best solution, as we send this header to client.

[ghost]

Right, that is an acceptable solution if you don't mind that the receiver knows about the proxy.

[llonchj]

@webzak what about using colly.Context?

[webzak]

Well it seems like possible too, but even less obvious. I've decided to go with extending of http.Transport wrapping the RoundTrip() method and customising Proxy func property. It is totally transparent solution and can be used with any http.Client..

[rainbowhat]

Sorry to hijack , what if the proxy is not responsive, does it skip the job or try another proxy on the list

[ghost]

@rainbowhat The call times out.

I still haven't had any time to do this yet, unfortunately.

[rainbowhat]

@matthew-noken ok, noted with thanks!

[jjhesk]

@matthew-noken pls submit this PR.. thanks

[vosmith]

Merged #222 but leaving this open until we can support all ProxyFuncs by capturing the proxy in the http client as mentioned earlier https://github.com/gocolly/colly/issues/110#issuecomment-371517557

[mazhigali]

I think you should close the issue, because it is done in https://github.com/gocolly/colly/blob/master/request.go#L52

[ghost]

Given that this satisfies the original use case I described (accessing the proxy URL in OnError, then handling as appropriate), I agree with @mazhigali. I'm closing the ticket. If you'd like to support deeper functionality, I think it would be best to create a new ticket that's more targeted to the specific need.

p.s. Sincere apologies for being unable to complete this. We never did end up prioritizing it for development, but I'm glad that someone else was able to finish it.

[kadimon]

I think you should close the issue, because it is done in https://github.com/gocolly/colly/blob/master/request.go#L52

I am trying to use this. But requests do not go through a proxy
c.OnRequest(func(r *colly.Request) { r.ProxyURL = rand_proxy })