Colly: Getting final request when there is a page redirect

Created on 29 Dec 2017 · 2Comments · Source: gocolly/colly

When there is a page redirect, colly automatically follows the redirect. In that case, I get a Request object in the OnHTML callback. It seems that colly provides the original Request and not the one after the redirect. Since I want to follow all the links on the html site, I use the Request object to get the absolute URL. However, in that case this doesn't work as expected, since the Request Object has the wrong URL. The example below illustrates the problem:

package main

import (
    "fmt"
    "net/http"
    "time"

    "github.com/gocolly/colly"
)

func main() {
    go func() {
        http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            http.Redirect(w, r, "/r/", http.StatusSeeOther)

        }))
        http.Handle("/r/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            fmt.Fprintf(w, `<a href="test">test</a>`)
        }))
        http.ListenAndServe("127.0.0.1:9999", nil)
    }()
    time.Sleep(500 * time.Millisecond)
    c := colly.NewCollector()
    c.AllowedDomains = []string{"127.0.0.1:9999"}
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        fmt.Println(e.Request.AbsoluteURL(e.Attr("href")))
    })
    c.Visit("http://127.0.0.1:9999/")
    c.Wait()
    time.Sleep(1000 * time.Hour)
}

The example gives "http://127.0.0.1:9999/test". However when I go to "http://127.0.0.1" via firefox and click on the link, I get redirected to "http://127.0.0.1:9999/r/test".

Is there a better way to mimic the behavior of the browser in this case?

bug

Source