Colly: strange behavior when setting user agent

[tamoyal]

I found a strange case where setting the user agent causes the crawl to not work but curl gets the site response fine with the same user agent. I provided example code below which you can run with and without the user agent to see it fail/succeed.

package main

import (
    "log"
    "regexp"

    "github.com/gocolly/colly"
)

func main() {
    escaped := regexp.QuoteMeta("cityoven.com")
    r := regexp.MustCompile(`^https?:\/\/[a-z]*\.?` + escaped + `.*`)
    c := colly.NewCollector(
        colly.URLFilters(r),
        colly.UserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"), // comment this out and the crawl will work
    )

    c.OnResponse(func(r *colly.Response) {
        log.Println("Got: ", r.Request.URL)
    })
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Attr("href")
        c.Visit(e.Request.AbsoluteURL(link))
    })
    c.Visit("http://cityoven.com")
    c.Wait()
}

There is no problem getting the response with curl and the same user agent header above.

curl --header "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.217 1.95 Safari/537.36" http://cityoven.com/

Any idea why this might happen?

[asciimoo]

I added an OnError callback and it showed that the server sent a HTTP Forbidden response.
Then I compared curl's and colly's requests and the only difference was an additional Accept: */* HTTP header. The example works if you add it to colly:

package main

import (
    "log"
    "regexp"

    "github.com/gocolly/colly"
)

func main() {
    escaped := regexp.QuoteMeta("cityoven.com")
    r := regexp.MustCompile(`^https?:\/\/[a-z]*\.?` + escaped + `.*`)
    c := colly.NewCollector(
        colly.URLFilters(r),
        colly.UserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"),
    )
    c.OnRequest(func(r *colly.Request) {
        r.Headers.Set("Accept", "*/*")
    })
    c.OnResponse(func(r *colly.Response) {
        log.Println("Got: ", r.Request.URL)
    })
    c.OnError(func(r *colly.Response, err error) {
        log.Println("Error: ", r.Request.URL, err)
    })
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        //link := e.Attr("href")
        //c.Visit(e.Request.AbsoluteURL(link))
        e.Request.Visit(e.Attr("href"))
    })

    c.Visit("http://cityoven.com")
    // c.Wait() <- this is only required in async mode
}

Perhaps, we can consider adding this header by default to Colly's requests.