Colly: Visit url using port?

Created on 23 Apr 2020  路  3Comments  路  Source: gocolly/colly

Is gocolly support request an url something like : http://domain.example:8080/index.php

Currently seem it does not request to the target url.

Thank you.

question

Most helpful comment

Yes this seem like a valid url so it should be working. Alo here is the working example. I've created a site on my local computer and tested on it.

func main() {
    c := colly.NewCollector()

    // Find and visit all links
    c.OnHTML("*", func(e *colly.HTMLElement) {
        fmt.Println(e)
        e.Request.Visit(e.Attr("href"))
    })

    c.OnRequest(func(r *colly.Request) {
        fmt.Println("Visiting", r.URL)
    })
    err := c.Visit("http://domain.example:5000/")

    if err != nil {
        fmt.Printf("Done with error %s", err.Error())
    }
}

All 3 comments

Yes this seem like a valid url so it should be working. Alo here is the working example. I've created a site on my local computer and tested on it.

func main() {
    c := colly.NewCollector()

    // Find and visit all links
    c.OnHTML("*", func(e *colly.HTMLElement) {
        fmt.Println(e)
        e.Request.Visit(e.Attr("href"))
    })

    c.OnRequest(func(r *colly.Request) {
        fmt.Println("Visiting", r.URL)
    })
    err := c.Visit("http://domain.example:5000/")

    if err != nil {
        fmt.Printf("Done with error %s", err.Error())
    }
}

Thanks for the answer @asalih

Hi,

 It's working now.
 Seem the reason in my case is when I allowed the domain "example.com:8080"  ( colly.AllowedDomains() )
 It was by the getDomainFromUrl function,  it need to cut off the port also.

Thank you.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

mazhigali picture mazhigali  路  5Comments

prologic picture prologic  路  6Comments

dzakaammar picture dzakaammar  路  3Comments

andystroz picture andystroz  路  3Comments

CrazyMouse picture CrazyMouse  路  3Comments