Colly: [Question] about collector.Visit("url") & collector.HasVisited(link)

[issmeftah]

Hello guys,

I have 2 questions about colly collector.

Question 01:
about the method ( collector.Visit("url") ) some ahref on the scrapped pages have urls like:

• /best-video-editing-software#respond
• /#comment-5700
• /#why-should-i-use-a-website-builder
• etc..

I mean no full url with schema (https:// or http://). does those urls get visited by the collector? or they just get skipped?

Question 02:
about the method ( visited, err := collector.HasVisited(link) )
I m using multiple goroutines to scrap multiple urls at the same time. where each goroutine will clone a main collector and start scrapping.

• I've enabled storage to the main collector using ( collector.SetStorage(&storage.InMemoryStorage{}) )
• I've set ( collector.AllowURLRevisit = false )

and I'm saving each visited url and print them in a .txt file. But i've noticed some duplicated visited URLs.
Please help me out.

Here is my current code:

func (s *Spider) collect(inputURL string) {
    // clone main collector
    spider := s.collector.Clone()

    // main domain
    raw, err := url.Parse(inputURL)
    if err != nil {
        // handle err
        // test
        return
    }

    // set allowed domains
    spider.AllowedDomains = []string{raw.Hostname()}

    // User-Agent & referer
    extensions.RandomUserAgent(spider)
    extensions.Referer(spider)

    // Set error handler
    spider.OnError(func(r *colly.Response, err error) {
        if err != nil {
        // handle err
        }
        // testing
        r.Request.Abort()
    })

    // start
    defer func() {
        spider.OnHTMLDetach("a[href]")
    }()
    spider.OnHTML("a[href]", func(e *colly.HTMLElement) {
        // find each link in html
        link := e.Attr("href")

        // remove white space
        link = strings.TrimSpace(link)

        // link check
        if linkCheck(link) {
            visited, err := spider.HasVisited(link)
            if err != nil {
                // handle err
                }
                // test
                e.Request.Abort()
            }
            if !visited {
                e.Request.Visit(e.Request.AbsoluteURL(link))
                // send to writer
                s.writer.visits <- link
            }
        }
    })

    // visit
    spider.Visit(inputURL)
    spider.Wait()
}

[issmeftah]

Hello, I've found solution to 1st question. (not issues) the solutions are already there, i did not check/test well.

Question 01:
about the method ( collector.Visit("url") ) some ahref on the scrapped pages have urls like:

/best-video-editing-software#respond
/#comment-5700
/#why-should-i-use-a-website-builder
etc..

simply use:

 e.Request.AbsoluteURL(e.Attr("href"))

instead of using directly

e.Attr("href")

Like this:

    spider.OnHTML("a", func(e *colly.HTMLElement) {
        link := e.Request.AbsoluteURL(e.Attr("href"))
            if visited, _ := spider.HasVisited(link); !visited {
                spider.Visit(link)
                s.writer.visits <- link
            }
    })

[asciimoo]

and I'm saving each visited url and print them in a .txt file. But i've noticed some duplicated visited URLs.

It can be possibly a race condition. Use the URL of the currently visited page from an OnRequest or OnResponse callback instead of all the links on the page.