Colly: Example using JSON POST?

Created on 25 Jun 2018 · 4Comments · Source: gocolly/colly

I am trying to scrape a page from a website that requires authentication first. The site processes logins using POST data and JSON payload. It appears that I need to use func (*Collector) Request and set the accept header first and then the request payload in JSON.

Are there any examples of how to craft this request? I've been stabbing at this for a few hours with no luck.

Source

expatmt

Most helpful comment

Thanks for the help. For the benefit of anyone else in the same situation, I got it working with this:

payload := []byte(`{"user":{"email":"[email protected]","password":"mypassword"}}`)
c := colly.NewCollector()
c.OnRequest(func(r *colly.Request) {
    r.Headers.Set("Content-Type", "application/json;charset=UTF-8")
})
err := c.PostRaw(URL, payload)

expatmt on 25 Jun 2018

👍7

All 4 comments

You can use collector.PostRaw (https://godoc.org/github.com/gocolly/colly#Collector.PostRaw) to send JSON. If you have to set additional headers, you can do it in a collector.OnRequest callback.

asciimoo on 25 Jun 2018

Thanks for the help. For the benefit of anyone else in the same situation, I got it working with this:

payload := []byte(`{"user":{"email":"[email protected]","password":"mypassword"}}`)
c := colly.NewCollector()
c.OnRequest(func(r *colly.Request) {
    r.Headers.Set("Content-Type", "application/json;charset=UTF-8")
})
err := c.PostRaw(URL, payload)

expatmt on 25 Jun 2018

👍7

but i can not get response.request.body data