Colly: Question: Handle encodings

[rubenwap]

I just tested colly, and loved how fast it performed. I have a question about encodings. In the docs it says that it has automatic encoding of non unicode responses. Can this be customized? I tried to grab content from https://www.nsd.ru/ru/db/news/ndcpress/ , which is windows-1251 encoded, and the contents came back unreadable, so I was wondering how can I set up colly to grab content specifying the encoding.

[asciimoo]

@rubenwap try with

c := colly.NewCollector(colly.DetectCharset())

or

c.DetectCharset = true

Does it solve the problem?

[rubenwap]

@asciimoo Thanks for checking this. Your suggestion didn't solve the problem, the text contents of the url just came back scrambled in a slightly different way, so it seems that in one way it tried to detect the charset as you say, but the guess was wrong and the chosen charset was not the right one.

I believe your suggestion of auto detecting it means that you cannot specify the one you want?

Thanks again

[asciimoo]

@rubenwap currently explicit encoding definition is not supported, however it seems that your problem cannot be solved without it. Would you like to implement it?

ps: it is interesting that chardet cannot detect the valid encoding..

[rubenwap]

@asciimoo Just in case it's useful for you to know: Chardet also failed when I tried to do this in Python. The popular requests module uses chardet to guess the encoding if you are simply calling something such as:

r = requests.get("https://www.nsd.ru/ru/db/news/ndcpress/").text

And it didn't work. To get it working, I had to call content from the request, rather than text, so in content it could read the information about the charset and apply the right one.

I would love to help implementing this, but I am afraid I am still very confused about how the encodings work in Go, so I wouldn't be a very good candidate. I will keep learning more about the subject and keep a close eye on this project and help with whatever I can. To be honest, my issue is already fixed by using Python, but I really want to repeat the task with colly because I like it better.

[asciimoo]

@rubenwap no problem, thanks for the info, then I'll do it in the following days. Perhaps the changes can help you to learn more about encodings in go.

[asciimoo]

Also, here's a great little article which explains go character encoding conversions: http://technosophos.com/2016/03/09/go-quickly-converting-character-encodings.html

[rubenwap]

Thank you for the article @asciimoo , I will definitely have a look at that, and keep learning about how Go works (also reading Colly's code). I want to move away some of my stuff from Python to Go, so this is being a great help.
Thanks again for your work.

[asciimoo]

@rubenwap i've added ResponseCharacterEncoding string member to colly.Request, so from now you can set explicit character encoding in an OnRequest callback. Could you verify the fix?

example:

c := colly.NewCollector()
c.OnRequest(func(r *colly.Request) {
    r.ResponseCharacterEncoding = "windows-1251"
}

[rubenwap]

@asciimoo Amazing! I have tested it and it works as good as expected. Thanks for your very quick reaction to this.

[moomdate]

@asciimoo it work for me Thaks!