Colly: robots.txt support

Created on 13 Oct 2017  路  5Comments  路  Source: gocolly/colly

Does it make sense to support abiding by the website robots.txt?

enhancement
Source
picture 0verbyte

Most helpful comment

I can.

picture 0verbyte on 13 Oct 2017
🎉3

All 5 comments

@voidpirate yes, this is definitely a good optional feature, thank you. Could you work on this?

asciimoo picture asciimoo on 13 Oct 2017

I can.

0verbyte picture 0verbyte on 13 Oct 2017
🎉3

Is this being worked on. Looks like it should just be a few lines of code if you integrate with https://github.com/temoto/robotstxt

vosmith picture vosmith on 23 Oct 2017

I planned on working on this, just haven't allocated the time as I have been busy with my job.

If anyone else wants to take this over feel free to.

0verbyte picture 0verbyte on 23 Oct 2017

closed by 5bc9a6c

asciimoo picture asciimoo on 27 Oct 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

CrazyMouse picture CrazyMouse  路  3Comments
peterhellberg picture peterhellberg  路  3Comments
prologic picture prologic  路  6Comments
killernova picture killernova  路  6Comments
ascode picture ascode  路  3Comments