Colly: Possibe to crawl and scrape text files?

[prologic]

I don't see an API for this besides Collector.OnResponse. Curious if its possible to scrap plain/text files? My use-case is scraping a bunch of hosted text files that _may_ have linked to other text files.

[prologic]

Ping?

[WGH-]

What's your expectations? You want colly to find URLs in plain text using some heuristics?

[prologic]

What's your expectations? You want colly to find URLs in plain text using some heuristics?

Yes. I want the same semantics that the colly API provides for HTML, but for plain text. Look for (_perhaps using regexes?_) things that might look like links to other documents.

In my case I'm wanting to scrape twtxt feeds looking for other links of the form <nick url>. I _realize_ this isn't exactly a HTML ink pe rse, but if the API could provide a custom search so I can define what to look for as "links" in documents?

[WGH-]

What's exactly wrong with doing such regex heuristics in OnResponse yourself?

[prologic]

This is what i did

[WGH-]

Strictly speaking, Colly doesn't really define what's an "HTML link" either: it's up to user to enqueue more URLs in his own custom c.OnHTML("a[href]", ...) handler.

I personally don't think there's a merit of incorporating such site-specific heuristics into Colly codebase, but let's see if there're other opinions about this.