Aut: Plain Text UDF that combines RemoveHTML + RemoveHttpHeader

[ianmilligan1]

Is your feature request related to a problem? Please describe.
Right now we have two related functions: RemoveHTML, which removes all of the HTML markup from a document; and RemoveHttpHeader which removes the HTTP header.

I'm increasingly convinced that we should be removing these HTTP headers from many of our derivatives, as they throw a lot of text analysis off.

Describe the solution you'd like
In a dream world, I would like to be able to execute a script like this:

import io.archivesunleashed._
import io.archivesunleashed.matchbox._

RecordLoader.loadArchives("/home/i2millig/aut-resources/Sample-Data/*.gz", sc)
  .keepValidPages()
  .map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTMLand Header(r.getContentString)))
  .take(1)

Note that the RemoveHTMLandHeader function combines both functionalities in one call.

Describe alternatives you've considered
You could probably script it up too but it'd be nice to do it in one fell swoop.

[greebie]

Hi Ian - I plan to implement this today. I wonder if just removing the header when we use RemoveHTML makes sense? I cannot think of a use-case where I'd want to keep the header, but not the HTML.

I can create a combined function as well. The only issue is that we would have a number of permutations of removing boilerplate, html, headers and any other additional elements down the road.

We could also expand the ArchiveRecord trait to have things like "r.textRaw," "r.textNoBoilerplate," and "r.textHtml." by default. This would mean that dataframes would have these additional columns available as well.

[ruebot]

I think @dportabella had a use-case for keeping just the headers, and/or http status codes. I'd imagine there are other use cases out there for just studying the headers, and/or status codes.

[greebie]

Okay -- I'm going to try and see if adding features to the trait works out. If I cannot or it seems needlessly confusing, I'll default to Ian's original request.

[ianmilligan1]

I'm now doubting my original request, as nesting them together (as in my AUK issue here) probably is just as easy. 🤦‍♂️

So I'm happy if we close this, or if you want to expand the trait, as that might be a real enhancement along the lines that @ruebot noted above.

[greebie]

Okay - I'll play with this today and either come up with a refactor on the ArchiveRecord or close the issue. I will include some comparative data to ensure no significant efficiency impacts for our usual gang of scripts.

[ruebot]

If we're chaining them together, and we get what we want, then I think we're fine to close this issue. No need to go down an unnecessary rabbit hole here.

[greebie]

I think that sounds fair. Maybe consider the refactor for (another) rainy day.