Aut: Migration of all RDD functionality over to DataFrames

[lintool]

We need to migrate all current RDD functionality over to DataFrames. This means porting all matchbox UDFs over to DF UDFs.

There are two possible ways to do this - we can simply take matchbox UDFs and wrap them, or rewrite them from scratch. I suggest we revisit one by one, which will give us an opportunity to refine the UDF we actually want.

For example, the current RDD matchbox ExtractDomain is implemented a bit differently than the DF version we've been playing with... it, for example, strips the prefix www, whereas the RDD impl doesn't. I like the newer implementation better, but open to discussion.

Also, this is an issue we'll come across sooner or later:
https://stackoverflow.com/questions/33664991/spark-udf-initialization

I have a general question we need to look into from the performance perspective: what's the lifecycle status of a Spark DF UDF? In particular, if there's initialization like compiling regexp, we don't want to do that over and over again... we want to have an init stage?

@TitusAn let's start developing in parallel the DF versions of the apps in #222 and try and work this out?

[ruebot]

@jrwiebe I have a note here from our call to "Go through the scala dir and identify all the functions that take in RDD and do not take in DF, then create tickets for JWb." Do you want me to do that granular of a level, or do you want to use this issue to take care of it?

[jrwiebe]

This is fine.

[ruebot]

@ianmilligan1 @lintool this look the basic inventory?

| rdd | data frame |
|----------------------------|------------------------|
| keepValidPages | |
| extractValidPages | extractValidPagesDF |
| | extractHyperlinksDF |
| | extractImageLinksDF |
| | extractImageDetailsDF|
| keepImages | |
| keepMimeTypes | |
| keepUrls | |
| keepUrlPatterns | |
| keepDomains | |
| keepLanguages | |
| keepContent | |
| discardMimeTypes | |
| discardDate | |
| discardUrls | |
| discardUrlPatterns | |
| discardDomains | |
| discardContent | |
| extractFromRecords | |
| extractFromScrapeText | |
| WriteGEXF | |
| ExtractPopularImages | |
| DomainFrequencyExtractor | |
| DomainGraphExtractor | |
| PlainTextExtractor | |
| ExtractGraphX | |

[lintool]

I think we should just leave this as a "catch-all" issue, open.

IMO, this should be driven by the documentation update - go through docs, everything that we do with RDDs, we make sure there's a corresponding DF code example. When the docs have everything in both RDD and DF, I think we're done.

[ruebot]

@SinghGursimran this one is tied to #372, and should become a lot clearer as to what needs to be done to close this one. I _think_ we're pretty close here.

[ruebot]

From @lintool

Sorry to be late to the party... I seem to faintly recall that we were going to rename all our UDFs to MyFuncDF and MyFuncRDD to disambiguate. Were we still going to do that? Circle around in another PR?

This spawned from a Slack convo Jimmy and I had in a non-public channel that never made it to the ticket.

I have a branch working through this now.

[SinghGursimran]

Final Inventory: Matchbox

RDD | Data Frame
--- | ---
ComputeImageSize | ComputeImageSizeDF
ComputeMD5RDD | ComputeMD5DF
ComputeSHA1RDD | ComputeSHA1DF
DetectLanguageRDD | DetectLanguageDF
DetectMimeTypeTikka | DetectMimeTypeTikkaDF
ExtarctBoilerPipeTextRDD | ExtarctBoilerPipeTextDF
ExtractDateRDD | ExtractDateDF
ExtarctDomainRDD | ExtarctDomainDF
ExtarctImageDetails | ExtarctImageDetailsDF
ExtarctImageLinksRDD | ExtarctImageLinksDF
ExtarctLinksRDD | ExtarctLinksDF
ExtractTextFromPDFs | -
GetExtensionMimeRDD | GetExtensionMimeDF
RemoveHTMLRDD | RemoveHTMLDF
RemoveHTTPHeaderRDD | RemoveHTTPHeaderDF
NERClassifier | -
RemovePrefixWWW | RemovePrefixWWWDF

[SinghGursimran]

Current Inventory: App

Functionality | RDD | Data Frame
--- | --- | ---
CommandLineApp | Yes | Yes
DomainFrequencyExtractor | Yes | Yes
DomainGraphExtractor | Yes | Yes
ExtractEntities | Yes | -
ExtractGraphX | Yes | No
ExtarctPopularImages | Yes | Yes
NERCombinerJson | Yes | -
PlainTextExtarctor | Yes | No
WriteGEXF | Yes | No
WriteGraph | Yes | No
WriteGraphML | Yes | No

[SinghGursimran]

Final Inventory: Serializable APIs

RDD | Data Frame
--- | ---
keepValidPages | keepValidPagesDF
keepMimeTypes | keepMimeTypesDF
keepMimeTypesTika | keepMimeTypesTikaDF
keepHttpStatus | keepHttpStatusDF
keepDate | keepDateDF
keepUrls | keepUrlsDF
keepUrlPatterns | keepUrlPatternsDF
keepImages | keepImagesDF
keepDomains | keepDomainsDF
keepLanguages | keepLanguagesDF
keepContent | keepContentDF
discardMimeTypes | discardMimeTypesDF
discardMimeTypesTika | discardMimeTypesDF
discardDate | discardDateDF
discardUrls | discardUrlsDF
discardHttpStatus | discardHttpStatusDF
discardUrlPatterns | discardUrlPatternsDF
discardDomains | discardDomainsDF
discardContent | discardContentDF
discardLanguages | discardLanguagesDF

[ruebot]

@SinghGursimran we still need to do discardMimeTypesTikaDF. Looks like you have discardMimeTypesDF duplicated.

[ruebot]

@SinghGursimran nvm. I see it now :-)

https://github.com/archivesunleashed/aut/blob/bc0d663fb42125af2ec9326c71a08e53f8a86457/src/main/scala/io/archivesunleashed/package.scala#L243-L250

[ruebot]

Created three new issues that _should_ cover the Python implementations of most of the work here.

• #408
• #409
• #410

[ruebot]

Merging https://github.com/archivesunleashed/aut/pull/441 fully resolved this.

[lintool]

🎉

[ruebot]

We forgot about ExtractBoilerplateRDD :man_facepalming:

[ruebot]

Oh, it's there ExtractBoilerpipeTextDF.

Sorry for the confusion. :man_facepalming: