In Scala RDD, we have:
import io.archivesunleashed._
import io.archivesunleashed.matchbox._
RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).keepValidPages()
.map(r => ExtractDomain(r.getUrl))
.countItems()
.take(10)
In Scala DF, we have:
import io.archivesunleashed._
import io.archivesunleashed.df._
RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
.select(ExtractBaseDomain($"Url").as("Domain"))
.groupBy("Domain").count().orderBy(desc("count"))
.show(20, False)
We should be consistent: ExtractDomain or ExtractBaseDomain?
Let's go with ExtractDomain.
Shall we just rename it to ExtractDomainDF (in the interim period between moving migrating all the RDDs to DFs)?
Any reason why you want the DF suffix?
I suppose there's the potential for confusion, but I like the beauty of uniformity...
Oh, just keeping it consistent with how we've done the same with the other DF functions like here https://github.com/archivesunleashed/aut/blob/e32ae17c55740c6c6adc177d42690b39fa6321dd/src/main/scala/io/archivesunleashed/package.scala#L115
...I'm happy to keep it uniform too.
Although per https://github.com/archivesunleashed/aut/issues/366 we would rename that to just hyperlinks? To make consistent with Python?
Okay, so stick to ExtractDomain?