Aut: ExtractDomain or ExtractBaseDomain?

Created on 25 Oct 2019  路  6Comments  路  Source: archivesunleashed/aut

In Scala RDD, we have:

import io.archivesunleashed._
import io.archivesunleashed.matchbox._

RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).keepValidPages()
  .map(r => ExtractDomain(r.getUrl))
  .countItems()
  .take(10)

In Scala DF, we have:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc).extractValidPagesDF()
  .select(ExtractBaseDomain($"Url").as("Domain"))
  .groupBy("Domain").count().orderBy(desc("count"))
  .show(20, False)

We should be consistent: ExtractDomain or ExtractBaseDomain?

Scala clean-up

All 6 comments

Let's go with ExtractDomain.

Shall we just rename it to ExtractDomainDF (in the interim period between moving migrating all the RDDs to DFs)?

https://github.com/archivesunleashed/aut/blob/e32ae17c55740c6c6adc177d42690b39fa6321dd/src/main/scala/io/archivesunleashed/df/package.scala#L34

Any reason why you want the DF suffix?
I suppose there's the potential for confusion, but I like the beauty of uniformity...

Oh, just keeping it consistent with how we've done the same with the other DF functions like here https://github.com/archivesunleashed/aut/blob/e32ae17c55740c6c6adc177d42690b39fa6321dd/src/main/scala/io/archivesunleashed/package.scala#L115

...I'm happy to keep it uniform too.

Although per https://github.com/archivesunleashed/aut/issues/366 we would rename that to just hyperlinks? To make consistent with Python?

Okay, so stick to ExtractDomain?

Was this page helpful?
0 / 5 - 0 ratings

Related issues

ruebot picture ruebot  路  4Comments

ruebot picture ruebot  路  5Comments

SamFritz picture SamFritz  路  4Comments

ruebot picture ruebot  路  5Comments

Natkeeran picture Natkeeran  路  8Comments