Aut: Refactor RecordLoader classes

[greebie]

We have a number of classes called from RecordLoader that have ArchiveRecord as a parent. We should refactor the RecordLoader object to use one class -- ArchiveRecord -- that detects records that are warc or arc.

(The commit adding the Generic class is https://github.com/archivesunleashed/aut/commit/77891e7d54ad1527dcddaa434120c916da661f65 )
I think Jeremy did us very well here -- just needed a moment to consider refactoring.

[MapleOx]

Currently, is there any reason someone would want to use loadArc or loadWarc instead of loadArchives? If not, I think we can consider getting rid of loadArc and loadWarc too.

[lintool]

Probably not.

[ianmilligan1]

Agreed, I think we have totally moved over to loadArchives.

[ruebot]

So, remove these two instances from the code base?

https://github.com/archivesunleashed/aut/blob/010fe244236d160113853f442f7dce25442c59dc/src/main/scala/io/archivesunleashed/spark/matchbox/RecordLoader.scala#L30-L33

https://github.com/archivesunleashed/aut/blob/010fe244236d160113853f442f7dce25442c59dc/src/main/scala/io/archivesunleashed/spark/matchbox/RecordLoader.scala#L35-L39

Along with where they show up in tests? If we're code reaping, I'll take care of it. Keep my :skull: stats up,

[ianmilligan1]

Yep, I think they're good to go – and I don't think they're called by anything else so should be an easy removal?

[MapleOx]

I took a look at the Scala and Java classes, and if we're going to always work with generic archive records, I think we can refactor as follows:

• In the Scala code:
  
  
  
  1. Remove the trait ArchiveRecord and classes ArcRecord and WarcRecord;
  
  
  2. Rename GenericArchiveRecord to ArchiveRecord.
  
  
• In the Java code:
  
  
  
  1. Under java.io.archivesunleashed.io, remove the classes ArcRecordWritable and WarcRecordWritable;
  
  
  2. Under java.io.archivesunleashed.mapreduce, remove the classes WacArcInputFormat and WacWarcInputFormat.
  
  

After possibly straightening up import statements and tests that use the aforementioned classes, I think this should work... Also, this might address the test issues in #106?

[ruebot]

@MapleOx yeah, that makes sense. As I'm digging deeper into cleaning up #106, I'm now down the rabbit hole of GenericArchiveRecord and ArchiveRecord. Definitely wasn't as simple as I thought it was last night :smile:

[lintool]

@MapleOx clean up the Scala end, but let's leave the Java stuff alone. InputFormats and Writables are independently useful for connecting to other systems...

Please send a PR.

[ruebot]

@MapleOx do you want me to push up what I have right now in another branch? In the interim I'll close #106.

[MapleOx]

@ruebot sure, thanks :) I have to head out soon, but I'll get a PR submitted this afternoon.

[ruebot]

@MapleOx yeah, I'm about to head out now as well. I'll push to issue-102-b. It's broken, but I've made progress. If it's too much of a mess, feel free to do your own thing and don't worry about what I've done.

[greebie]

@ruebot I can take a look at RecordRDDTest on Monday. I had a feeling tests would start breaking once I got into it as well. All this because of #101 ! In the end, this will be very good though. Simplify the codebase.

[ruebot]

Resolved with: https://github.com/archivesunleashed/aut/commit/5e99d63083ac0b3c13011a528f247dc47ed66004