Aut: Class issues when using aut-0.17.0-fatjar.jar

[ruebot]

Describe the bug

java.lang.NoClassDefFoundError and java.lang.NoClassDefFoundError errors are produced when using aut-0.17.0-fatjar.jar. Looks like a potential class conflict in the fatjar, but not when the using --packages.

To Reproduce

1. spark-shell --jars /home/nruest/Downloads/aut-0.17.0-fatjar.jar
2. Load libraries, and RecordLoader.loadArchives:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181)
Type in expressions to have them evaluated.
Type :help for more information.

scala> :paste
// Entering paste mode (ctrl-D to finish)

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
val r = RecordLoader.loadArchives("/home/nruest/tmp/auk/75/3552/warcs/ARCHIVEIT-3552-DAILY-IEDLRH-20130212071008-00001-wbgrp-crawl052.us.archive.org-6680.warc.gz", sc)

// Exiting paste mode, now interpreting.

java.lang.NoClassDefFoundError: net/sourceforge/cobertura/coveragedata/TouchCollector
  at io.archivesunleashed.package$RecordLoader$.__cobertura_init(package.scala)
  at io.archivesunleashed.package$RecordLoader$.<clinit>(package.scala)
  ... 49 elided
Caused by: java.lang.ClassNotFoundException: net.sourceforge.cobertura.coveragedata.TouchCollector
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 51 more

1. Do it again in the same session:

scala> :paste
// Entering paste mode (ctrl-D to finish)


import io.archivesunleashed._
import io.archivesunleashed.matchbox._
val r = RecordLoader.loadArchives("/home/nruest/tmp/auk/75/3552/warcs/ARCHIVEIT-3552-DAILY-IEDLRH-20130212071008-00001-wbgrp-crawl052.us.archive.org-6680.warc.gz", sc)


// Exiting paste mode, now interpreting.

java.lang.NoClassDefFoundError: Could not initialize class io.archivesunleashed.package$RecordLoader$
  ... 57 elided

1. @edsu example

Expected behavior

Well, this shouldn't happen. It should work.

Environment information

• AUT version: 0.17.0
• OS: Ubuntu 18.04
• Java version: Java 8
• Apache Spark version: Tested on 2.3.0, 2.3.2, 2.4.0
• Apache Spark w/aut: --jars
• Apache Spark command used to run AUT: spark-shell --jars /home/nruest/Downloads/aut-0.17.0-fatjar.jar

Additional context

@edsu notes in Slack that the 0.16.0 fatjar works.

[jrwiebe]

@ruebot, what was the build environment where the 0.17.0 release was performed? I checked out the 0.17.0 release branch, built it, and ran the same test, and did not receive an error. However, I do receive the error when using the released fatjar.

I think the problem is that the wrong SLF4J binding is being included in the fatjar. See this issue, and this background. It's not clear to me why I can't replicate the error when I build 0.17.0, though.

Side issue: I think we can drop cobertura-maven-plugin as a general dependency. It's listed as a dependency in <reporting>, which is sufficient for the mvn cobertura:cobertura target.

[ruebot]

@jrwiebe it would have been my laptop or desktop; Ubuntu 18.04, and OpenJDK 8 probably.

Yeah, the SLF4J binding makes since. It came up for me this morning in a quick cursory google.

...good catch on cobertura. That pom haunts me sometimes :smile:

[ruebot]

Helping out a team here at the datathon. This worked, so maybe we're fine on master?

$ ~/spark/bin/spark-shell --jars /home/ubuntu/aut/target/aut-0.17.1-SNAPSHOT-fatjar.jar --master local[*] --driver-memory 54G --conf spark.network.timeout=100000000 --conf spark.executor.heartbeatInterval=6000s --conf spark.driver.maxResultSize=12
2019-03-21 20:23:12 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://host-192-168-69-7.openstacklocal:4040
Spark context available as 'sc' (master = local[*], app id = local-1553199803313).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala> :paste
// Entering paste mode (ctrl-D to finish)

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
val r = RecordLoader.loadArchives("/mnt/data/dcpl/dc-punk/warcs/ARCHIVEIT-10427-TEST-JOB677841-20180831145751779-00001.warc.gz", sc)

// Exiting paste mode, now interpreting.

import io.archivesunleashed._
import io.archivesunleashed.matchbox._
r: org.apache.spark.rdd.RDD[io.archivesunleashed.ArchiveRecord] = MapPartitionsRDD[2] at map at package.scala:69

[jrwiebe]

As I mentioned yesterday, when I built the 0.17.0 release I also had no problem. I tried this with OpenJDK 8 under Linux (Ubuntu 18.04 and OS X) and with Oracle JDK 8 (just OS X).