Spark-nlp: DocumentAssembler in Java

[jeffisenhart]

Description

I am trying to create a DocumentAssembler instance in Java

Expected Behavior

The following code should create a DocumentAssembler instance

import com.johnsnowlabs.nlp.DocumentAssembler;

public class DocumentTest {

    public static void main(String[] args) {
        try {
            DocumentAssembler da = new DocumentAssembler();
            da.setInputCol("text").setOutputCol("document");
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }
}

Current Behavior

But I get this instead

java.lang.NoClassDefFoundError: org/apache/spark/ml/util/MLWritable$class
    at com.johnsnowlabs.nlp.DocumentAssembler.<init>(DocumentAssembler.scala:16)
    at com.johnsnowlabs.nlp.DocumentAssembler.<init>(DocumentAssembler.scala:56)
    at hack.spark.DocumentTest.main(DocumentTest.java:9)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.ml.util.MLWritable$class
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499)
    ... 3 more

• Version used:
  compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '2.4.0'
  compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '2.4.0'
  compile group: 'com.johnsnowlabs.nlp', name: 'spark-nlp_2.11', version: '1.8.2'
  compile group: 'com.johnsnowlabs.nlp', name: 'spark-nlp-ocr_2.11', version: '1.8.2'
  runtime group: 'org.apache.spark', name: 'spark-mllib_2.12', version: '2.4.0'

• Operating System and version (desktop or mobile):
  maxOS Mojave 10.14.1

[maziyarpanahi]

Can you please try to use Scala 2.11 for Spark as well? Not sure if it has anything to do with this error, but I don’t think Spark NLP supports 2.12 even on Spark’s side.

[jeffisenhart]

Yep, changing dependencies solved it

compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.4.0'
compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.4.0'
compile group: 'com.johnsnowlabs.nlp', name: 'spark-nlp_2.11', version: '1.8.2'
compile group: 'com.johnsnowlabs.nlp', name: 'spark-nlp-ocr_2.11', version: '1.8.2'
runtime group: 'org.apache.spark', name: 'spark-mllib_2.11', version: '2.4.0'