Spark-nlp: How to use Spark NLP in Java

[ansorre]

On home page of website: https://nlp.johnsnowlabs.com/ I read "Full Python, Scala, and Java support"

Unfortunately it's 3 days now I'm trying to use Spark NLP in Java without any success.

• I cannot find Java API (JavaDoc) of the framework.
• not event a single example in Java is available
• I do not know Scala, I do not know how to convert things like:
  val testData = spark.createDataFrame(Seq((1, "Google ..."),(2, "The Paris ..."))).toDF("id", "text")
  from Scala to Java neither searching on Google for 3 days was of any help, neither trying on my own I was able to make this conversion.

Sorry to annoy with this but is there anyone out there who is really using Spark NLP with Java? How do you do it? Are there online resources available to learn how to do it?
Thanks anyway.

[maziyarpanahi]

Hi @Antonio-Sorrentini
I put Java on the home page as I had the understanding that the APIs can be easily used in Java. I might be wrong or Java needs its own documentation like Scala and Python. So let me ping some other developers and we get to the bottom of this.

PS: your code example is pure Apache Spark to manually create a DataFrame, not really Spark NLP.

[ansorre]

Ok @maziyarpanahi thanks.

[alinapetukhova]

Hi, @Antonio-Sorrentini ! Here is example for Java pipeline with Spark-NLP. You can find lib in maven repo.

SparkSession spark = SparkSession
                .builder()
                .appName("PipelineExample")
                .config("spark.master", "local")
                .getOrCreate();

Dataset<Row> data = spark.read().format("csv")
                .option("inferSchema", "true")
                .option("header", "true")
                .option("multiLine", "true")
                .option("escape", "\"")
                .load("data.csv");

DocumentAssembler document_assembler = (DocumentAssembler) new DocumentAssembler().setInputCol("text").setOutputCol("document");

SentenceDetector sentence_detector = (SentenceDetector) ((SentenceDetector) new SentenceDetector().setInputCols(new String[] {"document"})).setOutputCol("sentence");

Tokenizer tokenizer = (Tokenizer)((Tokenizer) new Tokenizer().setInputCols(new String[] {"sentence"})).setOutputCol("token");

Normalizer normalizer = (Normalizer)((Normalizer) new Normalizer().setInputCols(new String[]{"token"})).setOutputCol("normalized");

LemmatizerModel lemmatizer = (LemmatizerModel)((LemmatizerModel) LemmatizerModel.pretrained("lemma_antbnc", "en", "public/models").setInputCols(new String[]{"normalized"})).setOutputCol("lemma");

Finisher finisher = new Finisher().setInputCols(new String[]{"document", "lemma"}).setOutputCols(new String[]{"document", "lemma"});

Pipeline pipeline = new Pipeline().setStages(new PipelineStage[]{document_assembler, sentence_detector, tokenizer, normalizer, lemmatizer, finisher});

// Fit the pipeline to training documents.
PipelineModel pipelineFit = pipeline.fit(data);
Dataset<Row> dataet = pipelineFit.transform(data);

[dkincaid]

I am trying to use a pretrained pipeline and running into an issue.

SparkSession spark = SparkSession
          .builder()
          .appName(appName)
          .master(master)
          .config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.2.1")
          .getOrCreate();

PretrainedPipeline pipeline = new PretrainedPipeline("explain_document_ml", "en",
        PretrainedPipeline$.MODULE$.apply$default$3());

List<StructField> fields = new ArrayList<StructField>();
StructField field1 = DataTypes.createStructField("id", DataTypes.IntegerType, true);
StructField field2 = DataTypes.createStructField("text", DataTypes.StringType, true);
fields.add(field1);
fields.add(field2);

StructType schema = DataTypes.createStructType(fields);

List<Row> rows = Lists.newArrayList(
          RowFactory.create(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
          RowFactory.create(2, "The Paris metro will soon enter the 21st century, ditching single-use paper tickets for rechargeable electronic cards.")
);

Dataset<Row>  testData = spark.createDataFrame(rows, schema);
Dataset<Row> annotation = pipeline.transform(testData);

I'm getting a NoClassDefFoundError that I can't quite figure out. It is getting thrown on the line creating the PretrainedPIpeline object. I tried explicitly including the scala-library jar( version 2.12.x) in my project, but that didn't fix it. Any ideas?

Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
    at com.johnsnowlabs.nlp.pretrained.PretrainedPipeline.<init>(PretrainedPipeline.scala:7)
    at com.idexx.nlp.johnsnow.poc.Runner.run(Runner.java:34)
    at com.idexx.nlp.johnsnow.poc.Runner.main(Runner.java:66)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 3 more

[dkincaid]

In case anyone else comes across this I did get my NoClassDefFound error fixed. My problem was that I was using the spark-code_2.12 version, but Spark NLP is built with _2.11 (which I should have seen originally since it's in the name of the library). Once I changed the spark lib to _2.11 it works.