Spark-nlp: Loading Custom tried pipeline throws Error

[subhashiniy]

While loading a saved pipeline model, error is thrown.

Description

Pipeline model after training is saved during training time. When a single document needs to be annotated, the same model that was used in training needs to be loaded. While doing so I run into "java.lang.ClassNotFoundException: com.johnsnowlabs.nlp.DocumentAssembler" Error

Expected Behavior

Current Behavior

This the output of the error that is raised.

20/03/17 10:39:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/prodData/aura/SparkNLP/envname/lib/python3.6/site-packages/pyspark/ml/util.py", line 362, in load return cls.read().load(path) File "/prodData/aura/SparkNLP/envname/lib/python3.6/site-packages/pyspark/ml/pipeline.py", line 242, in load return JavaMLReader(self.cls).load(path) File "/prodData/aura/SparkNLP/envname/lib/python3.6/site-packages/pyspark/ml/util.py", line 300, in load java_obj = self._jread.load(path) File "/prodData/aura/SparkNLP/envname/lib/python3.6/site-packages/py4j/java_gateway.py", line 1257, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/prodData/aura/SparkNLP/envname/lib/python3.6/site-packages/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/prodData/aura/SparkNLP/envname/lib/python3.6/site-packages/py4j/protocol.py", line 328, in get_return_value format(target_id, ".", name), value) py4j.protocol.Py4JJavaError: An error occurred while calling o35.load. : java.lang.ClassNotFoundException: com.johnsnowlabs.nlp.DocumentAssembler at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:238) at org.apache.spark.ml.util.DefaultParamsReader$.loadParamsInstance(ReadWrite.scala:651) at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$4.apply(Pipeline.scala:274) at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$4.apply(Pipeline.scala:272) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at org.apache.spark.ml.Pipeline$SharedReadWrite$.load(Pipeline.scala:272) at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:348) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)

Possible Solution

Steps to Reproduce

A bit of the code for saving looks like this:

`pipeline = Pipeline(stages=[ 
    document_assembler,
    sentence_detector,
    tokenizer,
    stop_words_cleaner,
    finisher
])

model = pipeline.fit(data)
savePath = '../../' + '/sparkPipeline'
#import pdb; pdb.set_trace()
model.save(savePath)`

Code for loading :
pipelineModel = PipelineModel.load(path)
1.
2.
3.
4.

Context

Your Environment

• Spark NLP version: 2.4.4
• Apache NLP version: 2.4.4
• Setup and installation (Pypi, Conda, Maven, etc.): pip
• Operating System and version: Ubuntu 18.04
• Link to your project (if any):

[maziyarpanahi]

Your code is correct however I need to see how you are starting SparkSession.

[maziyarpanahi]

PS: As stated here there was a bug in 2.4.2 in Python which was fixed in 2.4.3: https://github.com/JohnSnowLabs/spark-nlp/releases/tag/2.4.3

Either pipeline was saved in 2.4.2, or is being loaded in 2.4.2. You have to make sure the pip install --upgrade spark-nlp and also sparknlp.start() or manual SparkSession is also using 2.4.3 above. This way you can be sure this error won't happen. (The error is the exact same bug in 2.4.2, that is why I asked to see how SparkSession was created)
If you can make sure all the versions are 2.4.3 above and re-train and load the pipeline in 2.4.3 above this error won't happen.

[subhashiniy]

It was a matter or starting the session in the right version.
Thank you for your help!