Spark-nlp: Crossvalidation for Normalizer and other Spark-NLP-Pipeline fails (e.g. "feature slangDict is not set" )

[ahaas01]

Hi guys,

my program fails, when I input a Pipeline with Spark-NLP-Elements in a Crossvalidator. But no error occurs, if i run the pipeline without the Crossvalidator.

I created a Colab Notebook, to easily reproduce the error:

https://colab.research.google.com/drive/1UgFtVMowOT_jC4gj5R-U3r3M3SmWa_HB?usp=sharing

The error message is "feature slangDict is not set", but running it without crossvalidation doesn't require this param. Even by setting "setSlangDictionary" at the Normalizer-Stage does not remove the Error.
Replacing the Normalizer with other NLP-Stages creates similar Error-Messages (feature * is not set).

Stacktrace

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 13.0 failed 1 times, most recent failure: Lost task 1.0 in stage 13.0 (TID 23, localhost, executor driver): org.apache.spark.SparkException: Failed to execute user defined function($anonfun$dfAnnotate$1: (array,embeddings:array>>>) => array,embeddings:array>>)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)

....

Caused by: java.lang.Exception: feature slangDict is not set
at com.johnsnowlabs.nlp.serialization.Feature$$anonfun$getOrDefault$1.apply(Feature.scala:80)
at com.johnsnowlabs.nlp.serialization.Feature$$anonfun$getOrDefault$1.apply(Feature.scala:80)
at scala.Option.getOrElse(Option.scala:121)

Your Environment

• Spark NLP version: 2.6.3
• Spark (pyspark) version: 2.4.6 (fails in Java and PySpark)
• Java version (java -version): 8 (openjdk version "1.8.0_275")
• Setup and installation (Pypi, Conda, Maven, etc.): ---
• Operating System and version: Example in Colab, ( reproducable in Windows w Java)
• Link to your project (if any): https://colab.research.google.com/drive/1UgFtVMowOT_jC4gj5R-U3r3M3SmWa_HB?usp=sharing#scrollTo=83HHcSoI0okm

[maziyarpanahi]

Thanks for reporting this. We will take a deeper look, however, regardless of this error, I would highly suggest separating the NLP pipeline from the ML functions affected by CV.

Meaning, in this case, you will have your first pipeline as follow:

documentAssembler,
regexTokenizer,
normalized,
...
finisher

You will .fit and .transform this pipeline and save it on disk (parquet is a good format, you can even just select the finisher column to save space on disk). Then load it back by spark.read.parquet and then feed it into the second pipeline which has hashingTf and logreg. The reason is that CV will repeat the stages based on a matrix given by you, so you will be processing the very same documents with the very same results over and over. If the dataset is very very small and your annotators are simple pre-processing you won't notice this but, if you have a real dataset with more complicated annotators such as word embeddings then all that time and computations will be wasted.

That's being said, we will definitely investigate this further as to why the slangDict param not serializable when it is used alongside CrossValidator.