Spark-nlp: java.util.NoSuchElementException: Param detectLists does not exist in SentenceDetector

[chandramohanreddyab]

Description

<!--- Provide a more detailed introduction to the issue itself, and why you consider it to be a bug -->
Traceback (most recent call last) <ipython-input-13-43850c69908b> in <module>
----> 1 sent_df = sentenceDetector.transform(doc_df)

~/spark-2.3.0-bin-hadoop2.7/python/pyspark/ml/base.py in transform(self, dataset, params)
      171                 return self.copy(params)._transform(dataset)
      172             else:
--> 173                 return self._transform(dataset)
      174         else:
      175             raise ValueError("Params must be a param map but got %s." % type(params))

~/spark-2.3.0-bin-hadoop2.7/python/pyspark/ml/wrapper.py in _transform(self, dataset)
      302 
      303     def _transform(self, dataset):
--> 304         self._transfer_params_to_java()
      305         return DataFrame(self._java_obj.transform(dataset._jdf), dataset.sql_ctx)
      306 

~/spark-2.3.0-bin-hadoop2.7/python/pyspark/ml/wrapper.py in _transfer_params_to_java(self)
      122         for param in self.params:
      123             if param in paramMap:
--> 124                 pair = self._make_java_param_pair(param, paramMap[param])
      125                 self._java_obj.set(pair)
      126 

~/spark-2.3.0-bin-hadoop2.7/python/pyspark/ml/wrapper.py in _make_java_param_pair(self, param, value)
      111         sc = SparkContext._active_spark_context
      112         param = self._resolveParam(param)
--> 113         java_param = self._java_obj.getParam(param.name)
      114         java_value = _py2java(sc, value)
      115         return java_param.w(java_value)

~/spark-2.3.0-bin-hadoop2.7/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in __call__(self, *args)
      1158         answer = self.gateway_client.send_command(command)
      1159         return_value = get_return_value(
-> 1160             answer, self.gateway_client, self.target_id, self.name)
     1161 
     1162         for temp_arg in temp_args:

~/spark-2.3.0-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(*a, **kw)
       61     def deco(*a, **kw):
       62         try:
--->63             return f(*a, **kw)
       64         except py4j.protocol.Py4JJavaError as e:
       65             s = e.java_exception.toString()

~/spark-2.3.0-bin-hadoop2.7/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
       318                 raise Py4JJavaError(
       319                     "An error occurred while calling {0}{1}{2}.\n".
--> 320                     format(target_id, ".", name), value)
       321             else:
       322                 raise Py4JError(

Py4JJavaError: An error occurred while calling o177.getParam.
: java.util.NoSuchElementException: Param detectLists does not exist.
    at org.apache.spark.ml.param.Params$$anonfun$getParam$2.apply(params.scala:729)
    at org.apache.spark.ml.param.Params$$anonfun$getParam$2.apply(params.scala:729)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.ml.param.Params$class.getParam(params.scala:728)
    at org.apache.spark.ml.PipelineStage.getParam(Pipeline.scala:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)`

Expected Behavior

SparkNLP 2.6 should be compatible with spark 2.3 version

Current Behavior

Annotators and Pretained models are not getting executed

Possible Solution

No idea, because existing jupyter notebooks with SparkNLP 2.5 are running successfully on spark 2.3.0

Steps to Reproduce

1. Set up spark 2.3.0
2. Install SparkNLP 2.6
3. Run pretained models or Annotator like Sentence detector
4. https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/2.Text_Preprocessing_with_SparkNLP_Annotators_Transformers.ipynb

Context

Trying to execute jupyter notebooks with SparkNLP 2.6 on spark 2.3.0

Your Environment

• Spark NLP version: 2.6
• Java version (java -version): 1.8
• Setup and installation (Pypi, Conda, Maven, etc.): Conda

[maziyarpanahi]

Thanks @chandramohanreddyab for reporting this. I am going to reproduce this myself, where in that notebook this error happens?

[maziyarpanahi]

@chandramohanreddyab I am sorry but I cannot reproduce this. I can successfully execute the entire notebook with the following changes:

# Install pyspark
! pip install --ignore-installed -q pyspark==2.3.0
! pip install --ignore-installed -q spark-nlp==2.6.0

There are some issues with some of the PySpark functions from 2.4.x not being available in 2.3.x like arrays_zip but that is not from Spark NLP, you need to use whatever it's available for PySpark in 2.3.x.

[chandramohanreddyab]

Thanks @chandramohanreddyab for reporting this. I am going to reproduce this myself, where in that notebook this error happens?

Thanks @maziyarpanahi for prompt reply. Kindly check out these notebook https://github.com/chandramohanreddyab/NLP/blob/master/2.Text_Preprocessing_with_SparkNLP_Annotators_Transformers.ipynb

https://github.com/chandramohanreddyab/NLP/blob/master/1.%20Basic_Linguistic_Algorithms_PreTrainedModels.ipynb

[C-K-Loan]

@maziyarpanahi @chandramohanreddyab
I ran into a similar bug,

here is a minimal Collab example which reproduces this issue for the PretrainedPipeline for recognize_entities_bert.

I think they must be related, since this only started occurring with 2.6

[maziyarpanahi]

I think you forgot the Link to the Colab.

[C-K-Loan]

Oops my bad there you go
https://colab.research.google.com/drive/1zvylfhxwvZafzO4__m1NPLUUTQBfvwZD?usp=sharing

[maziyarpanahi]

@chandramohanreddyab
OK, this is a notebook I just did by using the built-in sparknlp.start() function. Thankfully, all the cells passed including the pretrained models and pipelines.

I think there must be something wrong with the Fat JARs. I am going to check that as well
Apache Spark 2.3.x-2.6.0.ipynb.zip

@C-K-Loan Your issue is something else. I have refactored the BertEmbeddings and removed the poolingLayer param. That pipeline should be reuploaded with the new changes so it won't complain about not having that parameter. (I'll do it now and it should be fixed)

[maziyarpanahi]

@chandramohanreddyab I found a problem. It's not really about PySpark 2.3.x and it's about the difference between Spark NLP 2.5.5 and 2.6.0 in SentenceDetector. In 2.6.0 the SetencenDetector has a new param called detectLists. Your error suggests you are having a mismatch between your PyPI and Java versions.

Here is the new notebook with your code which works as you can see:
Apache Spark 2.3.x (1).ipynb.zip

You may see you have 2.6.0 installed in PyPI, but if you load the 2.5.5 Fat JAR you will get this error as this param doesn't exist in that version. I highly suggest to first uninstall the PyPI and reinstall it. Also, remove anything about 2.5.5 from your SparkSession and make sure you restart the session/interrupt it and start clean with the correct Fat JAR from 2.6.0 to not have any mismatch version.

[chandramohanreddyab]

@chandramohanreddyab I found a problem. It's not really about PySpark 2.3.x and it's about the difference between Spark NLP 2.5.5 and 2.6.0 in SentenceDetector. In 2.6.0 the SetencenDetector has a new param called detectLists. Your error suggests you are having a mismatch between your PyPI and Java versions.

Here is the new notebook with your code which works as you can see:
Apache Spark 2.3.x (1).ipynb.zip

You may see you have 2.6.0 installed in PyPI, but if you load the 2.5.5 Fat JAR you will get this error as this param doesn't exist in that version. I highly suggest to first uninstall the PyPI and reinstall it. Also, remove anything about 2.5.5 from your SparkSession and make sure you restart the session/interrupt it and start clean with the correct Fat JAR from 2.6.0 to not have any mismatch version.

@maziyarpanahi yeah,You are right. I have reinstalled the PyPI and added new fat jar. Now all the cells passed successfully 💯 . Thank your support 👍

[AntonSmuit]

i have same problem here, upgraded from spark-nlp 2.5.5 to 2.6.1 and keep getting this error. i removed ivy java cache and reinstall spark-nlp with pip, but it keep getting jar from 2.5.5 version. where is this dependency hardcoded? can i tell spark nlp to load correct jars?
project run from IntelliJ IDEA from local pc.

[maziyarpanahi]

Hi @AntonSmuit

Could you please create a new issue. The issue comes with a template, we need all that information in order to help.

Many thanks

PS: we don't hardcode anything, it's your IDE caching stuff. I suggest restarting/invalidate and then refresh SBT to be sure everything is the same version in addition to PyPI via pip install if this is Python IDE otherwise no need for Intellij and Scala