Spark-nlp: AttributeError: 'Tokenizer' object has no attribute '_java_obj'

[rayli-anti]

import sparknlp
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.common import *
from sparknlp.embeddings import *

import sparknlp
spark = sparknlp.start()
tokenizer = Tokenizer().setInputCols(["doc"]).setOutputCol("token_sn")

word_embeddings = BertEmbeddings.pretrained('bert_base_cased', 'en') \
.setInputCols(["doc", "token_sn"]) \
.setOutputCol("embeddings")

bert_pipeline = Pipeline().setStages(
[
# document_assembler,
tokenizer,
word_embeddings
]
)
df_bert = bert_pipeline.fit(ps_df).transform(ps_df)
display(df_bert)

Traceback (most recent call last):
File "/Users/RuizLi/Desktop/BI_Online/Ray/ctv_recsys/src/contain-based.py", line 66, in
tokenizer = Tokenizer().setInputCols(["doc"]).setOutputCol("token_sn")
File "/usr/local/opt/apache-spark/libexec/python/pyspark/__init__.py", line 110, in wrapper
return func(self, *kwargs)
File "/anaconda3/envs/recsys/lib/python3.7/site-packages/sparknlp/annotator.py", line 168, in __init__
super(Tokenizer, self).__init__(classname="com.johnsnowlabs.nlp.annotators.Tokenizer")
File "/usr/local/opt/apache-spark/libexec/python/pyspark/__init__.py", line 110, in wrapper
return func(self, *kwargs)
File "/anaconda3/envs/recsys/lib/python3.7/site-packages/sparknlp/common.py", line 155, in __init__
self._java_obj = self._new_java_obj(classname, self.uid)
File "/usr/local/opt/apache-spark/libexec/python/pyspark/ml/wrapper.py", line 69, in _new_java_obj
return java_obj(*java_args)
TypeError: 'JavaPackage' object is not callable
Exception ignored in:
Traceback (most recent call last):
File "/usr/local/opt/apache-spark/libexec/python/pyspark/ml/wrapper.py", line 42, in __del__
AttributeError: 'Tokenizer' object has no attribute '_java_obj'

[maziyarpanahi]

• Cloud you please complete the full template, we need all that information.
• You need DocumentAssembler before Tokenizer, I see you don't have it and it's commented out. Please make sure you have it the same way we have it in all of our examples. (it's the first and mandatory annotator of any pipeline)

[rayli-anti]

After added document_assembler, I got the following:

Traceback (most recent call last):
File "/Users/RuizLi/Desktop/BI_Online/Ray/ctv_recsys/src/contain-based.py", line 65, in
document_assembler = DocumentAssembler()\
File "/usr/local/opt/apache-spark/libexec/python/pyspark/__init__.py", line 110, in wrapper
return func(self, *kwargs)
File "/anaconda3/envs/recsys/lib/python3.7/site-packages/sparknlp/base.py", line 148, in __init__
super(DocumentAssembler, self).__init__(classname="com.johnsnowlabs.nlp.DocumentAssembler")
File "/usr/local/opt/apache-spark/libexec/python/pyspark/__init__.py", line 110, in wrapper
return func(self, *kwargs)
File "/anaconda3/envs/recsys/lib/python3.7/site-packages/sparknlp/internal.py", line 72, in __init__
self._java_obj = self._new_java_obj(classname, self.uid)
File "/usr/local/opt/apache-spark/libexec/python/pyspark/ml/wrapper.py", line 69, in _new_java_obj
return java_obj(*java_args)
TypeError: 'JavaPackage' object is not callable

java version: 1.8.0_202
spark version: 3.0.0
spark-nlp version: 2.5.5

[maziyarpanahi]

Now that you have completed the template I can say that we don't support Apache Spark 3.0.0 (but you still need the DocumentAssembler if you use the support Apache Spark):

https://github.com/JohnSnowLabs/spark-nlp#apache-spark-support

[rayli-anti]

@maziyarpanahi Thanks a lot. I did all the steps in quick start and installed pyspark 2.4.4, but it seems in this new virtual env, my python is still calling spark version 3.0.0. Do you know how to change it?

[maziyarpanahi]

I am sorry you are still missing the template in order for me to help. Please copy it and fill all the parts in order to understand and reproduce.

[rayli-anti]

problem solved. I changed findspark.init() point to spark 2.4.4. Another question, spark = sparknlp.start(), can this session be used to read csv?

[maziyarpanahi]

spark=sparknlp.start() will start or get an existing SparkSession, so the spark will be your actual SparkSession. Whatever you can do with spark like spark.read.csv, spark.read.json etc. you can do it with the same spark.

Here is the code behind it, we just create SparkSession including Spark NLP packages and required configs, or it will get the existing session if it exists:
https://github.com/JohnSnowLabs/spark-nlp/blob/d02dd58a195f003501bb03134c387e6e349a22a6/python/sparknlp/__init__.py#L45

[rayli-anti]

Thanks, that is really helpful. BTW, is there a way to generate sentence embedding by using BertEmbeddings?

[maziyarpanahi]

There will be dedicated BertSentenceEmbeddings in the next release, however, we have SentenceEmbeddings which takes any word embeddings (glove, bert, albert, xlnet, etc.) and average them into sentence embeddings.

This example shows possible Sentence Embeddings for multi-class text classification: https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/5.Text_Classification_with_ClassifierDL.ipynb