spark-nlp 🚀 - Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader

Could you please paste your full code and the full error stack? Is this Jupyter Python3 kernel?

PS: Please run the following to update to the latest version:

pip install spark-nlp --upgrade

maziyarpanahi on 5 Apr 2020

Thanks for your reply @maziyarpanahi . Below is my code. The same error is coming when I upgrade to the latest version of spark-nlp.

from pyspark.sql import SparkSession
from pyspark.ml import Pipeline
from sparknlp.annotator import *
from sparknlp.common import *

from sparknlp.base import *
import sparknlp
spark = sparknlp.start()

bert = BertEmbeddings.pretrained('bert_base_cased', 'en')\
.setInputCols(["sentence",'token'])\
.setOutputCol("bert")\
.setCaseSensitive(False)\
.setPoolingLayer(0) # default 0

ganga7445 on 6 Apr 2020

Thanks for the code, I still need the full error and the where you are running this code?

maziyarpanahi on 6 Apr 2020

Thanks for your quick reply. below is the full error.I am running this code on jupyter python3.

bert_base_cased download started this may take some time.
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-4-6f2271246c52> in <module>
----> 1 bert = BertEmbeddings.pretrained('bert_base_cased', 'en')\
      2 .setInputCols(["sentence",'token'])\
      3 .setOutputCol("bert")\
      4 .setCaseSensitive(False)\
      5 .setPoolingLayer(0) # default 0

~/anaconda3/lib/python3.7/site-packages/sparknlp/annotator.py in pretrained(name, lang, remote_loc)
   1701     def pretrained(name="bert_base_cased", lang="en", remote_loc=None):
   1702         from sparknlp.pretrained import ResourceDownloader
-> 1703         return ResourceDownloader.downloadModel(BertEmbeddings, name, lang, remote_loc)
   1704 
   1705 

~/anaconda3/lib/python3.7/site-packages/sparknlp/pretrained.py in downloadModel(reader, name, language, remote_loc, j_dwn)
     30     def downloadModel(reader, name, language, remote_loc=None, j_dwn='PythonResourceDownloader'):
     31         print(name + " download started this may take some time.")
---> 32         file_size = _internal._GetResourceSize(name, language, remote_loc).apply()
     33         if file_size == "-1":
     34             print("Can not find the model to download please check the name!")

~/anaconda3/lib/python3.7/site-packages/sparknlp/internal.py in __init__(self, name, language, remote_loc)
    190     def __init__(self, name, language, remote_loc):
    191         super(_GetResourceSize, self).__init__(
--> 192             "com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.getDownloadSize", name, language, remote_loc)
    193 
    194 

~/anaconda3/lib/python3.7/site-packages/sparknlp/internal.py in __init__(self, java_obj, *args)
    127         super(ExtendedJavaWrapper, self).__init__(java_obj)
    128         self.sc = SparkContext._active_spark_context
--> 129         self._java_obj = self.new_java_obj(java_obj, *args)
    130         self.java_obj = self._java_obj
    131 

~/anaconda3/lib/python3.7/site-packages/sparknlp/internal.py in new_java_obj(self, java_class, *args)
    137 
    138     def new_java_obj(self, java_class, *args):
--> 139         return self._new_java_obj(java_class, *args)
    140 
    141     def new_java_array(self, pylist, java_class):

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
     65             java_obj = getattr(java_obj, name)
     66         java_args = [_py2java(sc, arg) for arg in args]
---> 67         return java_obj(*java_args)
     68 
     69     @staticmethod

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.getDownloadSize.
: java.lang.NoSuchMethodError: 'org.json4s.JsonAST$JValue org.json4s.jackson.JsonMethods$.parse(org.json4s.JsonInput, boolean)'
    at com.johnsnowlabs.nlp.pretrained.ResourceMetadata$.parseJson(ResourceMetadata.scala:61)
    at com.johnsnowlabs.nlp.pretrained.ResourceMetadata$$anonfun$readResources$1.applyOrElse(ResourceMetadata.scala:90)
    at com.johnsnowlabs.nlp.pretrained.ResourceMetadata$$anonfun$readResources$1.applyOrElse(ResourceMetadata.scala:89)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at scala.collection.Iterator$$anon$14.next(Iterator.scala:542)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
    at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:183)
    at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
    at scala.collection.AbstractIterator.to(Iterator.scala:1334)
    at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:294)
    at scala.collection.AbstractIterator.toList(Iterator.scala:1334)
    at com.johnsnowlabs.nlp.pretrained.ResourceMetadata$.readResources(ResourceMetadata.scala:92)
    at com.johnsnowlabs.nlp.pretrained.ResourceMetadata$.readResources(ResourceMetadata.scala:84)
    at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.downloadMetadataIfNeed(S3ResourceDownloader.scala:70)
    at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.resolveLink(S3ResourceDownloader.scala:81)
    at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.getDownloadSize(S3ResourceDownloader.scala:159)
    at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.getDownloadSize(ResourceDownloader.scala:394)
    at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader$.getDownloadSize(ResourceDownloader.scala:479)
    at com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.getDownloadSize(ResourceDownloader.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:834)

ganga7445 on 6 Apr 2020

Thanks that really helped, for the last try could you please download the model bert_base_cased from here:
https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/bert_base_cased_en_2.4.0_2.4_1580579557778.zip
Full list of models and pipelines: https://github.com/JohnSnowLabs/spark-nlp-models#english---models

Extract it (required), and load it instead of downloading it:

bert = BertEmbeddings.load('/tmp/bert_base_cased_en_2.4.0_2.4_1580579557778')\
.setInputCols(["sentence",'token'])\
.setOutputCol("bert")\
.setCaseSensitive(False)\
.setPoolingLayer(0)

I want to make sure the proxy, firewall, or internet connection is not causing this.

maziyarpanahi on 6 Apr 2020

👍1

One more information that could really help is the result of these two commands:

import sparknlp
spark = sparknlp.start()

sparknlp.version()
spark.version

maziyarpanahi on 6 Apr 2020

Thanks for the message.even i tried loading offline models too. It is giving other error.
please look at the below error.

Py4JJavaError Traceback (most recent call last)
~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(a, *kw)
62 try:
---> 63 return f(a, *kw)
64 except py4j.protocol.Py4JJavaError as e:

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
327 "An error occurred while calling {0}{1}{2}.\n".
--> 328 format(target_id, ".", name), value)
329 else:

Py4JJavaError: An error occurred while calling o51.load.
: java.lang.IllegalArgumentException: Unsupported class file major version 55
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:166)
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:148)
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:136)
at org.apache.xbean.asm6.ClassReader.(ClassReader.java:237)
at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)
at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:500)
at org.apache.xbean.asm6.ClassReader.readCode(ClassReader.java:2175)
at org.apache.xbean.asm6.ClassReader.readMethod(ClassReader.java:1238)
at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:631)
at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:355)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:307)
at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:306)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:306)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2100)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1409)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.take(RDD.scala:1382)
at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1423)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.first(RDD.scala:1422)
at org.apache.spark.ml.util.DefaultParamsReader$.loadMetadata(ReadWrite.scala:615)
at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:493)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:12)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:8)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:834)

During handling of the above exception, another exception occurred:

IllegalArgumentException Traceback (most recent call last)
in
5 # .setPoolingLayer(0) # default 0
6
----> 7 bert = BertEmbeddings.load('/home/gangadher/Downloads/bert_base_cased_en_2.4.0_2.4_1580579557778')\
8 .setInputCols(["sentence",'token'])\
9 .setOutputCol("bert")\

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/util.py in load(cls, path)
360 def load(cls, path):
361 """Reads an ML instance from the input path, a shortcut of read().load(path)."""
--> 362 return cls.read().load(path)
363
364

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/util.py in load(self, path)
298 if not isinstance(path, basestring):
299 raise TypeError("path should be a basestring, got type %s" % type(path))
--> 300 java_obj = self._jread.load(path)
301 if not hasattr(self._clazz, "_from_java"):
302 raise NotImplementedError("This Java ML type cannot be loaded into Python currently: %r"

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(a, *kw)
77 raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
78 if s.startswith('java.lang.IllegalArgumentException: '):
---> 79 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
80 raise
81 return deco

IllegalArgumentException: 'Unsupported class file major version 55'

ganga7445 on 6 Apr 2020

OK, now I can see the real issue. You need Java 8, you have a different Java version installed or set as default. In our requirements, you can see Java 8 is required because of Apache Spark only supporting Java 8.

$ sudo apt update
$ sudo apt install openjdk-8-jdk


# Check Java version
$ java -version

maziyarpanahi on 6 Apr 2020

great.offline models are loaded when downgraded to openjdk-8-jdk.
another error is coming when I train the model with below code

ner_pipeline = Pipeline(stages = [bert, nerTagger])
ner_model = ner_pipeline.fit(training_data)

error is:

Py4JJavaError Traceback (most recent call last)
~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(a, *kw)
62 try:
---> 63 return f(a, *kw)
64 except py4j.protocol.Py4JJavaError as e:

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
327 "An error occurred while calling {0}{1}{2}.\n".
--> 328 format(target_id, ".", name), value)
329 else:

: (, Py4JNetworkError('An error occurred while trying to connect to the Java server (127.0.0.1:42581)'))

During handling of the above exception, another exception occurred:

Py4JError Traceback (most recent call last)
in
1 ner_pipeline = Pipeline(stages = [bert, nerTagger])
----> 2 ner_model = ner_pipeline.fit(training_data)

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/base.py in fit(self, dataset, params)
130 return self.copy(params)._fit(dataset)
131 else:
--> 132 return self._fit(dataset)
133 else:
134 raise ValueError("Params must be either a param map or a list/tuple of param maps, "

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/pipeline.py in _fit(self, dataset)
107 dataset = stage.transform(dataset)
108 else: # must be an Estimator
--> 109 model = stage.fit(dataset)
110 transformers.append(model)
111 if i < indexOfLastEstimator:

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/base.py in fit(self, dataset, params)
130 return self.copy(params)._fit(dataset)
131 else:
--> 132 return self._fit(dataset)
133 else:
134 raise ValueError("Params must be either a param map or a list/tuple of param maps, "

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/wrapper.py in _fit(self, dataset)
293
294 def _fit(self, dataset):
--> 295 java_model = self._fit_java(dataset)
296 model = self._create_model(java_model)
297 return self._copyValues(model)

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/ml/wrapper.py in _fit_java(self, dataset)
290 """
291 self._transfer_params_to_java()
--> 292 return self._java_obj.fit(dataset._jdf)
293
294 def _fit(self, dataset):

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(a, *kw)
63 return f(a, *kw)
64 except py4j.protocol.Py4JJavaError as e:
---> 65 s = e.java_exception.toString()
66 stackTrace = '\n\t at '.join(map(lambda x: x.toString(),
67 e.java_exception.getStackTrace()))

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(a, *kw)
61 def deco(a, *kw):
62 try:
---> 63 return f(a, *kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()

~/Downloads/spark-2.4.5-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
334 raise Py4JError(
335 "An error occurred while calling {0}{1}{2}".
--> 336 format(target_id, ".", name))
337 else:
338 type = answer[1]

Py4JError: An error occurred while calling o346.toString

ganga7445 on 7 Apr 2020

Same problem here with PretrainedPipeline . I can download a model manually and load it, but do I have to download each model to create a pipeline? (annotator, tokenizer, stemmer, etc.?) Thanks for the info. I was wondering if there was any easy way to compare similarities between tho texts, and thought only with bert model would be enough.

Thanks

iuria21 on 4 Sep 2020

Hi @basque21

Could you please create a new issue with the complete template and steps to reproduce your issue? (to answer your questions, we have pretrained pipelines which come with required models altogether, but some users want flexibility and freedom to choose what annotator should be inside that pipeline so they use pretrained models and build their own custom pipelines. It is explained here and it's a very good practice: https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/3.SparkNLP_Pretrained_Models.ipynb)

maziyarpanahi on 4 Sep 2020

Facing the same Problem.
is there any solution available now??

55Ankur55 on 12 Sep 2020

downgrade your spark to 2.4.6 instead of 3.0

hammedb197 on 13 Oct 2020

👍1

A number of things can cause this issue, from the Internet, proxy, firewall, incompatible Pyspark version, Python version, etc.
In order to help we need the complete template to have as much information to reproduce this and help.

I am closing this issue as it is stalling since April and the other comments without any info are not helping. Please create a new issue with a complete template and it will be resolved with given information.

maziyarpanahi on 13 Oct 2020

👍1

Spark-nlp: Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader

Description

code

Your Environment

All 14 comments

Related issues