Spark-nlp: Bert Embeddings offline loading with AttributeError

Created on 10 Nov 2020 · 6Comments · Source: JohnSnowLabs/spark-nlp

I am trying to load Bert Sentence embedding with bert_embedding = BertEmbeddings.load("HDFS MODEL PATH").
And I had AttributeError: module 'sparknlp.embeddings' has no attribute 'BertEmbeddings'. Due to constraints on my working environment, I could not have SparkNLP to download the public model for me, and I have to load the model from my HDFS server.
Thank you in advance.

Description

I downloaded offline model from models#multi-language. For testing purpose, I downloaded BertEmbeddings (multi_cased) | bert_multi_cased | 2.4.0 | xx.
I upzipped and uploaded this model to HDFS.
I try to load this model with bert_embedding = BertEmbeddings.load("HDFS MODEL PATH") and I had AttributeError: module 'sparknlp.embeddings' has no attribute 'BertEmbeddings' while executing this line.

Expected Behavior

The BertEmbedding model should be loaded correctly.

Current Behavior

AttributeError: module 'sparknlp.embeddings' has no attribute 'BertEmbeddings'

Possible Solution

N/A

Steps to Reproduce

Download BertEmbeddings (multi_cased) from https://nlp.johnsnowlabs.com/docs/en/models#multi-language
Unzip it and upload it to HDFS
The code snippets below:

import pyspark
from pyspark.sql import SparkSession
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.embeddings import *

spark = SparkSession.builder.config("spark.driver.extraClassPath", "/usr/spark/spark-nlp/jars/spark-nlp-assembly-2.6.2.jar").config("spark.jars", "/usr/spark/spark-nlp/jars/spark-nlp-assembly-2.6.2.jar").getOrCreate()

# offline load pretrained pipeline, this worked though
pipeline = PretrainedPipeline.from_disk('hdfs:///run/MODELE_SPARKNLP/explain_document_lg_fr_2.4.0_2.4_1579709189947')
pipeline.annotate('Je ne sais pas!')
>>> {'entities': ['Je ne sais pas!'],
 'lemma': ['Je', 'ne', 'savoir', 'pas', '!'],
 'document': ['Je ne sais pas!'],
 'pos': ['PRON', 'ADV', 'VERB', 'ADV', 'PUNCT'],
 'token': ['Je', 'ne', 'sais', 'pas', '!'],
 'ner': ['I-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'I-MISC'],
 'embeddings': ['Je', 'ne', 'sais', 'pas', '!'],
 'sentence': ['Je ne sais pas!']}

# offline load BERT embeddings
bert_embedding = BertEmbeddings.load('/run/MODELE_SPARKNLP/bert_multi_cased_xx_2.6.0_2.4_1598341875191')
>>>---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-51-466d327eee34> in <module>
----> 1 bert_embedding = BertEmbeddings.load('/run/MODELE_SPARKNLP/bert_multi_cased_xx_2.6.0_2.4_1598341875191')

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/pyspark/ml/util.py in load(cls, path)
    360     def load(cls, path):
    361         """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
--> 362         return cls.read().load(path)
    363 
    364 

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/pyspark/ml/util.py in load(self, path)
    302             raise NotImplementedError("This Java ML type cannot be loaded into Python currently: %r"
    303                                       % self._clazz)
--> 304         return self._clazz._from_java(java_obj)
    305 
    306     def context(self, sqlContext):

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/pyspark/ml/wrapper.py in _from_java(java_stage)
    225         stage_name = java_stage.getClass().getName().replace("org.apache.spark", "pyspark")
    226         # Generate a default new instance from the stage_name class.
--> 227         py_type = __get_class(stage_name)
    228         if issubclass(py_type, JavaParams):
    229             # Load information from java_stage to the instance.

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/pyspark/ml/wrapper.py in __get_class(clazz)
    221             m = __import__(module)
    222             for comp in parts[1:]:
--> 223                 m = getattr(m, comp)
    224             return m
    225         stage_name = java_stage.getClass().getName().replace("org.apache.spark", "pyspark")

AttributeError: module 'sparknlp.embeddings' has no attribute 'BertEmbeddings'

Context

I cannot load Bert offline for downstream tasks with SparkNLP.

Your Environment

Spark NLP version: 2.6.3
PySpark version: 2.4.3
Apache NLP version: N/A
Java version (java -version): java version "1.8.0_192" (I am not sure)
Setup and installation (Pypi, Conda, Maven, etc.): Pypi
Operating System and version: Linux
Link to your project (if any): N/A
I have trouble in having all the env info, because I am working in Dataiku.

question

Source

sharockys

All 6 comments

Hi @sharockys,

Please remove from sparknlp.embeddings import * from your code. We never mentioned that in any of our examples, only keep the other two imports, please.

PS: BertEmbeddings and its models are for token-based embeddings, each token will have a vector. For sentence embeddings, you either need UniversalSentenceEncoder or BertSentenceEmbeddingd. In addition, you can use SentenceEmbeddings to average any word embeddings.

maziyarpanahi on 10 Nov 2020

Thank you very much. By removing the import with sparknlp.embeddings, it did resolve the problem.
But I don't understand why, does the import messed up the namespace? It would be nice if you could help me to understand this, please.

PS, I noticed that you are in France, tomorrow is jour férié, have a nice day off!

sharockys on 10 Nov 2020

❤1

I am trying Sentence embeddings this time.
While I was using BertSentenceEmbeddings, I have another error.
Is this is about the proxy? Why do we need proxy this time to load sentence embedding?
Thank you very much for your help.

bert_embedding_sent = BertSentenceEmbeddings.load('/run/MODELE_SPARKNLP/sent_bert_multi_cased_xx_2.6.0_2.4_1598347692999')
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-16-2db93d340251> in <module>
----> 1 bert_embedding = BertSentenceEmbeddings.load('/run/MODELE_SPARKNLP/sent_bert_multi_cased_xx_2.6.0_2.4_1598347692999')

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/pyspark/ml/util.py in load(cls, path)
    360     def load(cls, path):
    361         """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
--> 362         return cls.read().load(path)
    363 
    364 

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/pyspark/ml/util.py in load(self, path)
    298         if not isinstance(path, basestring):
    299             raise TypeError("path should be a basestring, got type %s" % type(path))
--> 300         java_obj = self._jread.load(path)
    301         if not hasattr(self._clazz, "_from_java"):
    302             raise NotImplementedError("This Java ML type cannot be loaded into Python currently: %r"

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/home/dataiku/spark-2.4.3-bin-hadoop2.6/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o831.load.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 35.0 failed 4 times, most recent failure: Lost task 0.3 in stage 35.0 (TID 105, 10.155.20.70, executor 15): java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
    at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:668)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:604)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2598)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1866)
    at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:721)
    at org.apache.spark.util.Utils$.fetchFile(Utils.scala:496)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$3.apply(Executor.scala:799)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$3.apply(Executor.scala:796)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:796)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:375)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: nameservice1
    ... 28 more

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
    at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1364)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
    at org.apache.spark.rdd.RDD.take(RDD.scala:1337)
    at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1378)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
    at org.apache.spark.rdd.RDD.first(RDD.scala:1377)
    at org.apache.spark.ml.util.DefaultParamsReader$.loadMetadata(ReadWrite.scala:615)
    at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:493)
    at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:12)
    at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:8)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
    at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:668)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:604)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2598)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1866)
    at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:721)
    at org.apache.spark.util.Utils$.fetchFile(Utils.scala:496)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$3.apply(Executor.scala:799)
    at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$3.apply(Executor.scala:796)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:796)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:375)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more
Caused by: java.net.UnknownHostException: nameservice1
    ... 28 more

sharockys on 10 Nov 2020

The model is very big, you are probably running out of memory (especially the Driver). Make sure you have enough available memory to use these big models and the configs like spark.driver.memory is correctly set.

maziyarpanahi on 10 Nov 2020

The model is very big, you are probably running out of memory (especially the Driver). Make sure you have enough available memory to use these big models and the configs like spark.driver.memory is correctly set.

Thank you very much, I will try to reach out my collegues to retry with more memory.

sharockys on 12 Nov 2020

Thank you very much! Indeed it's the memory issue. I can load these models correctly now!

sharockys on 21 Nov 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings