spark-nlp 2.4.5 BertEmbeddings.load(localpath) throws java.lang.NoClassDefFoundError

Created on 21 Apr 2020  路  15Comments  路  Source: JohnSnowLabs/spark-nlp


spark-nlp 2.4.5 BertEmbeddings.load(localpath) throws java.lang.NoClassDefFoundError

Description

Using bert_base_cased embedding with ner_dl_bert_en_2.4.0_2.4_1583223672963 works fine with spark-nlp 2.4.4, but after upgraded to spark-nlp 2.4.5, it throws the the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: com/typesafe/config/ConfigMergeable
at com.johnsnowlabs.util.ConfigHelper$.getConfigValue(ConfigHelper.scala:13)
at com.johnsnowlabs.util.ConfigHelper$.getConfigValueOrElse(ConfigHelper.scala:20)
at com.johnsnowlabs.util.ConfigHelper$.(ConfigHelper.scala:47)
at com.johnsnowlabs.util.ConfigHelper$.(ConfigHelper.scala)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.cacheFolder(ResourceDownloader.scala:63)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$$anonfun$3.apply(ResourceDownloader.scala:123)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$$anonfun$3.apply(ResourceDownloader.scala:123)
at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.(S3ResourceDownloader.scala:30)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.(ResourceDownloader.scala:123)
at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.(ResourceDownloader.scala)
at com.johnsnowlabs.nlp.HasPretrained$class.$init$(HasPretrained.scala:16)
at com.johnsnowlabs.nlp.annotator.package$BertEmbeddings$.(annotator.scala:132)
at com.johnsnowlabs.nlp.annotator.package$BertEmbeddings$.(annotator.scala)

Expected Behavior

Upgrade to 2.4.5 shouldn't make it to throw that exception

Context

Testing the performance of SparkNLP

Your Environment

Spark NLP version: 2.4.4
Apache NLP version: 2.4.4
Java version (java -version): 1.8.0_162
Setup and installation (Pypi, Conda, Maven, etc.): Maven
Operating System and version: CentOS 7.7
Requires more input question
Source
picture simon-oz

Most helpful comment

@simon-oz thanks for the details. I quickly tried the same code in scala notebook and local spark cluster and it works both for 2.4.4 and 2.4.5. I need to check it once in yarn cluster too. Now what I suspect is the uber-jar.
Based on the error it is clear the class ConfigMergeable from com.typesafe.config is not found. ConfigMergeable class is part of typesafe config from the beginning so it should not be some version issue.

Can you help us with one more information: Can you check in the uber jar with 2.4.4 and 2.4.5 if com.typesafe.config jar is included? or maybe run the below command to compare the output for the uber jar you create.

jar tf jarFile | grep "typesafe/confiConfigMergeable"

and also can you add this dependency i.e com.typesafe.config 1.3.0 and build the uber jar and try again.

What is surprising is why it works on 2.4.4 and not 2.4.5 !!

Thanks

picture rohit-nlp on 22 Apr 2020
👍2

All 15 comments

We have made some changes in the config loader to make the library compatible inside the cluster. It shouldn't be any problem as all the tests have passed.
Could you please provide your full code, how you start the SparkSession so we can reproduce it here?

maziyarpanahi picture maziyarpanahi on 21 Apr 2020

@simon-oz Thanks for bringing this to our notice. as @maziyarpanahi pointed we did some changes with config loaders in pre-trained models to make it compatible with yarn cluster mode. but that should not have this effect and all the test passed. So if you could share more details on how you run the pre-trained model and how you create the spark session? is it on a notebook or some IDE and how you integrate the spark nlp package so that we can replicate it locally. Thanks

rohit-nlp picture rohit-nlp on 21 Apr 2020
👍1

Hi thanks for your reply.

Following are spark-submit command and the code.
In my pom.xml, when use sparknlp 2.4.4, it works, but when change the version to 2.4.5, it throws exception as aforementioined.

  • command:

spark-submit --class sis.uni.NerBertpipeline \
--master yarn --deploy-mode client \
--driver-memory 4g \
--executor-memory 4g \
./target/uber-johnsnow-1.0-SNAPSHOT.jar

  • Code:

import org.apache.spark.sql.{ SparkSession, DataFrame, SQLContext, Dataset, Row }
import com.johnsnowlabs.nlp.base._
import com.johnsnowlabs.nlp.annotator._
import org.apache.spark.ml.Pipeline

object NerBertpipeline {

def main(args: Array[String]) {

val spark = SparkSession
  .builder
  .appName("ner bert test")
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .getOrCreate()

spark.sparkContext.setLogLevel("WARN")

def ROOTPATH = "/my/hdfs/root/"

def BERTBASECASED = "bert_base_cased_en_2.4.0_2.4_1580579557778"
def PATTERNBERT = "ner_dl_bert_en_2.4.0_2.4_1583223672963"

import spark.implicits._

val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("document")

val sentenceDetector = new SentenceDetector().
  setInputCols(Array("document")).
  setOutputCol("sentence")

val tokenizer = new Tokenizer()
  .setInputCols(Array("sentence"))
  .setOutputCol("token")

val normalizer = new Normalizer()
  .setInputCols("token")
  .setOutputCol("normal")

val bertEmbeddings = BertEmbeddings.load(ROOTPATH + BERTBASECASED)
  .setInputCols(Array("sentence", "token"))
  .setOutputCol("bertEmbd")
  .setCaseSensitive(true)

val nerBertModel = NerDLModel.load(ROOTPATH + PATTERNBERT)
  .setInputCols(Array("sentence", "token", "bertEmbd"))
  .setOutputCol("nerBert")

val nerConverter = new NerConverter()
  .setInputCols(Array("document", "token", "nerBert"))
  .setOutputCol("ner_converter")

val finisher = new Finisher()
  .setInputCols("nerBert", "ner_converter")
  .setIncludeMetadata(true)
  .setOutputAsArray(false)
  .setCleanAnnotations(false)
  .setAnnotationSplitSymbol("@")
  .setValueSplitSymbol("#")

val testing = Seq(
  (1, "Google is a famous company"),
  (2, "Peter Parker is a super heroe")).toDS.toDF("_id", "text")

val stages = (Array(
  documentAssembler,
  sentenceDetector,
  tokenizer,
  bertEmbeddings,
  nerBertModel,
  nerConverter,
  finisher))

val pipeline = new Pipeline().setStages(stages)

val result = pipeline.fit(Seq.empty[String].toDS.toDF("text")).transform(testing)

result.printSchema()

}
}

simon-oz picture simon-oz on 22 Apr 2020

@simon-oz thanks for the details. I quickly tried the same code in scala notebook and local spark cluster and it works both for 2.4.4 and 2.4.5. I need to check it once in yarn cluster too. Now what I suspect is the uber-jar.
Based on the error it is clear the class ConfigMergeable from com.typesafe.config is not found. ConfigMergeable class is part of typesafe config from the beginning so it should not be some version issue.

Can you help us with one more information: Can you check in the uber jar with 2.4.4 and 2.4.5 if com.typesafe.config jar is included? or maybe run the below command to compare the output for the uber jar you create.

jar tf jarFile | grep "typesafe/confiConfigMergeable"

and also can you add this dependency i.e com.typesafe.config 1.3.0 and build the uber jar and try again.

What is surprising is why it works on 2.4.4 and not 2.4.5 !!

Thanks

rohit-nlp picture rohit-nlp on 22 Apr 2020
👍2

Hi rohit-nlp,

Very sorry for the late reply.

  1. I included com.typesafe.config 1.3.0 in pom.xml, there are still the same issue
    The exception was threw at BertEmbeddings.load(ROOTPATH24 + BERTLARGECASED).

  2. Run jar tf myfatjar.jar | grep "typesafe/confiConfigMergeable", both 2.4.4 and 2.4.5 return nothing.

  3. following are maven dependency:tree for 2.4.4 and 2.4.5, we can see that in 2.4.4, com.typesafe:config:jar:1.3.0:compile is included, but com.typesafe:config:jar:1.3.0:compile doesn't in 2.4.5.

2.4.4
[INFO] +- com.johnsnowlabs.nlp:spark-nlp_2.11:jar:2.4.4:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] | +- com.typesafe:config:jar:1.3.0:compile
[INFO] | +- org.rocksdb:rocksdbjni:jar:6.5.3:compile
[INFO] | +- org.apache.hadoop:hadoop-aws:jar:3.2.0:compile

2.4.5
[INFO] +- com.johnsnowlabs.nlp:spark-nlp_2.11:jar:2.4.5:compile
[INFO] +- com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:jar:2.2.1:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] | - org.apache.pdfbox:jbig2-imageio:jar:3.0.2:compile

Hope that helps

simon-oz picture simon-oz on 24 Apr 2020

Thanks @simon-oz this is very helpful, for you to solve the problem just add the com.typesafe:config:jar:1.3.0 jar in your spark driver and executor as dependency jar and it would work. Please let me know if it did. Regarding the maven dependency, I will check in 2.4.5 thanks.

rohit-nlp picture rohit-nlp on 28 Apr 2020

@simon-oz just one more clarification how are you resolving the spark nlp dependencies currently? I checked on maven repo and the typesafe config is present in both 2.4.4 and 2.4.5. And actually I get the typesfeconfig when I resolve using sbt in my local project. Am curious to see why it is not present in your maven dependency tree
https://search.maven.org/artifact/com.johnsnowlabs.nlp/spark-nlp_2.11/2.4.5/jar

https://search.maven.org/artifact/com.johnsnowlabs.nlp/spark-nlp_2.11/2.4.4/jar

rohit-nlp picture rohit-nlp on 28 Apr 2020

Hello @rohit-nlp

Thanks for your reply on the 2.4.5 issue.

Please see the full output of maven dependency tree of spark-nlp 2.4.4 and 2.4.5 in our project.

Following is the dependency tree of spark-nlp 2.4.4
[INFO] +- com.johnsnowlabs.nlp:spark-nlp_2.11:jar:2.4.4:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] | +- com.typesafe:config:jar:1.3.0:compile
[INFO] | +- org.rocksdb:rocksdbjni:jar:6.5.3:compile
[INFO] | +- org.apache.hadoop:hadoop-aws:jar:3.2.0:compile
[INFO] | +- com.amazonaws:aws-java-sdk-core:jar:1.11.603:compile
[INFO] | | +- software.amazon.ion:ion-java:jar:1.0.2:compile
[INFO] | | - com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.6.7:compile
[INFO] | +- com.amazonaws:aws-java-sdk-s3:jar:1.11.603:compile
[INFO] | | +- com.amazonaws:aws-java-sdk-kms:jar:1.11.603:compile
[INFO] | | - com.amazonaws:jmespath-java:jar:1.11.603:compile
[INFO] | +- com.github.universal-automata:liblevenshtein:jar:3.0.0:compile
[INFO] | | +- com.google.code.findbugs:annotations:jar:3.0.1:runtime
[INFO] | | +- com.google.protobuf:protobuf-java-util:jar:3.0.0-beta-3:runtime
[INFO] | | +- it.unimi.dsi:fastutil:jar:7.0.12:runtime
[INFO] | | - org.projectlombok:lombok:jar:1.16.8:runtime
[INFO] | +- com.navigamez:greex:jar:1.0:compile
[INFO] | | - dk.brics.automaton:automaton:jar:1.11-8:compile
[INFO] | +- org.json4s:json4s-ext_2.11:jar:3.5.3:compile
[INFO] | | - org.joda:joda-convert:jar:1.8.1:compile
[INFO] | +- org.tensorflow:tensorflow:jar:1.15.0:compile
[INFO] | | +- org.tensorflow:libtensorflow:jar:1.15.0:compile
[INFO] | | - org.tensorflow:libtensorflow_jni:jar:1.15.0:compile
[INFO] | - net.sf.trove4j:trove4j:jar:3.0.3:compile
[INFO] +- com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:jar:2.2.1:compile
[INFO] | - org.apache.pdfbox:jbig2-imageio:jar:3.0.2:compile

But for 2.4.5, the dependency tree only output the following two lines:

[INFO] +- com.johnsnowlabs.nlp:spark-nlp_2.11:jar:2.4.5:compile
[INFO] +- com.johnsnowlabs.nlp:spark-nlp-ocr_2.11:jar:2.2.1:compile

So if I manually added com.typesafe:config:jar:1.3.0 jar, it only add that single one, but I still miss the rest on the dependency tree, as listed above.

I checked the size of the jars for 2.4.4 and 2.4.5, there are about the same. So it's strange to have this issue.

simon-oz picture simon-oz on 29 Apr 2020

Hi @rohit-nlp ,

I am working on a maven project and based on Nexus repository for all the dependencies. If a package is not included in Nexus, I will install it to the repository.

Regards,

simon-oz picture simon-oz on 29 Apr 2020

Hi @simon-oz the size of the jar will be same as the dependencies are not included in the jar. It is added in the dependency tree and resolved by maven at build time.

One way you can make sure the dependency is added by Spark for Spark NLP is adding the package directly when you do your spark-submit job. by adding
--packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5

rohit-nlp picture rohit-nlp on 29 Apr 2020

@simon-oz can you share you pom.xml if possible.
thanks

rohit-nlp picture rohit-nlp on 29 Apr 2020

Hi @rohit-nlp,

Thanks for your suggestions, I added --packages in spark submit but still doesn't work because the dependency is still not resolved.

I compared the spark-nlp_2.11-2.4.4.pom, and spark-nlp_2.11-2.4.5.pom in the local repository folders and found that spark-nlp_2.11-2.4.5.pom seems not generated properly. Following is the content in spark-nlp_2.11-2.4.5.pom.

<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.johnsnowlabs.nlp</groupId>
  <artifactId>spark-nlp_2.11</artifactId>
  <version>2.4.5</version>
  <description>POM was created from install:install-file</description>
</project>

Following is what in the spark-nlp_2.11-2.4.4.pom

<?xml version='1.0' encoding='UTF-8'?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.11</artifactId>
    <packaging>jar</packaging>
    <description>spark-nlp</description>
    <url>https://nlp.johnsnowlabs.com</url>
    <version>2.4.4</version>
    <licenses>
        <license>
            <name>Apache-2.0</name>
            <url>http://opensource.org/licenses/Apache-2.0</url>
            <distribution>repo</distribution>
        </license>
    </licenses>
    <name>spark-nlp</name>
    <organization>
        <name>com.johnsnowlabs.nlp</name>
        <url>https://nlp.johnsnowlabs.com</url>
    </organization>
    <scm>
        <url>https://github.com/JohnSnowLabs/spark-nlp</url>
        <connection>scm:[email protected]:JohnSnowLabs/spark-nlp.git</connection>
    </scm>
    <developers>
        <developer>
            <id>saifjsl</id>
            <name>Saif Addin</name>
            <email>[email protected]</email>
            <url>https://github.com/saifjsl</url>
        </developer>
        <developer>
            <id>maziyarpanahi</id>
            <name>Maziyar Panahi</name>
            <email>[email protected]</email>
            <url>https://github.com/maziyarpanahi</url>
        </developer>
        <developer>
            <id>albertoandreottiATgmail</id>
            <name>Alberto Andreotti</name>
            <email>[email protected]</email>
            <url>https://github.com/albertoandreottiATgmail</url>
        </developer>
        <developer>
            <id>danilojsl</id>
            <name>Danilo Burbano</name>
            <email>[email protected]</email>
            <url>https://github.com/danilojsl</url>
        </developer>
        <developer>
            <id>rohit13k</id>
            <name>Rohit Kumar</name>
            <email>[email protected]</email>
            <url>https://github.com/rohit13k</url>
        </developer>
        <developer>
            <id>aleksei-ai</id>
            <name>Aleksei Alekseev</name>
            <email>[email protected]</email>
            <url>https://github.com/aleksei-ai</url>
        </developer>
        <developer>
            <id>showy</id>
            <name>Eduardo Mu帽oz</name>
            <email>[email protected]</email>
            <url>https://github.com/showy</url>
        </developer>
        <developer>
            <id>C-K-Loan</id>
            <name>Christian Kasim Loan</name>
            <email>[email protected]</email>
            <url>https://github.com/C-K-Loan</url>
        </developer>
    </developers>
    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.12</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>2.4.4</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.4.4</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.scalatest</groupId>
            <artifactId>scalatest_2.11</artifactId>
            <version>3.0.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.typesafe</groupId>
            <artifactId>config</artifactId>
            <version>1.3.0</version>
        </dependency>
        <dependency>
            <groupId>org.rocksdb</groupId>
            <artifactId>rocksdbjni</artifactId>
            <version>6.5.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-aws</artifactId>
            <version>3.2.0</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-annotations</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>commons-configuration</groupId>
                    <artifactId>commons-configuration</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.amazonaws</groupId>
                    <artifactId>aws-java-sdk-bundle</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-common</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-core</artifactId>
            <version>1.11.603</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-annotations</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>commons-configuration</groupId>
                    <artifactId>commons-configuration</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.603</version>
        </dependency>
        <dependency>
            <groupId>com.github.universal-automata</groupId>
            <artifactId>liblevenshtein</artifactId>
            <version>3.0.0</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.commons</groupId>
                    <artifactId>commons-lang3</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.navigamez</groupId>
            <artifactId>greex</artifactId>
            <version>1.0</version>
        </dependency>
        <dependency>
            <groupId>org.json4s</groupId>
            <artifactId>json4s-ext_2.11</artifactId>
            <version>3.5.3</version>
        </dependency>
        <dependency>
            <groupId>org.tensorflow</groupId>
            <artifactId>tensorflow</artifactId>
            <version>1.15.0</version>
        </dependency>
        <dependency>
            <groupId>net.sf.trove4j</groupId>
            <artifactId>trove4j</artifactId>
            <version>3.0.3</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

Just wondering why spark-nlp_2.11-2.4.5.pom couldn't be generated properly when using maven install-file to local repository; while 2.4.4.pom can be generated correctly?

mvn install:install-file=/path/to/sparknlp-2.4.5.jar \
  -DgroupId=com.johnsnowlabs.nlp \
 -DartifactId=spark-nlp_2.11 \
 -Dversion=2.4.5 \
 -Dpackaging=jar

Thanks in advance.

ps. manually add dependencies in 2.4.4.pom to 2.4.5.pom can solve the dependency issue :)

simon-oz picture simon-oz on 30 Apr 2020

@simon-oz how you got the local repo built? We checked with both Maven and SBT for 2.4.4 and 2.4.5 and the dependency tree is the same.

rohit-nlp picture rohit-nlp on 30 Apr 2020

@rohit-nlp we built local repo using maven.

I checked our repo, spark-nlp-2.2.0, and 2.4.0 both no problem. They are all installed by using the same mvn install-file command, except the versions are different.

simon-oz picture simon-oz on 30 Apr 2020

Ok.. So as far as I see now the issue is somewhere when the local repo you have is updated for 2.4.5 .. For the public maven repo where we publish the jars have the same dependency tree.

For now, I will close the issue. But do update if you manage to find the issue with the local repo.

Thanks, @simon-oz for the details.

rohit-nlp picture rohit-nlp on 30 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ledgerW picture ledgerW  路  7Comments
mlocutus picture mlocutus  路  4Comments
maziyarpanahi picture maziyarpanahi  路  3Comments
CyborgDroid picture CyborgDroid  路  5Comments
bbodnar picture bbodnar  路  3Comments