Spark-nlp: Error loading sparknlp into SparkSession on Windows
Description
Getting Exception: Java gateway process exited before sending its port number when trying to start a SparkSession with spark-nlp.
Error occurs with both sparknlp.start() and with the manual SparkSession attempt...
spark = SparkSession.builder\
.config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5")\
.getOrCreate()
Although, I am able to successfully load and use an older version of spark-nlp (2.0.1) (in the same environment)
Thanks for your help!
Expected Behavior
Initiate spark session with access to spark-nlp
Current Behavior
Exception Traceback (most recent call last)
<ipython-input-16-23832edab525> in <module>
1 spark = SparkSession.builder\
----> 2 .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5")\
3 .getOrCreate()
~\Anaconda3\envs\mtgml\lib\site-packages\pyspark\sql\session.py in getOrCreate(self)
171 for key, value in self._options.items():
172 sparkConf.set(key, value)
--> 173 sc = SparkContext.getOrCreate(sparkConf)
174 # This SparkContext may be an existing one.
175 for key, value in self._options.items():
~\Anaconda3\envs\mtgml\lib\site-packages\pyspark\context.py in getOrCreate(cls, conf)
365 with SparkContext._lock:
366 if SparkContext._active_spark_context is None:
--> 367 SparkContext(conf=conf or SparkConf())
368 return SparkContext._active_spark_context
369
~\Anaconda3\envs\mtgml\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
131 " note this option will be removed in Spark 3.0")
132
--> 133 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
134 try:
135 self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
~\Anaconda3\envs\mtgml\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
314 with SparkContext._lock:
315 if not SparkContext._gateway:
--> 316 SparkContext._gateway = gateway or launch_gateway(conf)
317 SparkContext._jvm = SparkContext._gateway.jvm
318
~\Anaconda3\envs\mtgml\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf)
44 :return: a JVM gateway
45 """
---> 46 return _launch_gateway(conf)
47
48
~\Anaconda3\envs\mtgml\lib\site-packages\pyspark\java_gateway.py in _launch_gateway(conf, insecure)
106
107 if not os.path.isfile(conn_info_file):
--> 108 raise Exception("Java gateway process exited before sending its port number")
109
110 with open(conn_info_file, "rb") as info:
Exception: Java gateway process exited before sending its port number
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: commons-codec#commons-codec;1.11!commons-codec.jar, download
annotations;2.6.0!jackson-annotations.jar(bundle)]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1302)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:304)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[I 20:39:47.104 LabApp] Saving file at /notebooks/Untitled.ipynb
[I 20:55:51.122 LabApp] Saving file at /notebooks/Untitled.ipynb
[I 20:57:52.139 LabApp] Saving file at /notebooks/Untitled.ipynb
Ivy Default Cache set to: C:\Users\LWEST\.ivy2\cache
The jars for the packages stored in: C:\Users\LWEST\.ivy2\jars
:: loading settings :: url = jar:file:/C:/Users/LWEST/Anaconda3/envs/mtgml/Lib/site-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/i
com.johnsnowlabs.nlp#spark-nlp_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-f83e9cc3-02d7-4c45-8a34-393938e25589;1.0
confs: [default]
found com.johnsnowlabs.nlp#spark-nlp_2.11;2.4.5 in central
found com.typesafe#config;1.3.0 in central
found org.rocksdb#rocksdbjni;6.5.3 in central
found org.apache.hadoop#hadoop-aws;3.2.0 in central
found com.amazonaws#aws-java-sdk-core;1.11.603 in central
found commons-logging#commons-logging;1.1.3 in central
found org.apache.httpcomponents#httpclient;4.5.9 in central
found org.apache.httpcomponents#httpcore;4.4.11 in central
found commons-codec#commons-codec;1.11 in local-m2-cache
found software.amazon.ion#ion-java;1.0.2 in local-m2-cache
found com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.6.7 in local-m2-cache
found joda-time#joda-time;2.8.1 in local-m2-cache
found com.amazonaws#aws-java-sdk-s3;1.11.603 in central
found com.amazonaws#aws-java-sdk-kms;1.11.603 in central
found com.amazonaws#jmespath-java;1.11.603 in central
found com.fasterxml.jackson.core#jackson-databind;2.6.7.2 in central
found com.fasterxml.jackson.core#jackson-annotations;2.6.0 in local-m2-cache
found com.fasterxml.jackson.core#jackson-core;2.6.7 in local-m2-cache
found com.github.universal-automata#liblevenshtein;3.0.0 in central
found com.google.code.findbugs#annotations;3.0.1 in central
found net.jcip#jcip-annotations;1.0 in central
found com.google.code.findbugs#jsr305;3.0.1 in central
found com.google.protobuf#protobuf-java-util;3.0.0-beta-3 in central
found com.google.protobuf#protobuf-java;3.0.0-beta-3 in central
found com.google.code.gson#gson;2.3 in central
found it.unimi.dsi#fastutil;7.0.12 in central
found org.projectlombok#lombok;1.16.8 in central
found org.slf4j#slf4j-api;1.7.21 in central
found com.navigamez#greex;1.0 in central
found dk.brics.automaton#automaton;1.11-8 in central
found org.json4s#json4s-ext_2.11;3.5.3 in central
found joda-time#joda-time;2.9.5 in central
found org.joda#joda-convert;1.8.1 in central
found org.tensorflow#tensorflow;1.15.0 in central
found org.tensorflow#libtensorflow;1.15.0 in central
found org.tensorflow#libtensorflow_jni;1.15.0 in central
found net.sf.trove4j#trove4j;3.0.3 in central
:: resolution report :: resolve 850ms :: artifacts dl 45ms
:: modules in use:
com.amazonaws#aws-java-sdk-core;1.11.603 from central in [default]
com.amazonaws#aws-java-sdk-kms;1.11.603 from central in [default]
com.amazonaws#aws-java-sdk-s3;1.11.603 from central in [default]
com.amazonaws#jmespath-java;1.11.603 from central in [default]
com.fasterxml.jackson.core#jackson-annotations;2.6.0 from local-m2-cache in [default]
com.fasterxml.jackson.core#jackson-core;2.6.7 from local-m2-cache in [default]
com.fasterxml.jackson.core#jackson-databind;2.6.7.2 from central in [default]
com.fasterxml.jackson.dataformat#jackson-dataformat-cbor;2.6.7 from local-m2-cache in [default]
com.github.universal-automata#liblevenshtein;3.0.0 from central in [default]
com.google.code.findbugs#annotations;3.0.1 from central in [default]
com.google.code.findbugs#jsr305;3.0.1 from central in [default]
com.google.code.gson#gson;2.3 from central in [default]
com.google.protobuf#protobuf-java;3.0.0-beta-3 from central in [default]
com.google.protobuf#protobuf-java-util;3.0.0-beta-3 from central in [default]
com.johnsnowlabs.nlp#spark-nlp_2.11;2.4.5 from central in [default]
com.navigamez#greex;1.0 from central in [default]
com.typesafe#config;1.3.0 from central in [default]
commons-codec#commons-codec;1.11 from local-m2-cache in [default]
commons-logging#commons-logging;1.1.3 from central in [default]
dk.brics.automaton#automaton;1.11-8 from central in [default]
it.unimi.dsi#fastutil;7.0.12 from central in [default]
joda-time#joda-time;2.9.5 from central in [default]
net.jcip#jcip-annotations;1.0 from central in [default]
net.sf.trove4j#trove4j;3.0.3 from central in [default]
org.apache.hadoop#hadoop-aws;3.2.0 from central in [default]
org.apache.httpcomponents#httpclient;4.5.9 from central in [default]
org.apache.httpcomponents#httpcore;4.4.11 from central in [default]
org.joda#joda-convert;1.8.1 from central in [default]
org.json4s#json4s-ext_2.11;3.5.3 from central in [default]
org.projectlombok#lombok;1.16.8 from central in [default]
org.rocksdb#rocksdbjni;6.5.3 from central in [default]
org.slf4j#slf4j-api;1.7.21 from central in [default]
org.tensorflow#libtensorflow;1.15.0 from central in [default]
org.tensorflow#libtensorflow_jni;1.15.0 from central in [default]
org.tensorflow#tensorflow;1.15.0 from central in [default]
software.amazon.ion#ion-java;1.0.2 from local-m2-cache in [default]
:: evicted modules:
commons-logging#commons-logging;1.2 by [commons-logging#commons-logging;1.1.3] in [default]
joda-time#joda-time;2.8.1 by [joda-time#joda-time;2.9.5] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 38 | 0 | 0 | 2 || 36 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
[NOT FOUND ] commons-codec#commons-codec;1.11!commons-codec.jar (2ms)
==== local-m2-cache: tried
file:/C:/Users/LWEST/.m2/repository/commons-codec/commons-codec/1.11/commons-codec-1.11.jar
[NOT FOUND ] com.fasterxml.jackson.core#jackson-annotations;2.6.0!jackson-annotations.jar(bundle) (0ms)
==== local-m2-cache: tried
file:/C:/Users/LWEST/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: FAILED DOWNLOADS ::
:: ^ see resolution messages for details ^ ::
::::::::::::::::::::::::::::::::::::::::::::::
:: commons-codec#commons-codec;1.11!commons-codec.jar
:: com.fasterxml.jackson.core#jackson-annotations;2.6.0!jackson-annotations.jar(bundle)
::::::::::::::::::::::::::::::::::::::::::::::
Possible Solution
Maybe related to Windows8, updates between 2.0.1 and current, or Ivy directory (default location vs. inside conda env? - I'm not that familiar with the Java side of things)?
Steps to Reproduce
conda env create -f environment.yml
environment.yml
name: env-name
channels:
- conda-forge
- defaults
dependencies:
- python=3.6
- other packages...
- pip
- pip:
- requests_aws4auth
- spark-nlp==2.4.5
- pyspark==2.4.4
2.
import pyspark
from pyspark.sql import SparkSession
3. These do not work...
spark = SparkSession.builder\
.config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5")\
.getOrCreate()
OR
import sparknlp
spark = sparknlp.start()
4. This does work
spark = SparkSession.builder\
.config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.0.1")\
.getOrCreate()
Context
Want to use the latest version... with USE, for example.
Your Environment
- Spark NLP version: 2.4.5
- Apache NLP version:
- Java version (java -version): 1.8
- Setup and installation (Pypi, Conda, Maven, etc.):
Conda Env
Python 3.6
pip:
spark-nlp==2.4.5
pyspark==2.4.4 - Operating System and version: Windows 8
ledgerW
All 7 comments
Hi, in the newer version there are newer dependencies and the error indicates for some reason it cannot download one of them from Maven.
Are you by any chance behind proxy or firewall?
maziyarpanahi
on 5 May 2020
Thanks for the quick response.
Nope, no proxy, firewall, vpn. Just working from home. Is there
additional information I can provide that might help? Also, just to
confirm, the pyspark session initiates as expected when I don't call
spark-nlp.
On Tue, May 5, 2020 at 2:53 AM Maziyar Panahi notifications@github.com
wrote:
Hi, in the newer version there are newer dependencies and the error
indicates for some reason it cannot download one of them from Maven.
Are you by any chance behind proxy or firewall?—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/JohnSnowLabs/spark-nlp/issues/892#issuecomment-623885476,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AARYO65LSKJT7NWF2HVIADDRP6ZXPANCNFSM4MZGL3RA
.
ledgerW
on 5 May 2020
Just to be sure about the access to the source, could you please replace the following:
.config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5")\
with
.config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.4.5")\
The first one is Maven the second one is Spark Packages. I am hoping you can access the second one.
maziyarpanahi
on 5 May 2020
I'm getting the same exact error messages with Spark Packages as with
Maven. It fails to download the same two jars.
But, like before, everything works fine with the older version
.config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.0.1")
On Tue, May 5, 2020 at 12:50 PM Maziyar Panahi notifications@github.com
wrote:
Just to be sure about the access to the source, could you please replace
the following:.config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5")\
with
.config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.4.5")\
The first one is Maven the second one is Spark Packages. I am hoping you
can access the second one.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/JohnSnowLabs/spark-nlp/issues/892#issuecomment-624171753,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AARYO62NPGDFVDYY3WIBI7TRQA7TXANCNFSM4MZGL3RA
.
ledgerW
on 5 May 2020
spark-nlp:2.0.1 is very old. It uses a different Apache Spark (2.3.x). In the newer releases, we are compiling against Apache Spark 2.4.x which I am guessing these two jars are from there (they don't exist in Spark NLP).
Might I suggest removing C:/Users/LWEST/.m2/ and C:/Users/LWEST/.ivy2/ just to rule out conflicts from left-overs?
maziyarpanahi
on 5 May 2020
Yep, that fixed it. All is well again. Greatly appreciated!
ledgerW
on 5 May 2020
Fantastic! Thanks for the update.
maziyarpanahi
on 5 May 2020
Related issues
SoundBot
·
5Comments
subhashiniy
·
3Comments
SankethBK
·
3Comments
fzr72725
·
3Comments
santoshbs
·
5Comments
Most helpful comment
Yep, that fixed it. All is well again. Greatly appreciated!