Spark-nlp: Tensorflow Error - Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.

[TiGaI]

Description

`---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
~\Anaconda3\envs\sparknlp\lib\pyspark\sql\utils.py in deco(a, *kw)
62 try:
---> 63 return f(a, *kw)
64 except py4j.protocol.Py4JJavaError as e:

~\Anaconda3\envs\sparknlp\lib\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
327 "An error occurred while calling {0}{1}{2}.\n".
--> 328 format(target_id, ".", name), value)
329 else:

Py4JJavaError: An error occurred while calling o49.load.
: java.lang.IllegalArgumentException: NodeDef mentions attr 'incompatible_shape_error' not in Op z:bool; attr=T:type,allowed=[DT_BFLOAT16, DT_HALF, DT_FLOAT, DT_DOUBLE, DT_UINT8, ..., DT_QINT8, DT_QINT32, DT_STRING, DT_BOOL, DT_COMPLEX128]; is_commutative=true>; NodeDef: {{node accuracy/Equal}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
at org.tensorflow.Graph.importGraphDef(Native Method)
at org.tensorflow.Graph.importGraphDef(Graph.java:131)
at org.tensorflow.Graph.importGraphDef(Graph.java:115)
at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.readGraph(TensorflowWrapper.scala:209)
at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.readChkPoints(TensorflowWrapper.scala:384)
at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel$class.readTensorflowChkPoints(TensorflowSerializeModel.scala:128)
at com.johnsnowlabs.nlp.annotators.classifier.dl.SentimentDLModel$.readTensorflowChkPoints(SentimentDLModel.scala:157)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadSentimentDLTensorflowModel$class.readTensorflow(SentimentDLModel.scala:147)
at com.johnsnowlabs.nlp.annotators.classifier.dl.SentimentDLModel$.readTensorflow(SentimentDLModel.scala:157)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadSentimentDLTensorflowModel$$anonfun$4.apply(SentimentDLModel.scala:154)
at com.johnsnowlabs.nlp.annotators.classifier.dl.ReadSentimentDLTensorflowModel$$anonfun$4.apply(SentimentDLModel.scala:154)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead$1.apply(ParamsAndFeaturesReadable.scala:31)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead$1.apply(ParamsAndFeaturesReadable.scala:30)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$class.com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead(ParamsAndFeaturesReadable.scala:30)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read$1.apply(ParamsAndFeaturesReadable.scala:41)
at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read$1.apply(ParamsAndFeaturesReadable.scala:41)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:19)
at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:8)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

During handling of the above exception, another exception occurred:

IllegalArgumentException Traceback (most recent call last)
in
----> 1 ner = SentimentDLModel.load("sentimentdl_use_twitter_en_2.5.0_2.4_1589108892106")

~\Anaconda3\envs\sparknlp\lib\pyspark\ml\util.py in load(cls, path)
360 def load(cls, path):
361 """Reads an ML instance from the input path, a shortcut of read().load(path)."""
--> 362 return cls.read().load(path)
363
364

~\Anaconda3\envs\sparknlp\lib\pyspark\ml\util.py in load(self, path)
298 if not isinstance(path, basestring):
299 raise TypeError("path should be a basestring, got type %s" % type(path))
--> 300 java_obj = self._jread.load(path)
301 if not hasattr(self._clazz, "_from_java"):
302 raise NotImplementedError("This Java ML type cannot be loaded into Python currently: %r"

~\Anaconda3\envs\sparknlp\lib\py4j\java_gateway.py in __call__(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

~\Anaconda3\envs\sparknlp\lib\pyspark\sql\utils.py in deco(a, *kw)
77 raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
78 if s.startswith('java.lang.IllegalArgumentException: '):
---> 79 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
80 raise
81 return deco

IllegalArgumentException: "NodeDef mentions attr 'incompatible_shape_error' not in Op z:bool; attr=T:type,allowed=[DT_BFLOAT16, DT_HALF, DT_FLOAT, DT_DOUBLE, DT_UINT8, ..., DT_QINT8, DT_QINT32, DT_STRING, DT_BOOL, DT_COMPLEX128]; is_commutative=true>; NodeDef: {{node accuracy/Equal}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.)."`

Expected Behavior

I try it the online version and the offline version. Both of them give me the same errors. I can download pipeline and use those no problem.

Current Behavior

Possible Solution

Steps to Reproduce

1. New environment in window 10, create new conda enviroment
2. Open a spark sections
3. Load the model
4. Error occurs

Context

Just trying to load the perbuilt model, but the error keep happening.

Your Environment

• Spark NLP version: 2.5.5
• Apache Spark version: 2.4.6
• Java version (java -version): 1.8.0_251
• Setup and installation (Pypi, Conda, Maven, etc.): conda install
• Operating System and version: window 10
• Link to your project (if any):

[maziyarpanahi]

Thanks, we'll test this on Windows 10 to reproduce it.

[TiGaI]

Keep me updated. It works on Mac OS. So it may be a window issue.

[TiGaI]

Any progress? It is definitely a window issue. Linux and Mac OS work using the exact same step.

[maziyarpanahi]

I have tested this on Windows 8 Pro, could you please make sure you have followed the steps from here: https://github.com/JohnSnowLabs/spark-nlp/issues/982#issuecomment-670033754

Just to be sure you have the requirements in a Windows environment. But I was planning to install Windows 10 and reproduce this there as well later today/tomorrow.

[TiGaI]

I have no problem with PipelineModel.load with explain_document_ml_en_2.4.0_2.4_1580252705962.

The problem arrived when I try to use SentimentDLModel.

Py4JJavaError: An error occurred while calling o444.load.
: java.lang.IllegalArgumentException: NodeDef mentions attr 'incompatible_shape_error' not in Op z:bool; attr=T:type,allowed=[DT_BFLOAT16, DT_HALF, DT_FLOAT, DT_DOUBLE, DT_UINT8, ..., DT_QINT8, DT_QINT32, DT_STRING, DT_BOOL, DT_COMPLEX128]; is_commutative=true>; NodeDef: {{node accuracy/Equal}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).

[TiGaI]

My spark works on most of the functionality. Of course, I can't use the directly download, but I can download the pipeline and load it in, no problem.

Hey Maziyarpanahi,

I am not sure about window 8 PRO, but I follow your step one by one in window 10.

But when I load this it gives me the same exact error. Now it is saying a little more.
Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.pretrained.PythonResourceDownloader.downloadModel.
: java.lang.UnsupportedOperationException: Spark NLP tried to load a TensorFlow Graph using Contrib module, but failed to load it on this system. If you are on Windows, this operation is not supported. Please try a noncontrib model. If not the case, please report this issue. Original error message:

[maziyarpanahi]

Thanks, a couple of things:

• explain_document_ml has no TensorFlow so it's normal to work
• explain_document_dl has TF so that should fail
• The fact that you cannot download the model means you haven't set up the Apache Spark/Hadoop env correctly. Otherwise, it will download with no problem and one of the things that can cause this is Microsoft Visual C++ and maybe it is a different version for Windows 10
• Windows don't like having spark under sub-directory. So if you can put your Spark at the root like Hadoop would be better

I am installing Windows 10 and setting the envs right now, so I'll keep you updated today for all this

[maziyarpanahi]

OK, this is definitely a bug in TensorFlow supporting Windows 10, not just one model, all the TF related models.

To solve the pretrained, you need the followings:

• Create a C:\temp
• Create a C:\temp\hive
• C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/hive
• C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/
• have this installed: https://www.microsoft.com/en-us/download/confirmation.aspx?id=14632

This won't solve the TF issue but allows you to use pretrained() without any issue if other params are set correctly.

[maziyarpanahi]

The steps:

• Download OpenJDK from here: https://adoptopenjdk.net/?variant=openjdk8&jvmVariant=hotspot
  
  
  
  • Make sure it is 64-bit
  
  
  • Make sure you install it in the root C:\java Windows doesn't like space in the path.
  
  
  • During installation after changing the path, select setting Path
  
  
• Download winutils and put it in C:\hadoop\bin https://github.com/cdarlint/winutils/blob/master/hadoop-2.7.3/bin/winutils.exe
• Download Anaconda 3.6 from Archive, I didn't like the new 3.8 : https://repo.anaconda.com/archive/Anaconda3-2020.02-Windows-x86_64.exe
• Download Apache Spark 2.4.6 and extract it in C:\spark\
• Set the env for HADOOP_HOME to C:\hadoop and SPARK_HOME to C:\spark
• Set Paths for %HADOOP_HOME%\bin and %SPARK_HOME%\bin
• Install C++ (again the 64 bit) https://www.microsoft.com/en-us/download/confirmation.aspx?id=14632
• Create C:\temp and C:\temp\hive
• Fix permissions:
  
  
  
  • C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/hive
  
  
  • C:\Users\maz>%HADOOP_HOME%\bin\winutils.exe chmod 777 /tmp/
  
  

Either create a conda env for python 3.6, install pyspark==2.4.6 spark-nlp numpy and use Jupyter/python console, or in the same conda env you can go to spark bin for pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.11:2.5.5.

[TiGaI]

Hey @maziyarpanahi

I test it those step you provided ones by one pinpoint the error make sure. Nothing did it until I reinstall Apache Spark 2.4.6. It may be because I download it from a mirror site that was outdated, I don't really know. I am getting a mirror warning though, not sure if you encounter this before.

I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2

[maziyarpanahi]

This is a great news. Happy it worked out st the end.
It's a minor warning from TF regarding your CPU having some features not included in the main TF build which is totally fine.

[TiGaI]

Thanks!