Hub: Tensor_flow text Similarty _issue

Created on 17 Sep 2018  路  22Comments  路  Source: tensorflow/hub

Hello Every One ,
Can you please help out to resolve below issue.
2018-09-17 15:10:51.106133: I T:srcgithubtensorflowtensorflowcoreplatformcpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-09-17 15:10:52.536772: I T:srcgithubtensorflowtensorflowcorecommon_runtimegpugpu_device.cc:1405] Found device 0 with properties:
name: GeForce 940MX major: 5 minor: 0 memoryClockRate(GHz): 1.189
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.36GiB
2018-09-17 15:10:52.541692: I T:srcgithubtensorflowtensorflowcorecommon_runtimegpugpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-17 15:10:59.482063: I T:srcgithubtensorflowtensorflowcorecommon_runtimegpugpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-17 15:10:59.506852: I T:srcgithubtensorflowtensorflowcorecommon_runtimegpugpu_device.cc:971] 0
2018-09-17 15:10:59.521022: I T:srcgithubtensorflowtensorflowcorecommon_runtimegpugpu_device.cc:984] 0: N
2018-09-17 15:10:59.527459: I T:srcgithubtensorflowtensorflowcorecommon_runtimegpugpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3081 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0, compute capability: 5.0)
[None, None]
2018-09-17 15:11:26.985779: E T:srcgithubtensorflowtensorflowcorecommon_runtimeexecutor.cc:697] Executor failed to create kernel. Not found: No registered 'Size' OpKernel for GPU devices compatible with node module_apply_default/compound_bigrams/boolean_mask/Prod = SizeT=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"
(OpKernel was found, but attributes didn't match)
. Registered: device='CPU'; out_type in [DT_INT32]
device='CPU'; out_type in [DT_INT64]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT32]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT64]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT32]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT64]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT32]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT64]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT32]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT64]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT32]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT64]

     [[Node: module_apply_default/compound_bigrams/boolean_mask/Prod = Size[T=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](module_apply_default/compound_bigrams/concat_2/_91, ^module_apply_default/compound_bigrams/boolean_mask/Shape_1/_95)]]

Traceback (most recent call last):
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonclientsession.py", line 1278, in _do_call
return fn(*args)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonclientsession.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonclientsession.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: No registered 'Size' OpKernel for GPU devices compatible with node module_apply_default/compound_bigrams/boolean_mask/Prod = SizeT=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"
(OpKernel was found, but attributes didn't match)
. Registered: device='CPU'; out_type in [DT_INT32]
device='CPU'; out_type in [DT_INT64]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT32]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT64]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT32]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT64]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT32]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT64]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT32]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT64]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT32]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT64]

     [[Node: module_apply_default/compound_bigrams/boolean_mask/Prod = Size[T=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](module_apply_default/compound_bigrams/concat_2/_91, ^module_apply_default/compound_bigrams/boolean_mask/Shape_1/_95)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 3, in
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonclientsession.py", line 877, in run
run_metadata_ptr)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonclientsession.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonclientsession.py", line 1272, in _do_run
run_metadata)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonclientsession.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: No registered 'Size' OpKernel for GPU devices compatible with node module_apply_default/compound_bigrams/boolean_mask/Prod = SizeT=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"
(OpKernel was found, but attributes didn't match)
. Registered: device='CPU'; out_type in [DT_INT32]
device='CPU'; out_type in [DT_INT64]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT32]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT64]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT32]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT64]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT32]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT64]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT32]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT64]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT32]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT64]

     [[Node: module_apply_default/compound_bigrams/boolean_mask/Prod = Size[T=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](module_apply_default/compound_bigrams/concat_2/_91, ^module_apply_default/compound_bigrams/boolean_mask/Shape_1/_95)]]

Caused by op 'module_apply_default/compound_bigrams/boolean_mask/Prod', defined at:
File "", line 3, in
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflow_hubmodule.py", line 203, in __call__
name=name)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflow_hubnative_module.py", line 447, in create_apply_graph
restore_collections_predicate=(lambda key: key in import_collections))
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythontrainingsaver.py", line 1939, in import_meta_graph
*kwargs)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonframeworkmeta_graph.py", line 744, in import_scoped_meta_graph
producer_op_list=producer_op_list)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonutildeprecation.py", line 454, in new_func
return func(args, **kwargs)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonframeworkimporter.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonframeworkimporter.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonframeworkops.py", line 3289, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonframeworkops.py", line 3289, in
for c_op in c_api_util.new_tf_operations(self)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonframeworkops.py", line 3180, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "C:UsersRahulAnaconda3envstensorflowlibsite-packagestensorflowpythonframeworkops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): No registered 'Size' OpKernel for GPU devices compatible with node module_apply_default/compound_bigrams/boolean_mask/Prod = SizeT=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"
(OpKernel was found, but attributes didn't match)
. Registered: device='CPU'; out_type in [DT_INT32]
device='CPU'; out_type in [DT_INT64]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT32]
device='GPU'; T in [DT_HALF]; out_type in [DT_INT64]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_BFLOAT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT32]
device='GPU'; T in [DT_FLOAT]; out_type in [DT_INT64]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT32]
device='GPU'; T in [DT_DOUBLE]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT64]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT16]; out_type in [DT_INT64]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_UINT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT8]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX64]; out_type in [DT_INT64]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT32]
device='GPU'; T in [DT_COMPLEX128]; out_type in [DT_INT64]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT32]
device='GPU'; T in [DT_BOOL]; out_type in [DT_INT64]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT32]
device='GPU'; T in [DT_VARIANT]; out_type in [DT_INT64]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT32]
device='GPU'; T in [DT_INT32]; out_type in [DT_INT64]

     [[Node: module_apply_default/compound_bigrams/boolean_mask/Prod = Size[T=DT_STRING, out_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](module_apply_default/compound_bigrams/concat_2/_91, ^module_apply_default/compound_bigrams/boolean_mask/Shape_1/_95)]]
hub awaiting response text-embedding bug
Source
picture rahulgarg1
👍7

Most helpful comment

Thanks @rmanak and @lc0 for explaining the problem.

I've contacted the publisher and got an explanation:
"Yes, this is a known issue, the USE2 worked on GPU on tf 1.7 (and maybe 1.8). There are some ops removed or unregistered from GPU since tf 1.9, which caused the problem you pasted. A simple fix to make USE2 work on GPU is switching to a early version of tensorflow, e.g. 1.7."

So the way around this for now is to either use https://tfhub.dev/google/universal-sentence-encoder-large/3 or run this on tensorflow-gpu==1.7.

picture vbardiovskyg on 28 Sep 2018
👍8

All 22 comments

Hi Every One , 

     Please help me resolve of this error ..
rahulgarg1 picture rahulgarg1 on 17 Sep 2018

Hi @rahulgarg1,

can you please paste a minimal code snippet that reproduces this?

Also which version of tensorflow-gpu do you have installed?

vbardiovskyg picture vbardiovskyg on 18 Sep 2018

have the same issue with google colab @vbardiovskyg -> https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb

lc0 picture lc0 on 25 Sep 2018

and seems like it works with https://tfhub.dev/google/universal-sentence-encoder-large/3

lc0 picture lc0 on 25 Sep 2018
👍1

Tried to reproduce by clicking on the link, then "Runtime -> Run All", seems to work with "https://tfhub.dev/google/universal-sentence-encoder/2".

Did you make any changes to the colab before it failed?

vbardiovskyg picture vbardiovskyg on 25 Sep 2018

it works on CPU runtime, but GPU runtime has the same issues as from the original author cc @vbardiovskyg

lc0 picture lc0 on 25 Sep 2018
👍6

same issue here

rmanak picture rmanak on 25 Sep 2018

minimal code that generates the issue:

import tensorflow as tf
import tensorflow_hub as hub

with tf.Graph().as_default():
    module_url = "https://tfhub.dev/google/universal-sentence-encoder/2"
    embed = hub.Module(module_url)
    embeddings = embed(["A long sentence.", "single-word",
                      "http://example.com"])

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        sess.run(tf.tables_initializer())
        print(sess.run(embeddings))

aka the code example provided in the website!

rmanak picture rmanak on 25 Sep 2018
👍4

Thanks @rmanak and @lc0 for explaining the problem.

I've contacted the publisher and got an explanation:
"Yes, this is a known issue, the USE2 worked on GPU on tf 1.7 (and maybe 1.8). There are some ops removed or unregistered from GPU since tf 1.9, which caused the problem you pasted. A simple fix to make USE2 work on GPU is switching to a early version of tensorflow, e.g. 1.7."

So the way around this for now is to either use https://tfhub.dev/google/universal-sentence-encoder-large/3 or run this on tensorflow-gpu==1.7.

vbardiovskyg picture vbardiovskyg on 28 Sep 2018
👍8

@vbardiovskyg should we update the docs in this case?

I am just not sure how make it nice, since when colab has CPU env the version works fine.
Otherwise, I would at least limit to pip install at https://github.com/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb

What do you think?

lc0 picture lc0 on 28 Sep 2018

I think this should be documented on the module description.

Not sure if we should downgrade TF version on the Colab just because some modules do not work (there will always be modules that use older versions of TF).

We could add a comment next to the module selection that some modules might not be compatible with GPU, or pick two that are both compatible.

vbardiovskyg picture vbardiovskyg on 28 Sep 2018

None of the following helped me:

  1. Changing to https://tfhub.dev/google/universal-sentence-encoder-large/3
  2. running on tensorflow-gpu==1.7.
rosequ picture rosequ on 10 Dec 2018

Hi I tried on tensorflow-gpu==1.7 but somehow get this error
AttributeError: module 'tensorflow.python.estimator.estimator_lib' has no attribute 'Exporter'

dontloo picture dontloo on 11 Dec 2018
👍1

None of the following helped me:

  1. Changing to https://tfhub.dev/google/universal-sentence-encoder-large/3
  2. running on tensorflow-gpu==1.7.

I solved this problem by changing the module to https://tfhub.dev/google/universal-sentence-encoder-large/3.
Or you can try to set the device to tf.device('/cpu:0').

charmpeng picture charmpeng on 23 Dec 2018

Yep,

  1. running on tensorflow-gpu==1.7 doesn't help the DAN (https://tfhub.dev/google/universal-sentence-encoder/2) to work.
  2. DAN (https://tfhub.dev/google/universal-sentence-encoder/2) runs only in the CPU.
  3. Transformer architecture (https://tfhub.dev/google/universal-sentence-encoder-large/3) runs perfectly on the GPU without any error.
Aravinviju picture Aravinviju on 8 Jan 2019

@vbardiovskyg Has this been updated in the module documentation ? If not, please let me know the module owner name and I will follow up on this. Thanks !

Harshini-Gadige picture Harshini-Gadige on 12 Mar 2019

I tried using the https://tfhub.dev/google/universal-sentence-encoder-large/3 version on a fresh install of

  • tensorflow==1.7
  • tensorflow_hub==0.2.0
    with tf.device('/cpu:0')

Unfortunately the whole process is still running on the CPU

koolkao picture koolkao on 29 Apr 2019

Hi @koolkao, it seems like you are not installing tensorflow-gpu and placing it explicitly on CPU, so I am not sure why you expect to not run on CPU. (maybe a typo somewhere, but not clear where)

vbardiovskyg picture vbardiovskyg on 3 May 2019

Aside from koolkao's immediate problem, there has been some progress on the root issue:
Commit https://github.com/tensorflow/tensorflow/commit/5c4d1c87a2794ed80d8c1da8bb6fe8b6eb86242a fixes an issue with Grappler (graph optimizer) that may have caused the failures ever since TF 1.8.

Community input is welcome whether the problem can be avoided in the field by any of the following:

  • Use a tf-nightly-gpu past that commit (1.14.1.dev20190503 should do, or a day after).
  • Turn off the incriminated optimization by setting
    config.graph_options.rewrite_options.shape_optimization = 2
    in the tf.ConfigProto passed to the tf.Session.

If yes, the module documentation could be updated for that (and maybe again once the fix hits a release; might be as late as 1.15/2.0).

arnoegw picture arnoegw on 3 May 2019
👍2

tensorflow: 1.13.1
tensorflow-gpu: 1.13.1

I have got the same error. However, setting

config = tf.ConfigProto()
config.graph_options.rewrite_options.shape_optimization = 2
session = tf.Session(config=config)

fixed it for me. Thanks!

Dobatymo picture Dobatymo on 14 May 2019
👍1

Thanks @arnoegw for figuring out the state of and the solution for the problem. I will update the documentation to reflect this.

vbardiovskyg picture vbardiovskyg on 14 May 2019

try doing failed ml operation within tf.device block
with tf.device('/cpu:0'):
Running on the CPU saved my day

Aashish-1008 picture Aashish-1008 on 9 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

arnoegw picture arnoegw  路  5Comments
PradyumnaGupta picture PradyumnaGupta  路  3Comments
truas picture truas  路  4Comments
bhack picture bhack  路  3Comments
bzburr picture bzburr  路  4Comments