Hub: Universal Sentence Encoder not using GPU

[alvations]

I've been trying to figure out why is it that the https://tfhub.dev/google/universal-sentence-encoder/1 module is choking the GPU ram and not using it.

On one of a local machine with CUDA 9.0 Ubuntu 16.04, it's just not using any GPU compute when I did:

import tensorflow as tf
import tensorflow_hub as hub

model_name_dan = 'https://tfhub.dev/google/universal-sentence-encoder/1'
embed = hub.Module(model_name_dan)

with tf.Session() as session:
    session.run([tf.global_variables_initializer(), tf.tables_initializer()])

On nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    On   | 00000000:02:00.0 Off |                  N/A |
| 27%   37C    P2    37W / 180W |   7754MiB /  8114MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    On   | 00000000:03:00.0 Off |                  N/A |
| 27%   34C    P2    37W / 180W |   7722MiB /  8114MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 1080    On   | 00000000:81:00.0 Off |                  N/A |
| 27%   34C    P2    40W / 180W |   7722MiB /  8114MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 1080    On   | 00000000:82:00.0 Off |                  N/A |
| 27%   33C    P2    36W / 180W |   7722MiB /  8114MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

When tracking the htop, it looks like it's using the CPU instead of the GPU but somehow the RAM on the GPU is choked to the max when starting the session.

Is this the expected behavior? Anyone else have the same experience?

[akhorlin]

Would it possible to print out device placement (logging device placement
https://www.tensorflow.org/programmers_guide/using_gpu)? A tentative
theory is that the module is CPU bound because text pre-processing
(tokenization and ngramming) has to happen on CPU. The memory is taken up
by the embedding lookup. But to make sure that this conjunction is correct
it would be great to see the placement info.

On Fri, Jun 8, 2018 at 3:05 AM alvations [e-mail] wrote:

I've been trying to figure out why is it that the
https://tfhub.dev/google/universal-sentence-encoder/1 module is choking
the GPU ram and not using it.

On one of a local machine with CUDA 9.0 Ubuntu 16.04, it's just not using
any GPU compute when I did:

import tensorflow as tfimport tensorflow_hub as hub

embed = hub.Module(model_name_dan)
with tf.Session() as session:
session.run([tf.global_variables_initializer(), tf.tables_initializer()])

On nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26 Driver Version: 387.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 On | 00000000:02:00.0 Off | N/A |
| 27% 37C P2 37W / 180W | 7754MiB / 8114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 1080 On | 00000000:03:00.0 Off | N/A |
| 27% 34C P2 37W / 180W | 7722MiB / 8114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 1080 On | 00000000:81:00.0 Off | N/A |
| 27% 34C P2 40W / 180W | 7722MiB / 8114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 1080 On | 00000000:82:00.0 Off | N/A |
| 27% 33C P2 36W / 180W | 7722MiB / 8114MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

When tracking the htop, it looks like it's using the CPU instead of the
GPU but somehow the RAM on the GPU is choked to the max when starting the
session.

Is this the expected behavior? Anyone else have the same experience?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/hub/issues/70, or mute the thread
https://github.com/notifications/unsubscribe-auth/AbunTHnAhKD9EC34XKy2qSPDrkshpubQks5t6c3UgaJpZM4UfVzt
.

[pzelasko]

We're also having issues with this (but with model version 2): when model placement is forced with with tf.device('/gpu:0'):, we get error that OpKernel Size for type DT_STRING is not defined for GPU.

Do you support or plan for supporting these models on GPU? Perhaps it would make sense to force placement of string ops in this model on the CPU, and let the rest be allocated on GPU if it is present.

[ELind77]

I'm having the same problem, I can't get USE (Transformer) to run on a GPU. I'd really like to get some attention paid to this if possible. Also, I can't find source anywhere to submit a PR even if I were able to find where I needed to make a fix, so source would be nice as well.

[neil-119]

This was working (on my GTX 1080) till I tried to install tensorflow-serving-api via pip. Then, it must've modified something in my environment because I started getting an error similar to this one. I knew, just _knew_ that I should've backed up my environment before installing that package. Serves me right.

[andresusanopinto]

Was this working with an older tensorflow version? My guess is this is a regression in tf-1.9 and was working with 1.8?

[micahprice]

I'm getting the same error messages as #94 (just like @neil-119), but only when I try to load a SavedModel that includes USE (v2). Loading the model from the checkpoint (from regular model.ckpt files) is working fine for me though (on GPU). Don't understand why it works one way and not the other. Should we reopen #94?

[tinder-arozinov]

@andresusanopinto The errors in #94 disappear upon reverting to version 1.8. It's still to-be-determined if the GPU is fully being utilized though.

[ELind77]

To clarify, the issue I am having is not the errors as in https://github.com/tensorflow/hub/issues/94. I am seeing what @alvations indicated in the opening post, RAM is allocated for a job on my GPU but no compute usage, all compute appears to be on CPU. I'm running TF 1.8 and USE large/3.

[akhorlin]

One thing to try to see how GPU utilization depends on the size of the text
fed to USE embedding. It could be that computation is bottle-necked on text
parsing/pre-processing that happens inside USE and which happens on the
CPU. It might be interesting to run the model under tf.profiler and post
the resulting timelines.

On Wed, Aug 15, 2018 at 9:59 PM Eric Lind [e-mail] wrote:

To clarify, the issue I am having is not the errors as in #94
https://github.com/tensorflow/hub/issues/94. I am seeing what @alvations
https://github.com/alvations indicated in the opening post, RAM is
allocated for a job on my GPU but no compute usage, all compute appears to
be on CPU. I'm running TF 1.8 and USE large/3.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/tensorflow/hub/issues/70#issuecomment-413316969, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AbunTG2TUk-Jd0ejE3n4pK6MjVRF_gZQks5uRH2SgaJpZM4UfVzt
.

[DzianisH]

I have the same peoblem as pzelasko have with v2. I can use this encoder only on CPU.

[pzelasko]

As a workaround, I was able to run the "lite" model version on the GPU https://alpha.tfhub.dev/google/universal-sentence-encoder-lite/2

However, it would still be useful to be able to run the more powerful models on the GPU as well.

[daniellevy]

Any update on this by any chance?

[korymath]

@daniellevy I have a simple working example on JANN

with tf.device('/gpu:0'):
          chunk_line_embeddings = session.run(module(chunk_unencoded_lines))

[santiontanon]

I have the same problem, the only way I can make it run is to prefix the invocation of the model with:
with tf.device('CPU'):

[korymath]

Allow soft placement can also help handle code which might be run under both GPU and CPU. From: Tensorflow: Using GPUs

sess = tf.Session(config=tf.ConfigProto(
      allow_soft_placement=True, log_device_placement=True))

[OmarMAmin]

Maybe if it's inside an estimator it'll run on both GPU and CPU?

[rmothukuru]

@alvations ,
Can you please confirm if this issue is resolved, so that we can close this issue.

[alvations]

Looks like it's resolved in the latest version of the tf-hub. Anyone else have the old experience of GPU not movings when embedding/encoding sentences?

[rmothukuru]

Closing this issue as it has been resolved.

[MajesticKhan]

@alvations I attempted to run a DNN estimator on a GPU, but it crashes out and produces: CUDA_ERROR_OUT_OF_MEMORY. I tried allow_soft_placement=True, log_device_placement=True along with gpu_options.per_process_gpu_memory_fraction = 0.70.

[sarahwie]

Still not able to use GPU for USE model 'https://tfhub.dev/google/universal-sentence-encoder/4'

Specs:
tensorflow=2.1.0
tensorflow-hub=0.7.0
Ubuntu 16.04
GPUs available
Using import tensorflow.compat.v1 as tf for USE model compatability

Code:

import tensorflow.compat.v1 as tf
tf.disable_eager_execution()
import tensorflow_hub as hub

module_url = "https://tfhub.dev/google/universal-sentence-encoder/4" 
embed = hub.load(module_url)

with tf.Session() as session:
    session.run([tf.global_variables_initializer(), tf.tables_initializer()])
    with tf.device('/gpu:1'):
        embeds = session.run(embed(list_of_events))

Stack trace:

Op: Const
Node attrs: value=Tensor<type: string shape: [4] values: sad untrustworthy unlucky...>, dtype=DT_STRING
Registered kernels:
  device='XLA_CPU_JIT'; dtype in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64, DT_STRING]
  device='XLA_GPU_JIT'; dtype in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64, DT_STRING]
  device='GPU'; dtype in [DT_VARIANT]
  device='GPU'; dtype in [DT_BOOL]
  device='GPU'; dtype in [DT_COMPLEX128]
  device='GPU'; dtype in [DT_COMPLEX64]
  device='GPU'; dtype in [DT_UINT64]
  device='GPU'; dtype in [DT_INT64]
  device='GPU'; dtype in [DT_QINT32]
  device='GPU'; dtype in [DT_UINT32]
  device='GPU'; dtype in [DT_QUINT16]
  device='GPU'; dtype in [DT_QINT16]
  device='GPU'; dtype in [DT_INT16]
  device='GPU'; dtype in [DT_UINT16]
  device='GPU'; dtype in [DT_QINT8]
  device='GPU'; dtype in [DT_INT8]
  device='GPU'; dtype in [DT_UINT8]
  device='GPU'; dtype in [DT_DOUBLE]
  device='GPU'; dtype in [DT_FLOAT]
  device='GPU'; dtype in [DT_BFLOAT16]
  device='GPU'; dtype in [DT_HALF]
  device='GPU'; dtype in [DT_INT32]
  device='XLA_GPU'; dtype in [DT_UINT8, DT_QUINT8, DT_UINT16, DT_INT8, DT_QINT8, ..., DT_DOUBLE, DT_COMPLEX64, DT_COMPLEX128, DT_BOOL, DT_BFLOAT16]
  device='XLA_CPU'; dtype in [DT_UINT8, DT_QUINT8, DT_UINT16, DT_INT8, DT_QINT8, ..., DT_DOUBLE, DT_COMPLEX64, DT_COMPLEX128, DT_BOOL, DT_BFLOAT16]
  device='CPU'

     [[{{node Const_1}}]]

tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Const_1: Could not satisfy explicit device specification '/device:GPU:1' because no supported kernel for GPU devices is available.

Will the model be supported on GPU?

[paulegradie]

Why are you using compat.v1 with TF2.1? This should be totally unnecessary unless there is some massive backward compatibility requirement (like legacy code at a company).

If you are using the model as a standalone, you can just access the signature by name can use the embed as a callable object. Models with multiple input signatures will need those signatures accessed by dict lookup.

If you are using it as part of another model, the preferred way is to load the module as a hub.KerasLayer layer and use that in a custom model.

...That being said, I also don't see usemqa/3 using GPU cores. Wtf!

[pertschuk]

Second that, using use QA model/3 / K80 GPU. Not utilizing GPU cores

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.module = hub.load('https://tfhub.dev/google/universal-sentence-encoder-qa/3')


    def rank(self, query: str, choices: List[str],
             filter_results: type(defaults.filter_results) = defaults.filter_results
             ) -> Tuple[List[int], List[float]]:
        questions = [query]

        question_embeddings = self.module.signatures['question_encoder'](
            tf.constant(questions))
        response_embeddings = self.module.signatures['response_encoder'](
            input=tf.constant(choices),
            context=tf.constant(choices))

        scores = np.inner(question_embeddings['outputs'], response_embeddings['outputs'])
        scores = np.reshape(scores, (-1,))
        sorted_indices = list(np.argsort(scores)[::-1])
        return sorted_indices, scores[sorted_indices]

nvidia-smi shows

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   73C    P0    73W / 149W |     70MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15272      C   /usr/bin/python3                              59MiB |
+-----------------------------------------------------------------------------+

[arnoegw]

Looks like there's several issues here. I'll try to go step by step.

@sarahwie, I suspect the error you get is from using a Python value list_of_events that gets converted to a tf.constant under the device scope. Making a tf.constant outside that scope makes your code run for me. (It's beyond tensorflow/hub to argue about that TF pitfall.)

EDIT: Point being, this is a tf.constant of dtype string.