Addons: test_utils.run_distributed fails on GPU

Created on 27 Feb 2020  路  2Comments  路  Source: tensorflow/addons

The error is:

tensorflow.python.framework.errors_impl.UnknownError: Not enough GPUs to create virtual devices. num_gpus_to_use: 1 #virtual_devices: 2

This happens since number of physical GPUs != number of virtual GPUs.

The line which throws the error is here: gpu_device.cc.

The documentation for tf.config.set_logical_device_configuration seems to run that bit of code successfully.

Relevant log: https://source.cloud.google.com/results/invocations/606014e4-04b9-4cd5-a907-1b7a2661375a/targets/tensorflow_addons%2Fubuntu%2Fgpu%2Fpy3%2Fpresubmit/log

bug

Most helpful comment

@Squadrick I'm willing to take a look at this one. #969 needs to run distributed tests.

All 2 comments

@Squadrick I'm willing to take a look at this one. #969 needs to run distributed tests.

Closing this as it's going to be tracked in #1682

Was this page helpful?
0 / 5 - 0 ratings