Baselines: HER, out of memory using 3 or more CPUs and one GPU

Created on 11 Mar 2018 · 8Comments · Source: openai/baselines

Running the following HER command on my machine (Ubuntu 16.04, Tensorflow 1.5.0, one Titan X GPU, Python 3.5.2, latest version of baselines as of today, etc.) seems to work:

(py3-tensorflow) daniel@computer-name:~/baselines$ python -m baselines.her.experiment.train --num_cpu 2
2018-03-11 10:42:00.828727: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-03-11 10:42:00.833988: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-03-11 10:42:01.035000: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-11 10:42:01.035688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties: 
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 10.72GiB
2018-03-11 10:42:01.035702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-03-11 10:42:01.036552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-11 10:42:01.036967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties: 
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 10.60GiB
2018-03-11 10:42:01.036979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
Logging to /tmp/openai-2018-03-11-10-42-01-211699
Logging to /tmp/openai-2018-03-11-10-42-01-238422

after this the statistics and logs are reported which make sense and indicate improved performance.

I noticed while that was running, the nvidia-smi command shows that there are two python commands running but one use far more GPU memory than the other:

(py3-tensorflow) daniel@computer-name:~/baselines$ nvidia-smi 
Sun Mar 11 10:43:42 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 00000000:01:00.0  On |                  N/A |
| 29%   52C    P2    74W / 250W |  12035MiB / 12194MiB |     34%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15162      G   /usr/lib/xorg/Xorg                           625MiB |
|    0     15613      G   compiz                                       371MiB |
|    0     16006      G   /usr/lib/firefox/firefox                       2MiB |
|    0     16308      C   ...l/seita-venvs/py3-tensorflow/bin/python   547MiB |
|    0     16309      C   ...l/seita-venvs/py3-tensorflow/bin/python 10449MiB |
|    0     18716      G   /usr/lib/firefox/firefox                       2MiB |
+-----------------------------------------------------------------------------+

(py3-tensorflow) daniel@computer-name:~/baselines$ python -m baselines.her.experiment.train --num_cpu 3
2018-03-11 10:43:55.864451: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-03-11 10:43:55.872111: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-03-11 10:43:55.872111: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-03-11 10:43:56.149303: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-11 10:43:56.149754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties: 
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 10.68GiB
2018-03-11 10:43:56.149813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-03-11 10:43:56.153272: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-11 10:43:56.153800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties: 
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 10.44GiB
2018-03-11 10:43:56.153829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-03-11 10:43:56.153863: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-03-11 10:43:56.154212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties: 
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 10.44GiB
2018-03-11 10:43:56.154239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-03-11 10:43:56.333249: E tensorflow/stream_executor/cuda/cuda_driver.cc:936] failed to allocate 224.44M (235339776 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
Logging to /tmp/openai-2018-03-11-10-43-56-333366
Logging to /tmp/openai-2018-03-11-10-43-56-335464
Logging to /tmp/openai-2018-03-11-10-43-56-358276

and the out of memory error causes the program to abort.

I assumed naively that I could fix this by adjusting the ddpg.py file in HER:

    def _create_network(self, reuse=False):
        logger.info("Creating a DDPG agent with action space %d x %s..." % (self.dimu, self.max_u))

        #self.sess = tf.get_default_session()
        # Add these instead of the default session
        config = tf.ConfigProto()
        config.gpu_options.per_process_gpu_memory_fraction = 0.2 
        self.sess = tf.Session(config=config)

        if self.sess is None:
            self.sess = tf.InteractiveSession()

Unfortunately this does not seem to work due to un-initialized variables.
(I can post the full error message if it helps.)

The closest existing issue seems to be this one https://github.com/openai/baselines/issues/70 but where @olegklimov suggests that "it's [PPO] supposed to use the same GPU from several MPI workers. More that each MPI should use its own GPU on multi-GPU machine or multi-machine MPI." but I only have one GPU on this machine, and I'm not sure if there are subtle differences with PPO vs HER implementations.

Any advice would be appreciated. Thanks!

Source

DanielTakeshi

Most helpful comment

Thanks for the information.

It might be useful to add to the HER README the machine and specs that OpenAI uses to run these commands.

DanielTakeshi on 11 Mar 2018

👍2

All 8 comments

@DanielTakeshi I have not tried HER yet but many of the baseline algorthems need to add the following setting if you want to run multiple algorthems in parrelel on the same machine

tf_config.gpu_options.allow_growth = True

However, I do belive HER supports multiple workers and that there is a setting you can change. I remeber reading that the author had set this to 1 because he did not want to assume howmany cores a user might have, but that to re-produce the results from the paper, that one should be using a maching with 20 cores and set it to 19

Sohojoe on 11 Mar 2018

@Sohojoe This was addressed by @matthiasplappert in this issue https://github.com/openai/baselines/issues/314

However I don't know where to precisely put the tf_config.gpu_options.allow_growth = True command in the code for me to run this, if that's indeed the solution.

DanielTakeshi on 11 Mar 2018

Hmm ... I think I have fixed this by changing baselines/common/tf_util.py in make_session by changing the tf_config.gpu_options:

def make_session(num_cpu=None, make_default=False):
    """Returns a session that will use <num_cpu> CPU's only"""
    if num_cpu is None:
        num_cpu = int(os.getenv('RCALL_NUM_CPU', multiprocessing.cpu_count()))
    tf_config = tf.ConfigProto(
        inter_op_parallelism_threads=num_cpu,
        intra_op_parallelism_threads=num_cpu)
    #tf_config.gpu_options.allocator_type = 'BFC'
    tf_config.gpu_options.per_process_gpu_memory_fraction = 0.2 
    if make_default:
        return tf.InteractiveSession(config=tf_config)
    else:
        return tf.Session(config=tf_config)

The nvidia-smi command now shows the split allocation among the lone GPU:

(py3-tensorflow) daniel@computer-name:~/baselines$ nvidia-smi 
Sun Mar 11 13:20:20 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 00000000:01:00.0  On |                  N/A |
| 38%   65C    P2   103W / 250W |  11747MiB / 12194MiB |     41%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     15162      G   /usr/lib/xorg/Xorg                           625MiB |
|    0     15613      G   compiz                                       371MiB |
|    0     16006      G   /usr/lib/firefox/firefox                       2MiB |
|    0     18716      G   /usr/lib/firefox/firefox                       2MiB |
|    0     22579      C   ...l/seita-venvs/py3-tensorflow/bin/python  2673MiB |
|    0     22580      C   ...l/seita-venvs/py3-tensorflow/bin/python  2673MiB |
|    0     22581      C   ...l/seita-venvs/py3-tensorflow/bin/python  2673MiB |
|    0     22582      C   ...l/seita-venvs/py3-tensorflow/bin/python  2673MiB |
+-----------------------------------------------------------------------------+

DanielTakeshi on 11 Mar 2018

👍1

i have this:

    tf_config = tf.ConfigProto(
        inter_op_parallelism_threads=num_cpu,
        intra_op_parallelism_threads=num_cpu)
    tf_config.gpu_options.allow_growth = True
    tf_config.gpu_options.allocator_type = 'BFC'

Sohojoe on 11 Mar 2018

👍1

Thanks for the information.

It might be useful to add to the HER README the machine and specs that OpenAI uses to run these commands.

DanielTakeshi on 11 Mar 2018

👍2

The HER code is indeed not intended for GPU but instead is meant to use on a machine with many CPU cores. You can of course run it on GPU but that's probably going to cause issues when you have multiple MPI workers. I'll add a note to the readme that gives some details on what machine we used in our experiments.