Ludwig: gpu training allocates too much memory

[baderth]

Maybe I missed something, but trying to get the image classification example going, i always end up with the following error just after the training starts:

_Allocator (GPU_0_bfc) ran out of memory trying to allocate 10.99GiB. Current allocation summary follows._

No matter what images I use, or if I do it with resize_image in the input_features or bigger or tinier width/height, it always wants to allocate those 10.99GiB (which my GTX1080 hasnt :))

Even when using _--gpu_fraction_ it wants those 10.99GiB (tried values from 0.1 to 0.5)

I'm a tensorfloor noob, so maybe I just got something completely wrong.

Definition:
_input_features:
-
name: image_path
type: image
encoder: stacked_cnn
resize_image: true
width: 28
height: 28

output_features:
-
name: class
type: category_

[ksteinfe]

I had a similar issue, and tried the --gpu_fraction setting to no avail. Ended up addressing it by adjusting the batch size, like this (note the lack of dashes in the training section)...

input_features:
  -
    name: image_path
    type: image
    encoder: stacked_cnn
  -
    name: some_number
    type: numerical

output_features:
  -
    name: label
    type: category

training:
    batch_size: 64

[w4nderlust]

So, TensorFlow by default allocates all the memory on the GPU, and then has an internal allocation system. The fact that it runs out of memory means that there is not enough memory on your GPU for the size of both weights and activation tensors you are computing.

--gpu_fraction instructs TensorFlow to allocate only that fraction of the memory at the beginning of the training, but then it can increase that amount elastically as needed. In your case it need more than your full memory, so it obviously needs also more than the fraction you specify.

You have a few solutions, to use in combination:

1. Resize your images
2. decrease the number of filters of the convolutional stack (admittedly the default values are a bit high for most GPUs, in the next release I'm going to decrease them)
3. decrease the batch size
   A combination of these will decrease memory consumption and let you train your model.

I'm closing this, let me know if this doesn't solve your problem.