Ludwig: Problems of training image feature

[allenkao]

I found training simple image classification is difficult to reduce the loss. it's may be from resizing of image into hdf5 format.
When using default "crop_or_pad" resizing method, hdf5 data contains non-normalized image data which is required for making model converged.

In [7]: f["image_path_data"][0,:,:,:]
Out[7]:
array([[[105, 164, 244],
[ 93, 152, 232],
[ 85, 146, 226],
...,
[ 84, 152, 237],
[ 82, 150, 235],
[ 79, 147, 232]],

When inspecting hdf5 using "interpolate" resizing method, it's even worse that most image values contains zeros or few ones.

In [14]: f["image_path_data"][0,:,:,2]
Out[14]:
array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8)

In [15]: import numpy as np

In [16]: np.sum(f["image_path_data"][0,:,:,:])
Out[16]: 4

In [17]: np.sum(f["image_path_data"][1,:,:,:])
Out[17]: 2

In [18]: np.sum(f["image_path_data"][3,:,:,:])
Out[18]: 0

Basically ludwig initial release seems a little buggy for image feature training.

[w4nderlust]

Thanks for the feedback @allenkao . Every initial release of any large enough software contains bug, no matter how thoroughly you tested it, and we are here trying to solve all the bugs that show up as quickly as possible, a little bit of understanding (and maybe help) would be much more appreciated.
Anyway, we tested on both MNIST and CIFAR-10 and with a small amount of hyperparameter tuning we were able to get results really close to state of the art on those datasets.

We will have a look into the issue with the interpolate option. I suspect it's a datatype issue, as there's an implicit assumption of the type to be int8, and that may be the issue here, the output of the interpolation may actually be a normalized float and that makes for a 0 value ending up in the hdf5.

Regarding the normalization, we can add an optional "normalize" parameter to the image encoder that will perform that.

@ydudin3 can you look into this please?

[allenkao]

I also used the following model on CIFAR10 dataset but it just early stop after 3 epochs without loss/accuracy gain. Not sure if it's Windows-specific issue.

training:
  batch_size: 128
  epochs: 50

input_features:
  -
    name: image_path
    type: image
    in_memory: false
    encoder: stacked_cnn

output_features:
  -
    name: class
    type: category

[w4nderlust]

You have to play with the hyperparameters.

[ydudin3]

@allenkao - this commit https://github.com/uber/ludwig/commit/a119e4c41b35178abb555a1d3a79e37803d195e1 should have fixed the issue with having a bunch of zeros after interpolation.

[w4nderlust]

So interpolation is fixed and we are on the way to work on normalization as an enhancement. Feel free to open the issue again if the commit doesn't solve your issue.