Autokeras: Upgrade raw image reading to imageio.imread

[mlaradji]

Bug Description

I have folders data/train and data/val that contain jpg images. I created two labels.csv (in data/train and data/val) file that contains two columns: 'File Name' and 'Label', and filled them with the labels of their respective image sets. I can successfully run the following code:

from autokeras.image_supervised import ImageClassifier, load_image_dataset

x_train, y_train = load_image_dataset(csv_file_path="data/train/labels.csv",
                                      images_path="data/train")
print(x_train.shape)
print(y_train.shape)

x_val, y_val = load_image_dataset(csv_file_path="data/val/labels.csv",
                                    images_path="data/val")

with the output being:

(13750,)
(13750,)

I run into errors when trying to use x_train. When executing the following code:

if __name__ == '__main__':

    clf = ImageClassifier(verbose=True)
    clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
    clf.final_fit(x_train, y_train, x_val, y_val, retrain=True)
    y = clf.evaluate(x_val, y_val)
    print(y)

the following error is raised:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/Software/anaconda3/lib/python3.6/site-packages/autokeras/image_supervised.py in _validate(x_train, y_train)
     24     try:
---> 25         x_train = x_train.astype('float64')
     26     except ValueError:

ValueError: setting an array element with a sequence.

During handling of the above exception, another exception occurred:

[OUTPUT TRUNCATED]

The output of print(x_train) is:

array([array([[[169, 173, 174],
        [169, 173, 174],
        [169, 173, 174],
        ...,
        [187, 187, 179],
        [184, 183, 178],
        [182, 181, 176]],

       [[169, 173, 174],
        [169, 173, 174],
        [169, 173, 174],
        ...,
        [184, 184, 176],
        [182, 181, 176],
        [180, 179, 174]],

       [[168, 172, 171],
        [168, 172, 171],
        [168, 172, 171],
        ...,
        [183, 183, 175],
        [180, 179, 174],
        [179, 178, 173]],

       ...,

[OUTPUT TRUNCATED]

       [[ 42,  83, 111],
        [ 40,  75, 103],
        [ 69,  96, 123],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]],

       [[ 45,  86, 114],
        [ 49,  84, 112],
        [ 79, 106, 133],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]],

       [[ 40,  81, 109],
        [ 51,  86, 114],
        [ 84, 111, 138],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]]], dtype=uint8)], dtype=object)

Reproducing Steps

Steps to reproduce the behavior:

• Step 1: Import dataset using autokeras.image_supervised.load_image_dataset from a folder with jpg images and a labels.csv file.
• Step 2: Execute clf = ImageClassifier(), clf.fit(x_train, y_train).

Expected Behavior

I expected training on the dataset to start.

Setup Details

• OS type and version: Ubuntu 18.04.1 LTS
• Python: 3.6.5
• autokeras: 0.2.14
• scikit-learn: 0.19.2
• numpy: 1.14.5
• keras: 2.2.2
• scipy: 1.1.0
• tensorflow: 1.10.1
• pytorch: None

[raijinspecial]

I'm getting the same problem using autokeras on colab.
When I brought in the mnist example, after the reshape x_train was (60000, 28, 28, 1), but using load_image_dataset on jpgs is also giving me a 1D array of arrays. I'm unsure how to reshape it properly.

[mlaradji]

I think I've figured out why I was running into problems. The problem was that my jpg images were not of the same resolution.

Fix?

I normalized my images to the same resolution by adding the following lines, and a new option normalized_size, to autokeras.image_supervised.read_images:

import cv2
...
                img = cv2.imread(img_path)
                img = cv2.resize(img, dsize=normalized_size, interpolation=cv2.INTER_CUBIC)

Executing the following lines results in the desired output:

x_train, y_train = load_image_dataset(csv_file_path="data/train/labels.csv",
                                      images_path="data/train", normalized_size = (100,100))
print(x_train.shape)
print(y_train.shape)

Output:

(13750, 100, 100, 3)
(13750,)

Running the training then proceeds without error.

Pull Request

I think requiring that the images be of the same resolution is good behavior, as I've read that usually speeds up the training. I think at least one of the following functionality could be added to autokeras:

1. I think it is useful having an option in autokeras.image_supervised.load_image_dataset, or perhaps a separate function, to normalize an image dataset to the same resolution. However, there are possible problems that could occur. Reducing an image's resolution could cause data loss, and, depending on the way the resolution is changed, the images might be changed too much and it might throw off the training. Note that my knowledge in this area is very limited.
2. help(autokeras.image_supervised.load_image_dataset) could include a line indicating that images need to be of the same size.
3. Allow image datasets that have multiple resolutions. This is probably the most desirable, but I foresee that it would require changes to functions outside of image_supervised.py.

If one (or more) of the options seems desirable, I could code in the relevant sections and submit a pull request.

Changed Functions

For reference, the following are the changed functions in image_supervised.py:

def read_images(img_file_names, images_dir_path, normalized_size = None):
    """Read the images from the path and return their numpy.ndarray instance.
        Return a numpy.ndarray instance containing the training data.

    Args:
        img_file_names: List containing images names.
        images_dir_path: Path to the directory containing images.
    """
    x_train = []
    if os.path.isdir(images_dir_path):
        for img_file in img_file_names:
            img_path = os.path.join(images_dir_path, img_file)
            if os.path.exists(img_path):
                #img = ndimage.imread(fname=img_path)
                img = cv2.imread(img_path)
                img = cv2.resize(img, dsize=normalized_size, interpolation=cv2.INTER_CUBIC)
                if len(img.shape) < 3:
                    img = img[..., np.newaxis]
                x_train.append(img)
            else:
                raise ValueError("%s image does not exist" % img_file)
    else:
        raise ValueError("Directory containing images does not exist")
    return np.asanyarray(x_train)


def load_image_dataset(csv_file_path, images_path, normalized_size = None):
    """Load images from the files and labels from a csv file.

    Second, the dataset is a set of images and the labels are in a CSV file.
    The CSV file should contain two columns whose names are 'File Name' and 'Label'.
    The file names in the first column should match the file names of the images with extensions,
    e.g., .jpg, .png.
    The path to the CSV file should be passed through the `csv_file_path`.
    The path to the directory containing all the images should be passed through `image_path`.

    Args:
        csv_file_path: CSV file path.
        images_path: Path where images exist.

    Returns:
        x: Four dimensional numpy.ndarray. The channel dimension is the last dimension.
        y: The labels.
    """
    img_file_name, y = read_csv_file(csv_file_path)
    x = read_images(img_file_name, images_path, normalized_size = normalized_size)
    return np.array(x), np.array(y)

[raijinspecial]

Awesome, great work! I figured it was probably due to the different resolutions. I used this code:
def proc_images():
"""
Returns two arrays:
x is an array of resized images
y is an array of labels
"""

drone="yes"

x = [] # images as arrays
y = [] # labels Infiltration or Not_infiltration
WIDTH = 128
HEIGHT = 128

for img in images:
    base = os.path.basename(img)
    finding = labels["Label"][labels["File Name"] == base].values[0]

    # Read and resize image
    full_size_image = cv2.imread(img)
    x.append(cv2.resize(full_size_image, (WIDTH,HEIGHT), interpolation=cv2.INTER_CUBIC))

    # Labels
    if drone in finding:
        #finding = str(disease)
        finding = 1
        y.append(finding)

    else:
        #finding = "Not_" + str(disease)
        finding = 0
        y.append(finding)

return x,y

which I borrowed from a kaggle kernel: https://www.kaggle.com/crawford/resize-and-save-images-as-numpy-arrays-128x128

to resize the images, and this worked for autokeras. It made me think if there is any automated, or in principle method to find the optimal resizing parameters for any arbitrary set of images, as you rightly point out that resizing can result in information loss or distortion. I don't have a solution for that yet but if I find one I will report it.

[raijinspecial]

The following changes to image_supervised.py worked for me. Used imageio.imread to bring in images and skimage.transform.resize to make all images the same dimension by giving (x, y, z) where z=channels.

I tried to make a PR but I'm a noob and I think I did it in the wrong place.

def read_images(img_file_names, images_dir_path, normalized_size=None):
    """Read the images from the path and return their numpy.ndarray instance.
        Return a numpy.ndarray instance containing the training data.

    Args:
        img_file_names: List containing images names.
        images_dir_path: Path to the directory containing images.
        normalized_size: tuple specifying resize shape (x, y, z), z = channels
    """
    x_train = []
    if os.path.isdir(images_dir_path):
        for img_file in img_file_names:
            img_path = os.path.join(images_dir_path, img_file)
            if os.path.exists(img_path):
                # img = ndimage.imread(fname=img_path)
                img = imageio.imread(img_path)
                img = resize(img, output_shape=normalized_size, anti_aliasing=True)
                if len(img.shape) < 3:
                    img = img[..., np.newaxis]
                x_train.append(img)
            else:
                raise ValueError("%s image does not exist" % img_file)
    else:
        raise ValueError("Directory containing images does not exist")
    return np.asanyarray(x_train)