Keras: Attribute Error: Custom Generator object has no attribute 'shape'

[fjur]

I am using a custom data generator, when I try using fit_generator it throws an error saying "object has no attribute 'shape'". This is being run on a Google Colab environment.

I followed this simple tutorial for the implementation of the generator:
https://medium.com/datadriveninvestor/keras-training-on-large-datasets-3e9d9dbc09d4

Data is being inputted as a list.

Tensorflow version: 1.13
Keras version: 2.2.4

Error:
/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training.pyc in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1424 use_multiprocessing=use_multiprocessing,
1425 shuffle=shuffle,
-> 1426 initial_epoch=initial_epoch)
1427
1428 def evaluate_generator(self,

/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_generator.pyc in model_iteration(model, data, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch, mode, batch_size, **kwargs)
113 batch_size=batch_size,
114 epochs=epochs - initial_epoch,
--> 115 shuffle=shuffle)
116
117 do_validation = validation_data is not None

/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/training_generator.pyc in convert_to_generator_like(data, batch_size, steps_per_epoch, epochs, shuffle)
375
376 # Create generator from NumPy or EagerTensor Input.
--> 377 num_samples = int(nest.flatten(data)[0].shape[0])
378 if batch_size is None:
379 raise ValueError('You must specify batch_size')

AttributeError: 'MY_Generator' object has no attribute 'shape'

gist: https://gist.github.com/fjur/2815f235f84b8b666107207599482428

[ceshine]

I ran into the same problem. Importing the Sequence class from Tensorflow fixed it for me:

from tensorflow.python.keras.utils.data_utils import Sequence

[fjur]

Thanks that solved it! I was using from keras.utils import Sequence which seemed to be causing issues.

[Nestak2]

@ceshine @fjur
Using from tensorflow.python.keras.utils.data_utils import Sequence solved the issue for me, too, but then I got the error message:

Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(self._args, *self._kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/data_utils.py", line 742, in _run
sequence = list(range(len(self.sequence)))
TypeError: 'numpy.float64' object cannot be interpreted as an integer

Do you know how to fix this?
My generator is this (you can slo take a look at my notebook) :

from numpy.random import uniform, randint
from tensorflow.python.keras.preprocessing.image import ImageDataGenerator
import numpy as np
from tensorflow.python.keras.utils.data_utils import Sequence

class CustomImagesGenerator(Sequence):
    def __init__(self, x, zoom_range, shear_range, rescale, horizontal_flip, batch_size=32):
        self.x = x
        self.zoom_range = zoom_range
        self.shear_range = shear_range
        self.rescale = rescale
        self.horizontal_flip = horizontal_flip
        self.batch_size = batch_size
        self.__img_gen = ImageDataGenerator()
        self.__batch_index = 0

    def __len__(self):
        # steps_per_epoch, if unspecified, will use the len(generator) as a number of steps.
        # hence this
        return np.floor(self.x.shape[0]/self.batch_size)

    @property
    def shape(self):
        return self.x.shape

    def next(self):
        return self.__next__()

    def __next__(self):
        start = self.__batch_index*self.batch_size
        stop = start + self.batch_size
        self.__batch_index += 1
        if stop > len(self.x):
            raise StopIteration
        transformed = np.array(self.x[start:stop])  # loads from hdf5
        for i in range(len(transformed)):
            zoom = uniform(self.zoom_range[0], self.zoom_range[1])
            transformations = {
                'zx': zoom,
                'zy': zoom,
                'shear': uniform(-self.shear_range, self.shear_range),
                'flip_horizontal': self.horizontal_flip and bool(randint(0,2))
            }
            transformed[i] = self.__img_gen.apply_transform(transformed[i], transformations)
        import pdb;pdb.set_trace()
        return transformed * self.rescale`

And I call it like this:

import h5py
import tables 

in_hdf5_file = tables.open_file("gdrive/My Drive/Colab Notebooks/dataset.hdf5", mode='r')
images = in_hdf5_file.root.train_img

my_gen = CustomImagesGenerator(
    images,
    zoom_range=[0.8, 1],
    batch_size=32,
    shear_range=6, 
    rescale=1./255, 
    horizontal_flip=False
)

classifier.fit_generator(my_gen, steps_per_epoch=100, epochs=1, verbose=1)

[marshsh]

For me, the problema was that I was inputting a list of entry values x = [a,b,c,d,e,f,g], and a value for the outcome y = z.
But you need to bundle them into np.arrays that have shape, and the first shape number of the input must be the same length as that of the output:
x = np.array([[a,b,c,d,e,f,g]])
y = np.array([z])

That's all I was missing when I got that error.

[dterg]

I’m facing the same issue with multiple outputs. Single output works fine, i.e returning X, Y but when returning X, [y1, y2] get the above error 😕. Also tried returning a dictionary X, {‘out1’: y1, ‘out2’: y2} but still get the same issue that list doesn’t have the attribute shape.

[marshsh]

Try turning everything into an np.array()
And make sure the inputs and outputs have the same length. (Which will determine the length of the batch)

... to make a batch of length 1, surround input and output with another np.array([...])
{'input' : np.array([x]), 'out1' : np.array([y1]), 'out2' : np.array([y2]) }

you need the [] inside the np.array() because if you have a single value ( y = 3.5 ) then np.array(3.5) = 3.5 is still not a np.array() with a shape.

[sainatarajan]

I have this exact same error and importing the lines mentioned above does not seem to work. I have two systems with exact same TensorFlow and Keras library versions. On one it runs perfectly while on the other it fails.

[Donaldidus]

I had the same problem. It worked fine in the jupyter notebook but if I put the exact cell code into a python file and imported the class it would throw the error.

The problem was, I imported keras in the python file with

import keras

Changing it to:

from tensorflor import keras

fixed the problem problem for me.

I guess the naming is ambiguous.

[sainatarajan]

I had to solve it by creating a new Conda environment and installing TensorFlow and Keras.

[Arjun-Arvindakshan]

I am currently working on a sales prediction project by using timeseriesgenerator.
Everything seems to be going well until the fitting part. It returns

AttributeError: ‘TimeseriesGenerator’ object has no attribute ‘shape’

the code goes like this:
train = np.array(train)
test = np.array(test)
n_input = 30
n_features = 1
generator = TimeseriesGenerator(train,train,length = n_input,batch_size=16)
model = keras.models.Sequential()
model.add(keras.layers.LSTM(100,activation = ‘relu’,return_sequences=True,input_shape (n_input,n_features)))
model.add(keras.layers.LSTM(50,activation = ‘relu’,return_sequences = True))
model.add(keras.layers.LSTM(25))
model.add(keras.layers.Dense(1))
model.fit_generator(generator,epochs = 100)

when it reaches fitting it shows this error.
both training and testing were passed as numpy arrays.
could anyone please tell what might be a possible reason behind it?

[sainatarajan]

@Arjun-Arvindakshan Either create a new Keras/TF environment and try your code there or try some of the solutions mentioned above in my previous post or completely move to PyTorch. Since your model looks very small, it should be easier to move to PyTorch rather than experimenting with other solutions.

[Arjun-Arvindakshan]

@Arjun-Arvindakshan Either create a new Keras/TF environment and try your code there or try some of the solutions mentioned above in my previous post or completely move to PyTorch. Since your model looks very small, it should be easier to move to PyTorch rather than experimenting with other solutions.

But there should be a logical explanation as to which a time series generator has this problem,right?
Why isnt it accepting this attribute?

[sainatarajan]

@Arjun-Arvindakshan You're right about it. I have not used a time series generator, but I am working on Computer Vision models where I need to load a huge dataset of images for training. I think both of our cases must be similar. Even though I have solved the issue, by running in a new anaconda environment, I have completely moved my codebase to PyTorch because I don't want the same error to happen again in my work in the future. I don't know why this is issue has been closed since many of us have been getting this error lately.

[MADHAVAN001]

I faced the same issue with incompatible types from keras and tf.keras. I noticed that while upgrading to Tensorflow 2.0, this was happening. It was not intuitive as to why there must be a difference here.

Thanks to the above solution to from tf.keras import Sequence. :)

[RSwarnkar]

My script kept getting error. While importing the PreProcessing from keras:
Use below:
from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator
Instead of:
from keras.preprocessing.sequence import TimeseriesGenerator

[GiganticRay]

from tensorflow.python.keras.utils.data_utils import Sequence
OMG, thanks a lot! this drop me in a dilema one day, i used a tool-kit: keras-video-generators in my project, but it just report that error above, and then i checked its src code and i find its way to import Sequence is wrong, and finally that works when i change that with yours！

[optimalnooo]

I face with the same issue.
If your generator returns a list (something like: [X, Y]), try return a tuple (X, Y).
It helps me.