Keras: fit_generator crashes though keras.utils.data_utils.Sequence was used
When model.fit_generator is used with workers=0 and subclasses of keras.utils.data_utils.Sequence for both training and validation data, API of Sequence is not recognized inside evaluate_generator, it raises:
File ".../keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File ".../keras/engine/training.py", line 1415, in fit_generator
initial_epoch=initial_epoch)
File ".../keras/engine/training_generator.py", line 230, in fit_generator
validation_steps,
File ".../keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File ".../keras/engine/training.py", line 1469, in evaluate_generator
verbose=verbose)
File ".../keras/engine/training_generator.py", line 298, in evaluate_generator
else:
ValueError: `steps=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps` or use the `keras.utils.Sequence` class.
Example code:
from keras import Sequential
from keras.layers import Dense
from keras.utils.data_utils import Sequence
import numpy as np
class Dataset(Sequence):
def __getitem__(self, index):
return np.random.uniform(size=(16, 8)), np.random.uniform(size=(16, 1))
def __len__(self):
return 128
model = Sequential([Dense(4, activation='relu', input_shape=(8,)),
Dense(1, activation='sigmoid')])
model.compile(loss='mse', optimizer='adam')
model.fit_generator(generator=Dataset(), validation_data=Dataset(),
workers=0)
Issue can be fixed here by replacing:
if isinstance(val_data, Sequence):
val_enqueuer_gen = iter(val_data)
with
if isinstance(val_data, Sequence):
val_enqueuer_gen = iter(val_data)
validation_steps = len(val_data)
arquolo
All 40 comments
Just updated to Keras 2.2.3
Bug still persists, but now it raises:
ValueError Traceback (most recent call last)
<ipython-input-3-9dd18d05ed53> in <module>
3 model.compile(loss='mse', optimizer='adam')
4 model.fit_generator(generator=Dataset(), validation_data=Dataset(),
----> 5 workers=0)
...\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
89 warnings.warn('Update your `' + object_name + '` call to the ' +
90 'Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper
...\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1416 use_multiprocessing=use_multiprocessing,
1417 shuffle=shuffle,
-> 1418 initial_epoch=initial_epoch)
1419
1420 @interfaces.legacy_generator_methods_support
...\keras\engine\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
233 val_enqueuer_gen,
234 validation_steps,
--> 235 workers=0)
236 else:
237 # No need for try/except because
...\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
89 warnings.warn('Update your `' + object_name + '` call to the ' +
90 'Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper
...\keras\engine\training.py in evaluate_generator(self, generator, steps, max_queue_size, workers, use_multiprocessing, verbose)
1470 workers=workers,
1471 use_multiprocessing=use_multiprocessing,
-> 1472 verbose=verbose)
1473
1474 @interfaces.legacy_generator_methods_support
...\keras\engine\training_generator.py in evaluate_generator(model, generator, steps, max_queue_size, workers, use_multiprocessing, verbose)
300 steps = len(generator)
301 else:
--> 302 raise ValueError('`steps=None` is only valid for a generator'
303 ' based on the `keras.utils.Sequence` class.'
304 ' Please specify `steps` or use the'
ValueError: `steps=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps` or use the `keras.utils.Sequence` class.
I reviewed source of training_generator and found that in:
124 elif val_gen:
125 val_data = validation_data
126 if isinstance(val_data, Sequence):
127 val_enqueuer_gen = iter_sequence_infinite(generator)
128 else:
129 val_enqueuer_gen = val_data
in line 127 val_enqueuer_gen uses generator which is training data, instead of val_data, and it doesn't set validation_steps though val_data is keras.Sequence
arquolo
on 3 Oct 2018
Bug still exists in keras 2.2.4
ValueError: `steps_per_epoch=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps_per_epoch` or use the `keras.utils.Sequence` class.
And it also warns:
WARNING:tensorflow:Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.
It seems that there is no work around, if I feed the steps_per_epoch like so:
model.fit_generator(generator=x,
validation_data=x,
steps_per_epoch=x.__len__(), # https://github.com/keras-team/keras/issues/10855
use_multiprocessing=True,
workers=6,
shuffle=False)
I still get an exception:
AttributeError: 'Sequential' object has no attribute 'metrics_names'
KIC
on 10 Nov 2018
@KIC
Provide more details, please:
- what is
xandmodel? - which version of tensorflow do you use?
- where AttributeError was raised from (provide full traceback)?
Also, it looks like your x is not instance of keras.utils.Sequence at all.
arquolo
on 10 Nov 2018
Here we go:
class DataGenerator(keras.utils.Sequence):
'Generates data for Keras'
def __init__(self, dataframe, features, labels, train_test_factor=0.8, batch_size=32, lstm_memory_size=52*5):
'Initialization'
self.dataframe = dataframe
self.batch_size = batch_size
self.features = features
self.labels = labels
self.train_test_factor = train_test_factor
self.shuffle = False
self.on_epoch_end()
def __len__(self):
'Denotes the number of batches per epoch'
return int(np.floor(self.dataframe.shape[0] * self.train_test_factor)) - self.batch_size
def __getitem__(self, index):
'Generate one batch of data like [batch_size, lstm_memory_size, features]'
# features
features = np.apply_along_axis(np.hstack,
1,
self.dataframe[self.features] \
.iloc[index:index + self.batch_size].values)
# labels
labels = np.apply_along_axis(np.hstack,
1,
self.dataframe[self.labels] \
.iloc[index + 1:index + self.batch_size + 1].values)
return [features], [labels]
def on_epoch_end(self):
'Updates indexes after each epoch'
# later we could track/plot the model weights after each epoch
if self.shuffle == True:
print("dadasdasd -----------------------------")
raise ValueError('Shuffeling not implemented yet')
def get_dataframe(self):
return self.dataframe
and
x = DataGenerator(df, ["sigma", "cos_dow"], ["sigma"], 0.8, 2)
x.__getitem__(x.__len__() - 1)
isinstance(x, keras.utils.Sequence) #True
Finally
model.fit_generator(generator=x,
validation_data=x,
#steps_per_epoch=x.__len__(), # https://github.com/keras-team/keras/issues/10855
use_multiprocessing=True,
workers=6,
shuffle=False)
And the stack trace:
WARNING:tensorflow:Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-10-5b629adf0d9e> in <module>
21 use_multiprocessing=True,
22 workers=6,
---> 23 shuffle=False)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2175 use_multiprocessing=use_multiprocessing,
2176 shuffle=shuffle,
-> 2177 initial_epoch=initial_epoch)
2178
2179 def evaluate_generator(self,
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
66 steps_per_epoch = len(generator)
67 else:
---> 68 raise ValueError('`steps_per_epoch=None` is only valid for a'
69 ' generator based on the `keras.utils.Sequence`'
70 ' class. Please specify `steps_per_epoch` or use'
ValueError: `steps_per_epoch=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps_per_epoch` or use the `keras.utils.Sequence` class.
keras version is 2.2.4
tensorflow version was 1.10.1, is now upgraded to 1.12.0 but the issue remains
KIC
on 10 Nov 2018
Hi, i was struggling with the exact same error message today on Windows, Tensorflow 1.12.0.
For me it turned out that I used tensorflow import (import tensorflow.keras.utils.Sequence as sequence) for the sequence generator, but for the model I was training I used the keras import (from keras.layers import..... and from keras.models import....etc.)
changing to tensorflow.keras.layers, tensorflow.keras.models....etc worked for me.
rodvei
on 13 Nov 2018
indeed this solves the problem! this was not obvious! thx!
KIC
on 13 Nov 2018
So, it's not Keras issue, but Tensorflow one. That's why it's still closed.
By the way, tensorflow.keras isn't always updated with latest changes in keras, so it's better to use either only keras imports or only tensorflow.keras imports, never both.
arquolo
on 13 Nov 2018
Same issue here. no solution?
ghasemikasra39
on 16 Nov 2018
@ghasemikasra39 Traceback? Sample code? TF/Keras versions?
arquolo
on 16 Nov 2018
Same issue here.
Keras 2.2.4
tensorflow-gpu 1.12.0
synchro--
on 3 Dec 2018
@synchro-- traceback and code?
arquolo
on 3 Dec 2018
Same issue too.
keras.utils.Sequence class not recognize.
ValueError:
steps_per_epoch=Noneis only valid for a generator based on thekeras.utils.Sequenceclass. Please specifysteps_per_epochor use thekeras.utils.Sequenceclass.
Keras: 2.1.5 and later
tensorflow-gpu: 1.10.1
AA12321
on 4 Dec 2018
@AlessioA123 Update Keras to 2.2.4, and try again. If it's still fails provide sample code, otherwise I cannot help you
arquolo
on 4 Dec 2018
Tested with Keras 2.2.4 and get same error.
from keras.utils import Sequence
class Batchloader(Sequence):
def __init__(...):
def __len__(...):
def __getitem__(...):
A quick test
gen = Batchloader(...)
model.compile(...)
model.fit_generator(generator=gen, verbose=1, epochs=2)
Obtaining previous error.
Debugging i have see in /keras/engine/training.py on line: 2084 and after on line 2092
is_sequence = isinstance(generator, Sequence)
is_sequence is false, but generator (Batchloader) is a keras.utils.Sequence
AA12321
on 4 Dec 2018
@AA12321 how have you debugged line 2084 in keras/engine/training.py if there are only 1522 lines? Maybe you use tensorflow.keras.Model (which training.py is still broken) instead of keras.Model.
arquolo
on 4 Dec 2018
I don't undestand how can be tensorflow, traceback says:
/lib/python3.6/site-packages/keras/engine/training.py
without no reference to tensowflow.
It's a virtualenv with only tensorflow-gpu (not used inside scripts, only as backend for keras) and keras.
I have do some test but got the same issue with keras 2.1.5, 2.2.3, 2.2.4 and I use only keras.Model.
You suggest that is still the tensorflow bug?
AA12321
on 5 Dec 2018
@AA12321 run this:
import tensorflow as tf
print(tf.__version__, tf.__file__)
import keras
print(keras.__version__, keras.__file__)
According to traceback, you use keras installed to system, not virtualenv.
Maybe your virtualenv isn't activated.
arquolo
on 5 Dec 2018
You right, i don't understand how it's possible but the old venv doesn't work properly. I create a new one and get same error.
Got these
2.1.5 /envKeras/lib/python3.6/site-packages/keras/__init__.py
2.2.4 /envKeras/lib/python3.6/site-packages/keras/__init__.py
Both tensorflow and keras was installed with
pip3 install tensorflow-gpu
pip3 install keras==2.X.X
At least now training.py file have 1522 line.
AA12321
on 5 Dec 2018
@AA12321 So you created new virtualenv like:
# in project folder
python3 -m venv kerasenv
source ./kerasenv/bin/activate
python3 -m pip list
python3 -m pip install keras==2.2.4 tensorflow-gpu
python3 ./myscript.py
And it's still fails?
arquolo
on 5 Dec 2018
@arquolo Thank you for help and reply.
Yes still fails, same error, after doing all commands that you have reported (now all traceback message are from the right file path).
ValueError:
steps_per_epoch=Noneis only valid for a generator based on thekeras.utils.Sequenceclass. Please specifysteps_per_epochor use thekeras.utils.Sequenceclass.
AA12321
on 5 Dec 2018
@AA12321 Try code sample I provided in the first message. If it's also fails, then I cannot help.
arquolo
on 5 Dec 2018
Same issue on colab!
mirfan899
on 5 Dec 2018
@AA12321 @mirfan899 , You can try this notebook
arquolo
on 5 Dec 2018
@AA12321 Try code sample I provided in the first message. If it's also fails, then I cannot help.
@arquolo Tried your sample code and working.
So in keras 2.2.4 with sequential model works, i use a custom model that required keras 2.1.4 to load the weights, so i think that's my problem. Thanks!
AA12321
on 5 Dec 2018
The solution which I got is steps_per_epoch=len(x_train)/batch_size or whatever you want the number of steps for each epoch by setting steps_per_epoch in fit_generator
mirfan899
on 5 Dec 2018
same error here:
Trying to run the cifar 10 cnn example returns:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "C:\Program Files\JetBrains\PyCharm 2018.2.4\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2018.2.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/Omar/tensorflow_test/examples_keras_new/cifar10_cnn.py", line 118, in <module>
workers=4)
File "C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19\lib\site-packages\keras\engine\training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19\lib\site-packages\keras\engine\training_generator.py", line 55, in fit_generator
raise ValueError('`steps_per_epoch=None` is only valid for a'
ValueError: `steps_per_epoch=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps_per_epoch` or use the `keras.utils.Sequence` class.
from an anaconda environment with:
Keras 2.2.4
tensorflow-gpu 1.12.0
clean installation, full list of packages as follows:
(tensor19) C:\Users\Omar>conda list
# packages in environment at C:\Users\Omar\AppData\Local\conda\conda\envs\tensor19:
#
# Name Version Build Channel
_tflow_select 2.1.0 gpu anaconda
absl-py 0.6.1 py36_0 anaconda
astor 0.7.1 py36_0 anaconda
blas 1.0 mkl anaconda
ca-certificates 2018.03.07 0 anaconda
certifi 2018.10.15 py36_0 anaconda
cudatoolkit 9.0 1 anaconda
cudnn 7.1.4 cuda9.0_0 anaconda
cycler 0.10.0 <pip>
gast 0.2.0 py36_0 anaconda
grpcio 1.16.1 py36h351948d_1 anaconda
h5py 2.8.0 py36h3bdd7fb_2 anaconda
hdf5 1.10.2 hac2f561_1 anaconda
icc_rt 2017.0.4 h97af966_0 anaconda
intel-openmp 2019.1 144 anaconda
Keras 2.2.4 <pip>
keras-applications 1.0.6 py36_0 anaconda
keras-preprocessing 1.0.5 py36_0 anaconda
kiwisolver 1.0.1 <pip>
libprotobuf 3.6.1 h7bd577a_0 anaconda
markdown 3.0.1 py36_0 anaconda
matplotlib 3.0.2 <pip>
mkl 2018.0.3 1 anaconda
mkl_fft 1.0.6 py36hdbbee80_0 anaconda
mkl_random 1.0.1 py36h77b88f5_1 anaconda
numpy 1.15.4 py36ha559c80_0 anaconda
numpy-base 1.15.4 py36h8128ebf_0 anaconda
openssl 1.1.1 he774522_0 anaconda
pip 18.1 py36_0
protobuf 3.6.1 py36h33f27b4_0 anaconda
pyparsing 2.3.0 <pip>
python 3.6.8 h9f7ef89_0
python-dateutil 2.7.5 <pip>
PyYAML 3.13 <pip>
scikit-learn 0.20.2 <pip>
scipy 1.1.0 py36h4f6bf74_1 anaconda
setuptools 40.6.3 py36_0
six 1.11.0 py36_1 anaconda
sklearn 0.0 <pip>
sqlite 3.26.0 he774522_0
tensorboard 1.12.0 py36he025d50_0 anaconda
tensorflow 1.12.0 gpu_py36ha5f9131_0 anaconda
tensorflow-base 1.12.0 gpu_py36h6e53903_0 anaconda
tensorflow-gpu 1.12.0 h0d30ee6_0 anaconda
tensorflow-probability 0.5.0 <pip>
termcolor 1.1.0 py36_1 anaconda
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_0
werkzeug 0.14.1 py36_0 anaconda
wheel 0.32.3 py36_0
wincertstore 0.2 py36h7fe50ca_0
zlib 1.2.11 h62dcd97_3 anaconda
omarcr
on 5 Jan 2019
The solution which I got is
steps_per_epoch=len(x_train)/batch_sizeor whatever you want the number of steps for each epoch by settingsteps_per_epochinfit_generator
This works for me
majianjia
on 7 Jan 2019
I have keras 2.2.4 and tensorflow 1.12, when I downgrade to tensorflow 1.8.0, the problem goes away for cifar10_resnet.py (given in keras/examples). Might not be optimal soln, but at least there's a version of tensorflow for which sample code works.
smgutstein
on 8 Feb 2019
A simple work around would be adding a steps_per_epoch value to keras.models.Sequential().fit_generator(steps_per_epoch = "Constant here")
chason94
on 10 Feb 2019
The solution which I got is
steps_per_epoch=len(x_train)/batch_sizeor whatever you want the number of steps for each epoch by settingsteps_per_epochinfit_generatorThis works for me
It's works. Thanks!
TomLiu59
on 12 Feb 2019
@TomLiu59 @chason94 Your solution looks good. But if steps_per_epoch=len(x_train)/batch_size is added, isn't it taking very long time to train ?
ghost
on 15 Feb 2019
@TomLiu59 @chason94 Your solution looks good. But if steps_per_epoch=len(x_train)/batch_size) is added, isn't it taking very long time to train ?
May be I can try giving a constant instead of giving len(x_train)/batch_size.
ghost
on 15 Feb 2019
The solution which I got is
steps_per_epoch=len(x_train)/batch_sizeor whatever you want the number of steps for each epoch by settingsteps_per_epochinfit_generatorThis works for me
It's works. Thanks!
It's really works and helpful to me ,great!
voscar-zhang
on 15 Mar 2019
change steps_per_epoch=None to steps_per_epoch=nb_train_samples // batch_size
JanuaryThomas
on 19 Mar 2019
Using the solution mentioned here (posted by voscar-zhang) I get a new error:
ValueError: Must specify 'validation_steps' to perform validation when doing step-wise training.
cpagravel
on 19 Mar 2019
Using the solution mentioned here (posted by voscar-zhang) I get a new error:
ValueError: Must specify 'validation_steps' to perform validation when doing step-wise training.
Here is my working codes
model.fit_generator(train_generator, epochs=epochs, steps_per_epoch=nb_train_samples // batch_size,
validation_data=test_generator, nb_val_samples=nb_val_samples //batch_size,
class_weight='auto')
JanuaryThomas
on 20 Mar 2019
The solution which I got is
steps_per_epoch=len(x_train)/batch_sizeor whatever you want the number of steps for each epoch by settingsteps_per_epochinfit_generatorThis works for me
It's works. Thanks!
me too, it works!!!! thanks
鎴戜篃鍙互鐢ㄤ簡
m10416012
on 7 Apr 2019
Using datagen.sample_per_validation//BS still falls short of 56 samples for batch size 64. Any solution to fill the last batch?
Sawatdatta
on 30 Apr 2019
The solution which I got is
steps_per_epoch=len(x_train)/batch_sizeor whatever you want the number of steps for each epoch by settingsteps_per_epochinfit_generatorThis works for me
It's works. Thanks!
me too, it works!!!! thanks
鎴戜篃鍙互鐢ㄤ簡
works for me too, thanks!
XiongXiaoji
on 10 Jun 2019
Hello I have this issue:
AttributeError: 'ImageDataGenerator' object has no attribute 'fit_generator'
can you please tell me how to solve it?
also, I think I have no problem of library versions or installation requirements.
thanks
YavuzKahraman
on 6 May 2020
Related issues
snakeztc
路
3Comments
harishkrishnav
路
3Comments
zygmuntz
路
3Comments
KeironO
路
3Comments
NancyZxll
路
3Comments
Most helpful comment
The solution which I got is
steps_per_epoch=len(x_train)/batch_sizeor whatever you want the number of steps for each epoch by settingsteps_per_epochinfit_generator