Keras: Issue with Generator: Trying to call index_generator from next function but getting the error : Traceback (most recent call last): generator_ouput=next(output_generator) StopIteration.

[coder1508]

It would be great if someone can suggest me how I can fix this stop iteration problem. Based on the property of the generator whenever we call this function, once we’ve exhausted all of the yields within index_generator() running next() again results in a StopIteration error and in my case once reaches to the last batch of the directory iterator, it will stop working. This issue is not allowing me to compile my model.

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• [ ] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [ ] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [ ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[datumbox]

@coder1508 if you update the ticket with a snippet that reproduces the problem I'll be happy to have a look.

[dhlrd]

This is a bug of keras 2.0.9, when i change keras back to 2.0.8, using generator to generate data for training is OK.

[datumbox]

Off the top of my head, there are two main changes on the generators on 2.0.9 that could affect you:

1) The Iterator class inherits from the Sequence instead of object. If you have custom generator that inherits from this class you might need to update it (hard to provide directions without more info). Some minor changes exist also on the fit_generator() method which uses different Enqueuers depending on the type of generator that you provide.
2) The second is related to the GeneratorEnqueuer class which now supports finite generators and throw StopIterator exception if the undelying iterator is finite and is exhausted. Note that the original API on 2.0.8 required you to pass always an infinite generator (else the implementation would break), so the update is supposed to allow you to use also finite ones in case you are doing destributed predictions using Spark. The update should not have any effects on infinite generators.

Unfortunately without a snippet that reproduces the problem it is very hard to tell which one is bitting you. If you provide a snippet, I'm happy to take a look.

[anguszxd]

Hi, @datumbox , I think I meet the same problem when I use keras 2.0.9. I'm trying a Keras implementation of FCN, here is the link https://github.com/aurora95/Keras-FCN. In this repository, the author uses a DataGenerator class to load image and label. This part can be find in file utils/SegDataGenator.py .
When I try to use this code, I get the following error:

lr: 0.010000
Epoch 1/250
Traceback (most recent call last):
File "train.py", line 246, in
label_suffix=label_suffix, ignore_label=ignore_label, label_cval=label_cval)
File "train.py", line 163, in train
class_weight=class_weight
File "/home/zxd/.local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(args, *kwargs)
File "/home/zxd/.local/lib/python2.7/site-packages/keras/engine/training.py", line 2046, in fit_generator
generator_output = next(output_generator)
File "/home/zxd/.local/lib/python2.7/site-packages/keras/utils/data_utils.py", line 518, in get
raise StopIteration(e)
StopIteration

The error message is not very detailed, but that all what I get. The error occurs when keras begins to use model.fit_generator(...) function in trian.py.
But it works well when I use keras version 2.0.5.
This may be an other reference: https://stackoverflow.com/questions/46302911/what-raises-stopiteration-in-mine-keras-model-fit-generator. The quizzer seems to meet the same problem. I don't know what version of keras he uses ...

Hope that I have provided enough information for you ...

[datumbox]

Hey @anguszxd,

Unfortunately it is not feasible for me to read the entire code base of the other project and debug it. Please note that if this a problem related to another lib, it might be worth raising the issue on that project.

If you manage to reproduce the problem in a short, pure-keras snippet I'll be happy to debug it for you.

[datumbox]

BTW with a very quick look on the lib it seems that your problem is caused by the 1st point that I suggested. The SegDirectoryIterator inherits from Iterator which now extends Sequence. I believe the problem will be resolved if the SegDataGenerator class is updated to either use Keras' new generator logic or stop inheriting from Iterator.

[tomonari-t]

Now by this PR solves this problem at aurora95/Keras-FCN

[yokisir]

I use Keras 2.1.2 and I counter the same error "StopIteration", and after I change the value of generator's steps it work. I hope that can help.

[candou1611]

I also encounter the same issue, @smallYoki How to change steps_per_epoch?
@coder1508 Did you fix this issue or find any workaround?

[yokisir]

You can try set the value of 'steps_per_epoch' = numberofallsamples // mini_batchsize. Is better to make sure the number of all samples is integral multiple of batch size. Else, delete samples to make sure it.

[MerouaneB]

I have the same error too(StopIteration in the next function of fit_generator). I have tried all the issues proposed, however i still get the error. I am using keras 2.1.4. any solutions ?
Thanks

[MerouaneB]

File "/usr/local/lib/python3.4/site-packages/keras/utils/data_utils.py", line 564, in get
inputs = self.queue.get(block=True).get()
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(args, *kwds))
File "/usr/local/lib/python3.4/site-packages/keras/utils/data_utils.py", line 390, in get_index
return _SHARED_SEQUENCES[uid][i]
File "/usr/local/lib/python3.4/site-packages/keras/preprocessing/image.py", line 799, in __getitem__
return self._get_batches_of_transformed_samples(index_array)
File "/usr/local/lib/python3.4/site-packages/keras/preprocessing/image.py", line 845, in _get_batches_of_transformed_samples
raise NotImplementedError
NotImplementedError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "cnn.py", line 176, in
main(sys.argv)
File "cnn.py", line 172, in main
_main()
File "cnn.py", line 161, in _main
trainModel(train_generator, val_generator, model, initial_epoch)
File "cnn.py", line 89, in trainModel
initial_epoch=initial_epoch)
File "/usr/local/lib/python3.4/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(args, *kwargs)
File "/usr/local/lib/python3.4/site-packages/keras/engine/training.py", line 2217, in fit_generator
generator_output = next(output_generator)
File "/usr/local/lib/python3.4/site-packages/keras/utils/data_utils.py", line 570, in get
six.raise_from(StopIteration(e), e)
File "", line 3, in raise_from
StopIteration