Autokeras: BaseCollectiveExecutor::StartAbort Out of range: End of sequence

[SunYanCN]

Bug Description

when I run IMDB example, I got this:

Epoch 1/30
2019-11-22 08:22:06.636069: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
    800/Unknown - 48s 60ms/step - loss: 0.4854 - accuracy: 0.76162019-11-22 08:22:53.047730: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
     [[Fill_2/_41]]
2019-11-22 08:22:53.047733: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
800/800 [==============================] - 59s 73ms/step - loss: 0.4854 - accuracy: 0.7616 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 2/30
799/800 [============================>.] - ETA: 0s - loss: 0.2679 - accuracy: 0.89422019-11-22 08:23:50.083179: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
     [[IteratorGetNext/_2]]
2019-11-22 08:23:50.083288: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
800/800 [==============================] - 57s 72ms/step - loss: 0.2679 - accuracy: 0.8942 - val_loss: 0.2739 - val_accuracy: 0.8848

Setup Details

Include the details about the versions of:

• OS type and version:
• Python: 3.6
• autokeras: 1.0
• tensorflow: 2.0.0

[haifeng-jin]

https://github.com/tensorflow/tensorflow/issues/31509
It seems it is an issue of tensorflow.
We can only expect the next version of tensorflow to fix it.
Thanks.

[phemmer]

Hi, I'm from the future, using 2.1.0, and it didn't fix it :-)

[PavelKovalets]

Same issue here, with some custom code, is there any way to fix it?

[MasoudBNJ]

I had same issue, when using tensorflow.keras. I had written a custom data generator using python yield instruction. My problem was that I used generator for validation data, but didn't determine the validation_steps (So the data collector was going to out of range):

model.fit(generate_train_data(),
          steps_per_epoch=train_size // batch_size,
          validation_data=generate_val_data(),
          epochs=epochs)

And fixed it by using validation steps:

model.fit(generate_train_data(),
          steps_per_epoch=train_size // batch_size,
          validation_data=generate_val_data(),
          validation_steps=validation_size // batch_size,
          epochs=epochs)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.