Keras: Fit generator error executing Carvana unet example

[manningn]

I'm trying to execute the unet_linux.R example in Ubuntu 16.04 with Python 2.7.
When I run the last bit of code to fit the model:

model %>% fit_generator(
  train_iterator,
  steps_per_epoch = as.integer(length(train_index) / batch_size), 
  epochs = epochs, 
  validation_data = val_iterator,
  validation_steps = as.integer(length(val_index) / batch_size),
  verbose = 1,
  callbacks = callbacks_list
)

I get the following error:

Error occurred in generator: task 1 failed - "missing value where TRUE/FALSE needed"
Epoch 1/30
Error occurred in generator: task 1 failed - "missing value where TRUE/FALSE needed"
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  StopIteration: 

Detailed traceback: 
  File "/home/user/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/user/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/training.py", line 1415, in fit_generator
    initial_epoch=initial_epoch)
  File "/home/user/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/training_generator.py", line 177, in fit_generator
    generator_output = next(output_generator)
  File "/home/user/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/utils/data_utils.py", line 785, in get
    raise StopIteration()

Any suggestions? I also posted this over at stackoverflow.

[skeydan]

Hi,
I can't currently get the dataset from Kaggle and so can't test, but did you try removing the parallel code and see if you can get a traceback / find out where exactly the error occurs?
You could also try executing train_generator by itself (possibly line-by-line) to track down the problem...

[mjbajwa]

Hey @skeydan and @manningn,

I'm running the same code but on a different subset of images. I removed the parallelization bit and the model training started, but now getting the following error after the first epoch:

Epoch 1/30
237/237 [==============================] - 107s 454ms/step - loss: 1.0996 - : 0.5632 - val_loss: 1.1010 - val_: 0.5691
 Show Traceback

 Rerun with Debug
 Error in py_call_impl(callable, dots$args, dots$keywords) : 
  RuntimeError: Evaluation error: arguments imply differing number of rows: 1, 0. 

Traceback points to the following (haven't dug deep but will report back once I do):

19.
stop(structure(list(message = "RuntimeError: Evaluation error: arguments imply differing number of rows: 1, 0.", 
    call = py_call_impl(callable, dots$args, dots$keywords), 
    cppstack = structure(list(file = "", line = -1L, stack = c("/home/mjbajwa/R/x86_64-pc-linux-gnu-library/3.4/reticulate/libs/reticulate.so(Rcpp::exception::exception(char const*, bool)+0x7a) [0x7f0d350151aa]", 
    "/home/mjbajwa/R/x86_64-pc-linux-gnu-library/3.4/reticulate/libs/reticulate.so(Rcpp::stop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x27) [0x7f0d35015307]",  ... 
18.
python_function at call.py#21
17.
on_epoch_end at callback.py#32
16.
on_epoch_end at callbacks.py#77
15.
fit_generator at training_generator.py#247
14.
fit_generator at training.py#1415
13.
wrapper at interfaces.py#91
12.
(structure(function (...) 
{
    dots <- py_resolve_dots(list(...))
    result <- py_call_impl(callable, dots$args, dots$keywords) ... 
11.
do.call(func, args) 
10.
call_generator_function(object$fit_generator, list(generator = generator, 
    steps_per_epoch = as.integer(steps_per_epoch), epochs = as.integer(epochs), 
    verbose = as.integer(verbose), callbacks = normalize_callbacks(view_metrics, 
        callbacks), validation_data = validation_data, validation_steps = as_nullable_integer(validation_steps),  ... 
9.
fit_generator(., train_iterator, steps_per_epoch = as.integer(length(train_index)/batch_size), 
    epochs = epochs, validation_data = val_iterator, validation_steps = as.integer(length(val_index)/batch_size), 
    verbose = 1) 
8.
function_list[[k]](value) 
7.
withVisible(function_list[[k]](value)) 
6.
freduce(value, `_function_list`) 
5.
`_fseq`(`_lhs`) 
4.
eval(quote(`_fseq`(`_lhs`)), env, env) 
3.
eval(quote(`_fseq`(`_lhs`)), env, env) 
2.
withVisible(eval(quote(`_fseq`(`_lhs`)), env, env)) 
1.
model %>% fit_generator(train_iterator, steps_per_epoch = as.integer(length(train_index)/batch_size), 
    epochs = epochs, validation_data = val_iterator, validation_steps = as.integer(length(val_index)/batch_size), 
    verbose = 1) 

[skeydan]

Hi,

I've downloaded the images and after a little poking around, I get the same error as you @mjbajwa.

I've initially removed all callbacks (being unsure about "val_python_function"), as well as all validation code itself, thus reducing the call to fit to a simple

model %>% fit_generator(
  train_iterator,
  steps_per_epoch = as.integer(length(train_index) / batch_size), 
  epochs = epochs
)

Interestingly, the stacktrace shows a code path through callbacks.py and callback.py all the same!

> model %>% fit_generator(
+   train_iterator,
+   steps_per_epoch = as.integer(length(train_index) / batch_size), 
+   epochs = epochs
+ )
Epoch 1/1
1/1 [==============================] - 32s 32s/step - loss: 1.0333 - : 0.4751
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  RuntimeError: Evaluation error: arguments imply differing number of rows: 1, 0.

Detailed traceback: 
  File "/home/key/anaconda3/envs/tf-10-cpu/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/key/anaconda3/envs/tf-10-cpu/lib/python3.6/site-packages/keras/engine/training.py", line 1415, in fit_generator
    initial_epoch=initial_epoch)
  File "/home/key/anaconda3/envs/tf-10-cpu/lib/python3.6/site-packages/keras/engine/training_generator.py", line 247, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/home/key/anaconda3/envs/tf-10-cpu/lib/python3.6/site-packages/keras/callbacks.py", line 77, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/home/key/R/x86_64-redhat-linux-gnu-library/3.5/keras/python/kerastools/callback.py", line 32, in on_epoch_end
    self.r_on_epoch_end(epoch, logs)
  File "/home/key/R/x86_64-redhat-linux-gnu-library/3.5/reticulate/python/rpytools/
> 

To be continued....

[skeydan]

OK - having written the above, I was thinking, so what else happens at end of epoch... such as, metrics are calculated...

Replacing that custom loss/metric by "mse", it runs:

 model %>% compile(
    optimizer = optimizer_rmsprop(lr = 0.0001),
    #loss = bce_dice_loss,
    loss = "mse",
    metrics = "mse"
    #metrics = c(dice_coef)
  )

Epoch 1/1
1/1 [==============================] - 55s 55s/step - loss: 0.2310 - mean_squared_error: 0.2310

So it should be related to that loss function ...

[skeydan]

Okay! So it works if we use custom_metric to wrap the dice_coef:

 model %>% compile(
    optimizer = optimizer_rmsprop(lr = 0.0001),
    loss = bce_dice_loss,
    metrics = custom_metric("dice_coef", dice_coef)
  )

[skeydan]

Okay wrapping up (having uncommented all callbacks again) the only change necessary to make the example run should be the above:

 model %>% compile(
    optimizer = optimizer_rmsprop(lr = 0.0001),
    loss = bce_dice_loss,
    metrics = custom_metric("dice_coef", dice_coef)
  )

Epoch 1/1
6/6 [==============================] - 60s 10s/step - loss: 0.9215 - dice_coef: 0.5509 - val_loss: 2.5594 - val_dice_coef: 0.4965

[mjbajwa]

Excellent catch @skeydan! Training continues past the first epoch like a charm. Thank you so much!

[skeydan]

thanks for the collaboration! :-)