Keras: Can't use R keras fit_generator() with a custom data generator

[PVitorino]

This is my first post here so any help and/or advice on problem description is quite welcome.

That being said, let's head into the problem I've been facing for a couple of hours:

In order to train a vgg16 model, I'm using a custom R data generator to preprocess the data which comes from keras:flow_from_directory. And although I don't have nearly enough CPU processing power on my laptop, I was able to make it work with a reduced amount of batch_size, epochs, and steps_per_epoch. A simpler code to reproduce my success can be found here:
https://stackoverflow.com/questions/53357901/using-a-custom-r-generator-function-with-fit-generator-keras-.

But the problem starts once I get to my GPU-equipped computer and try using the fit_generator function with this custom R generator. I'm simply stuck at the first step of the first epoch without any response from R console whatsoever. And that happens to both my model and to the example model listed above. Here is what I get:

> library(keras)
> # example data
> data <- data.frame(
+   x = runif(80),
+   y = runif(80),
+   z = runif(80)
+ )
> # example generator
> data_generator <- function(data, x, y, batch_size) {
+   
+   # start iterator
+   i <- 1
+   
+   # return an iterator function
+   function() {
+
+     # reset iterator if already seen all data
+     if ((i + batch_size - 1) > nrow(data)) i <<- 1
+ 
+     # iterate current batch's rows
+     rows <- c(i:min(i + batch_size - 1, nrow(data)))
+     
+     # update to next iteration
+     i <<- i + batch_size
+     
+     # create container arrays
+     x_array <- array(0, dim = c(length(rows), length(x)))
+     y_array <- array(0, dim = c(length(rows), length(y)))
+     
+     # fill the container
+     x_array[1:length(rows), ] <- data[rows, x]
+     y_array[1:length(rows), ] <- data[rows, y]
+     
+     # return the batch
+     list(x_array, y_array)
+     
+   }
+   
+ }
> # set-up a generator
> gen <- data_generator(
+   data = data.matrix(data),
+   x = 1:2, # it is flexible, you can use the column numbers,
+   y = c("y", "z"), # or the column name
+   batch_size = 32
+ )
> # set up a simple keras model
> model <- keras_model_sequential() %>% 
+   layer_dense(32, input_shape = c(2)) %>% 
+   layer_dense(2)
2020-07-16 19:37:32.040393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-07-16 19:37:35.098731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-07-16 19:37:35.116724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 970 computeCapability: 5.2
coreClock: 1.266GHz coreCount: 13 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 208.91GiB/s
2020-07-16 19:37:35.117090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-07-16 19:37:35.124119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-16 19:37:35.129961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-07-16 19:37:35.132197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-07-16 19:37:35.137769: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-07-16 19:37:35.141461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-07-16 19:37:35.153316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-16 19:37:35.153789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-16 19:37:35.154322: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-07-16 19:37:35.164029: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x24a290c5750 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-16 19:37:35.164373: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-16 19:37:35.165056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 970 computeCapability: 5.2
coreClock: 1.266GHz coreCount: 13 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 208.91GiB/s
2020-07-16 19:37:35.165386: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-07-16 19:37:35.165896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-16 19:37:35.166327: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-07-16 19:37:35.166681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-07-16 19:37:35.166922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-07-16 19:37:35.167133: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-07-16 19:37:35.167332: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-16 19:37:35.167569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-16 19:37:35.680951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-16 19:37:35.681298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-07-16 19:37:35.681438: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-07-16 19:37:35.681704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2991 MB memory) -> physical GPU (device: 0, name: GeForce GTX 970, pci bus id: 0000:01:00.0, compute capability: 5.2)
2020-07-16 19:37:35.684551: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x24a4bef6320 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-16 19:37:35.684840: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 970, Compute Capability 5.2
> model %>% compile(
+   optimizer = "rmsprop",
+   loss = "mse"
+ )
> # fit using generator
> model %>% fit_generator(
+   generator = gen,
+   steps_per_epoch = 100, # will auto-reset after see all sample
+   epochs = 10,
+   max_queue_size = 50
+   
+ )
2020-07-16 19:37:48.296325: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
Epoch 1/10
  1/100 [..............................] - ETA: 0s - loss: 0.4254

As I've previously said, while running this same example code with a CPU installation of Keras in my laptop, it runs smoothly.

Has anyone ever faced a similar problem or at least know what could be causing this? If needed, I'm glad to provide more information in order to better clarify my issue.

Thanks in advance!

Additional info: I can normally train models using GPU installation of Tensorflow when I try a standard generator such as the one I get directly from flow_from_directory

[dfalbel]

Hi this is a known bug. see #986
There's a deadlock somewhere when using R generators...

The current workaround is to use keras::train_on_batch()

We will try to solve this for the next Keras release.

[PVitorino]

Hi this is a known bug. see #986
There's a deadlock somewhere when using R generators...

The current workaround is to use keras::train_on_batch()

We will try to solve this for the next Keras release.

Thanks for the response! So, with this alternative approach, there is no room for learning rate/checkpoints callbacks or validation data metrics, right? Or is there a way around I could use to get these into keras:train_on_batch()? Could not see anything on the documentation.

[mattwarkentin]

I have had a similar issue of _deadlocking_ when trying to use a custom R batch generator. I am also interested in how train_on_batch() can be used as a workaround. It is not clear to me how exactly it should be used. Any insight would be appreciated.

[mattwarkentin]

Hi @dfalbel,

I made an attempt to use train_on_batch() with an R data generator to avoid the _deadlocking_. I am hoping you can take a look and see if this looks like it makes sense, even though it is just a toy example. The custom generator just creates random samples from iris, but could be extended to more complex data structures. I added a progress bar to give the user some feedback about progress and performance metrics.

This should be a fully reproducible example, assuming the requisite packages are installed. Hopefully others may find this helpful. Relevant to #986, #1059, and #1073.

library(keras)
library(tidyverse)
library(cli)
library(progress)

batch_size <- 64
epochs <- 200
steps_per_epoch <- 5

batch_generator <- function(data, batch_size) 
{
  function() {
    ids <- sample(1:nrow(data), batch_size, replace = FALSE)
    list(
      as.matrix(data[ids, -5]), 
      to_categorical(as.integer(data[ids, 5]) - 1)
      )
  }
}

model <- keras_model_sequential()

model %>% 
  layer_dense(units = 20, input_shape = c(4)) %>% 
  layer_dense(units = 50) %>% 
  layer_dropout(0.25) %>% 
  layer_dense(units = 50) %>% 
  layer_dropout(0.25) %>% 
  layer_dense(units = 3, activation = "softmax")

summary(model)

model %>% 
  compile(
    optimizer = "rmsprop",
    loss = "categorical_crossentropy",
    metrics = c("accuracy")
  )

generator <- batch_generator(iris, batch_size)

pb <- progress_bar$new(
  total = epochs,
  clear = FALSE,
  format = "Epoch: :current/:total :elapsedfull [:bar] Loss: :loss, Acc: :acc",
  complete = symbol$full_block,
  current = "-"
)

fit_custom <- function()
{
  new_batch <- generator()
  metrics <- train_on_batch(model, new_batch[[1]], new_batch[[2]])
  tibble(
    .loss = metrics[[1]],
    .accuracy = metrics[[2]]
  )
}

map_dfr(
  1:epochs,
  function(x) {
    epoch_steps_all <- map_dfr(1:steps_per_epoch, ~fit_custom())
    epoch_steps_last <- slice(epoch_steps_all, steps_per_epoch)
    loss <- format(round(pull(epoch_steps_last, .loss), 3), nsmall = 3)
    acc <- format(round(pull(epoch_steps_last, .accuracy), 3), nsmall = 3)
    pb$tick(tokens = list(loss = loss, acc = acc))
    return(epoch_steps_last)
  }, 
  .id = ".epoch"
  )

Epoch: 200/200 00:00:15 [█████████████████] Loss: 0.076, Acc: 0.969

# A tibble: 200 x 3
   .epoch .loss .accuracy
   <chr>  <dbl>     <dbl>
 1 1      1.75      0.359
 2 2      1.05      0.547
 3 3      1.19      0.594
 4 4      0.991     0.609
 5 5      0.796     0.688
 6 6      0.745     0.688
 7 7      0.585     0.734
 8 8      0.540     0.734
 9 9      0.434     0.781
10 10     0.424     0.812
# … with 190 more rows

[alexuadler]

Thanks for the code @mattwarkentin

Fairly new to fit generators. I'm trying to follow this tutorial for a skipgram generator here and running into the issue:
Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 500000 batches). You may need to use the repeat() function when building your dataset.

Your fix seems promising for that, but I'm not 100% this is the same underlying problem. If you think it is, how would I adapt your iris solution for a skipgram generator.

Thank you!

EDIT: After some more digging, this seems to be an issue with the tutorial i was following from the RStudio AI Blog. https://github.com/rstudio/keras/issues/244 This references missing/invalid sentences. Removing such sentences fixes my error. I'll keep my comment here in case anyone ends up here searching for the solution to the issue with that tutorial.

[rdinnager]

Hi, this issue is causing me some major headaches, so I am hoping it gets fixed soon. However, I discovered a way to get custom R generators to work with fit_generator, so I am posting it here as a workaround others can use, but also because hopefully it can provide some clue as to what the problem is. I don't know why this works, but if you create a new object from your generator using new_gen <- keras:::as_generator.function(orig_gen), and then use new_gen as your generator function in fit_generator, it works! This is strange because fit_generator calls as_generator on the function internally. The only difference I can think of is that if you run as_generator on the function first, it returns a Python iterator, which then is not treated as a "main thread" generator by fit_generator, as a regular R function would be. And this apparently works. So I suppose this may not be a "safe" workaround, I'm not too familiar with these threading concepts myself. I'm also not sure if performance might be negatively impacted by doing this? Anyway, here is a reproducible example based on @mattwarkentin 's example from above:

library(keras)

batch_size <- 64
epochs <- 200
steps_per_epoch <- 5

batch_generator <- function(data, batch_size)
{
  function() {
    ids <- sample(1:nrow(data), batch_size, replace = FALSE)
    list(
      as.matrix(data[ids, -5]),
      to_categorical(as.integer(data[ids, 5]) - 1)
    )
  }
}

model <- keras_model_sequential()

model %>%
  layer_dense(units = 20, input_shape = c(4)) %>%
  layer_dense(units = 50) %>%
  layer_dropout(0.25) %>%
  layer_dense(units = 50) %>%
  layer_dropout(0.25) %>%
  layer_dense(units = 3, activation = "softmax")

summary(model)
#> Model: "sequential"
#> ________________________________________________________________________________
#> Layer (type)                        Output Shape                    Param #     
#> ================================================================================
#> dense (Dense)                       (None, 20)                      100         
#> ________________________________________________________________________________
#> dense_1 (Dense)                     (None, 50)                      1050        
#> ________________________________________________________________________________
#> dropout (Dropout)                   (None, 50)                      0           
#> ________________________________________________________________________________
#> dense_2 (Dense)                     (None, 50)                      2550        
#> ________________________________________________________________________________
#> dropout_1 (Dropout)                 (None, 50)                      0           
#> ________________________________________________________________________________
#> dense_3 (Dense)                     (None, 3)                       153         
#> ================================================================================
#> Total params: 3,853
#> Trainable params: 3,853
#> Non-trainable params: 0
#> ________________________________________________________________________________

model %>%
  compile(
    optimizer = "rmsprop",
    loss = "categorical_crossentropy",
    metrics = c("accuracy")
  )

## This code hangs on first step
model %>%
  fit_generator(batch_generator(iris, batch_size),
                steps_per_epoch = steps_per_epoch,
                epochs = epochs)

## This code works
gen <- keras:::as_generator.function(batch_generator(iris, batch_size))

model %>%
  fit_generator(gen,
                steps_per_epoch = steps_per_epoch,
                epochs = epochs)

#>2020-08-31 09:54:01.435990: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic
#>library libcublas.so.11
#>Epoch 1/200
#>5/5 [==============================] - 1s 174ms/step - loss: 1.8746 - accuracy: 0.3938
#>Epoch 2/200
#>5/5 [==============================] - 0s 53ms/step - loss: 1.3048 - accuracy: 0.5312
#>Epoch 3/200
#>5/5 [==============================] - 0s 50ms/step - loss: 1.1217 - accuracy: 0.5562
#>Epoch 4/200
#>5/5 [==============================] - 0s 49ms/step - loss: 0.9743 - accuracy: 0.5781
#>Epoch 5/200
#>5/5 [==============================] - 0s 49ms/step - loss: 0.8745 - accuracy: 0.5875
#> ...

[mattwarkentin]

Thanks for sharing your code, @rdinnager.

[mattwarkentin]

To add a little more context for what works and what doesn't, and maybe some clues for figuring out the issue...

## Coercing the generator to a python iterator/generator works
gen <- keras:::as_generator.function(batch_generator(iris, batch_size))

model %>%
    fit_generator(gen,
                  steps_per_epoch = steps_per_epoch,
                  epochs = epochs)

class(gen)



md5-13548bbb02631b9f2217c5b68d9f5814



[1] "python.builtin.iterator"  "python.builtin.generator"
[3] "python.builtin.object"   



md5-ce601201b622c8aa7dfbecf92bc0cb2e



```r
class(gen)



md5-13548bbb02631b9f2217c5b68d9f5814



[1] "python.builtin.iterator"      
[2] "rpytools.generator.RGenerator"
[3] "python.builtin.object"  

reticulate::py_iterator() is supposed to return threadsafe iterators, but it doesn't seem to work with keras::fit_generator(). Not really sure why it doesn't work.

Some Python APIs use generators to parallelize operations by calling the generator on a background thread and then consuming its results on the foreground thread. The py_iterator() function creates threadsafe iterators by ensuring that the R function is always called on the main thread (to be compatible with R's single-threaded runtime) even if the generator is run on a background thread.

[maju116]

Thanks @rdinnager I used your solution for my custom generator for object detection and image segmentation: https://github.com/maju116/platypus

[rdinnager]

@maju116 Glad it helped, though I guess it is not ideal for a package since it needs a keras function that isn't exported. But until this gets an official fix, I guess we have little choice. And thanks for sharing your package, it looks really cool. It is good to see more well-established and useful architectures getting R implementations!

[maju116]

@rdinnager for now I've just copied function code into my pkg

[maju116]

@rdinnager @mattwarkentin @dfalbel I found another strange thing: using keras:::as_generator.function with custom generator works with fit_generator and predict_generator, but not with evaluate_generator (sometimes it returns an error like with yolo3_generator, sometimes it returns all zeros like with segmentation_generator from my package)