Keras: fit_generator() more than twice as slow as fit()

[t-kalinowski]

Hello, I noticed that fit_generator() is significantly slower than fit(). I'm not sure if this is surprising behavior, or if it's an inevitable fact of life due to the overhead between calling an R function from python. I'm hopeful that there is some easy optimization that can put these two functions on equal footing.

Here is an example with some dummy data and a minimal keras model.

library(keras)

batch_size = 1024
steps_per_epoch = 1000
d = c(batch_size * steps_per_epoch, 128, 2)
X <- array(rnorm(prod(d)), d)
Y <- matrix(0, nrow = nrow(X), ncol = 10)
Y[cbind(seq_len(nrow(X)), sample(10, nrow(X), replace = TRUE ))] <- 1

new_model <- function() {
  keras_model_sequential() %>%
    layer_flatten( input = c(128, 2)) %>%
    layer_dense(10, use_bias = FALSE, activation = 'softmax') %>%
    compile(optimizer = 'adam', loss = 'mae')
}

new_data_generator <- function() {
  start <- 1L
  function() {
    end <- start + batch_size - 1L
    idx <- start:end
    if(end > nrow(X)) {
      over <- idx > nrow(X)
      idx[over] <- seq_len(sum(over))
    }

    on.exit(start <<- idx[batch_size] + 1L)
    list(X[idx,,], Y[idx,])
  }
}

generator <-  new_data_generator()

model <- new_model()
fit_timing <- system.time(fit(model, x = X, y = Y, batch_size = batch_size,
                              epochs = 3, verbose = 1))

model <- new_model()
fit_generator_timing <- system.time(fit_generator(model, generator = generator,
                          steps_per_epoch = steps_per_epoch, epochs = 3, verbose = 1))

fit_timing
#>    user  system elapsed 
#>  12.521   1.592   8.737
fit_generator_timing
#>    user  system elapsed 
#>  25.036   1.587  20.269

Created on 2018-04-09 by the reprex package (v0.2.0).

[jjallaire]

Yeah, I there is per-call overhead every time we go between R and Python (and in fact every time we call an R function and have to copy data out of it's return value). In this case we're doing it 3,000 times so we end up with a lot of overhead.

One thing we're working on is trying to move all data ingestion and preprocessing on to the TF graph (so not in Python or in R, but instead all C++ tensor operations). If you are using a generator to stream data from disk then it's likely this can be made to work well. If you are using a generator to dynamically generate data this might be harder. What's the nature of the data you are attempting to apply the generator to?

[t-kalinowski]

Thanks. That's not that surprising I guess, but it's unfortunate.

I would love to hear your suggestions for how best to setup feeding data
during training to maybe avoid this overhead. As it happens I have some
flexibility in that I can either choose to generate the data at runtime, or
pregenerate a large amount of data and then stream from disk. Currently, I
_am_ streaming pregenerated data from disk. I organized things into python pickle
files just because that was the most convenient, but I could just as easily
convert to another format if it enables streaming from disk easier.

I have a directory of a couple thousand pickle files that I pregenerated, then
I wrote a generator function that would at each epoch prepare a "large" data
set by loading a bunch of pickle files and combining then appropriately. This
"largeish" dataset (about 8gb) sits in RAM for the whole epoch, then then
for each batch step, the generator would feed batches from it, similarly to
the way chunks are fed above.

Each pickle file consists of a single 2 dimensional numpy array of dtype
float64 and shape (timesteps, channels). There are always two channels, but
the size of the timesteps axis varies. To form the training data I randomly
take slices of a fixed size along the timestep axis and combine then along the
first dimension. So, as an example, say using a timestep window size of 128,
yields a training X dataset of shape (?, 128, 2).

The Y (model output) is encoded in the filename of the picklefile, which I
parse with some regexes then convert to a numeric or a onehot. However, there
are multiple output types, and this approach of encoding the Y data in the
filename is not scaling very well as the problem-space I'm exploring grows. I
have been considering transitioning to encoding the metadata in perhaps a
python dictionary that travels with the pickle file, or doing something more
robust with a sqlite database that keeps track of the metadata with each file.

Do you have any recommendations about how to move all the data feeding onto
the tf graph? Right now, fit_generator() is too slow to be practical for my
needs. My next approach was going to be to do something like
for (i in 1:epochs) { DATA <- load_new_data_set() # loads about 8gb into ram fit(model, x = DATA$X, y = DATA$Y, epochs = 1) }
but that feels like it'll be cumbersome (and I think it means I lose the real-time Rstudio metrics).

[jjallaire]

Okay, I am completely convinced that we can solve your problem (and massively improve the performance of your ingest/preprocessing) using TensorFlow Datasets. We have an R interface to TF Datasets available here: https://tensorflow.rstudio.com/tools/tfdatasets/articles/introduction.html.

As illustrated in those docs, you can pass a dataset directly to fit_generator() and it will incur none of the additional overhead you are seeing now. Google has recently extended datasets to support direct loading into device memory (i.e. go direct to GPU with no intermediate load into CPU memory).

The R interface to datasets currently supports CSV files and TFRecord files (protobufs). Parallel and sharded reading is supported for all data sources. They just added support for SQLite (we don't yet have that in the R interface but could add it easily if you needed it).

If you think that TF Datasets looks like a reasonable fit with your data we'd be more than happy to work with you to make any changes to TF Datasets required to get things working. Maybe you could start by giving us a small sample of your data (or if it's confidential a simulated version of your data) along with the model you are building and we can see what it would take to use datasets.

[jjallaire]

FYI I just added support for SQLite datasets. Docs are here: https://tensorflow.rstudio.com/tools/tfdatasets/articles/introduction.html#sqlite-databases

[t-kalinowski]

Thank you @jjallaire! . I started looking through tfdatasets yesterday. It looks like it's going to be awesome once I figure out how to use it :).
Some thing that wasn't immediately obvious was how to create tfrecords from numpy arrays or R objects. I see the example here, but I still haven't fully worked through it.

I will take you up on your generous offer to help. It think the most efficient approach will be, if you agree, I will prepare a self-standing script that will create some dummy data files as RDS objects, then a separate script that loads those data files, preprocesses them, and feeds them to a minimal keras model. I'd be interested then in seeing how you (or someone from your team) translates that 2nd script and moves all the data loading and pre-processing onto the tf graph. It should be a good 'real-world' usecase to help figure out which parts of tfrecords could be fleshed out.

[jjallaire]

Yes, tfdatasets is currently too raw for us to promote more broadly, but our goal is to close this gap and eventually encourage everyone to use tfdatasets.

Your approach sounds exactly right: if you provide us with script(s) that do everything using stock R we should be able to figure out how to do the equivalent in tfdatasets on the TF graph. This will likely expose some holes we need to fill in tfdatasets but there's no better way for us to do this than working with real examples like yours.

cc @kevinykuo @javierluraschi @andrie @skeydan

[t-kalinowski]

I've posted a minimal gist solving a toy-problem here

[t-kalinowski]

@jjallaire let me know if you run into any issues running the code or have questions. Thank you again!

[jjallaire]

Thanks very much! I just chatted with @kevinykuo about it, he should be able to take a look late this week or early next week. My initial guess is that you'll want to write the data as TFRecord files (as opposed to pickled objects) but we'll soon see!

[kevinykuo]

Starting to look into this. Will report back with findings. Gotta say, the documentation on working with these things is pretty sparse 😐

[jjallaire]

Yes, the docs are ridiculously sparse. I view that as an opportunity -- if we can actually make this easy to use from R it will be a huge win!

[dfalbel]

I started a small project to write tfrecords from R. currently I can write matrix and sparse matrices, but supporting arrays should be fine.

you can take a look at tests to see how it works.

http://github.com/dfalbel/tfrecords

[t-kalinowski]

omg, programming in tensorflow feels trying to make your way barefoot through a pitch-black room full of coffee tables and legos strewn across the floor.

in any case, I started working on figuring out how to get my data into tfrecords. This might be helpful to you.
While preparing these R helpers, these two resources were invaluable:
the unit tests for protobuffers (especially the constructor, and the little lambda near the top)
https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/kernel_tests/parsing_ops_test.py#L1030

and this plain-English description:
https://planspace.org/20170323-tfrecords_for_humans/

# ------------- constructors ------------------

bytes_feature <- function(...) {
  v <- c(...)
  stopifnot(is.character(v) || is.raw(v))
  # make sure it makes it into python as a list, not a scalar or list of a list
  if(is_scalar(v)) v <- list(v)
  tf$train$Feature(bytes_list = tf$train$BytesList(value = v))
}

int64_feature <- function(...) {
  v <- as.integer(c(...))
  # make sure it makes it into python as a list, not a scalar or list of a list
  if(is_scalar(v)) v <- list(v)

  tf$train$Feature(int64_list = tf$train$Int64List(value = v))
}

float_feature <- function(...) {
  v <- as.double(c(...))
  # make sure it makes it into python as a list, but not a list of a list
  if(is_scalar(v)) v <- list(v)

  tf$train$Feature(float_list = tf$train$FloatList(value = v))
}

features <- function(...) {
  d <- as_dict(...)
  if(!isTRUE(all(vapply(py_to_r(d), is_valid_feature, TRUE))))
    stop("Each element passed to features() must be named and consist of either\n",
         "float_feature(), bytes_feature() or int64_feature()")
  tf$train$Features(feature = d)
}

feature_list <- function(...) {
  l <- list(...)
  if(is_scalar(l) && is.list(l[[1]]))
    l <- l[[1]]

  if(!is.null(names(l)))
    stop("Named arguments not allowed to feature_list")

  valid <- vapply(l, is_valid_feature, TRUE)
  if(!all(valid)) stop(
    "Each feature passed to feature_list() must be constructed with either\n",
    "int64_feature(), bytes_feature(), or float_feature()")

  if(!is_scalar(unique(vapply(l, feature_type, ""))))
    stop("Eall the elements in a feature_list() must be of the same type\n",
         "i.e., all made with int64_feature(), or all made with bytes_feature(),\n",
          "or all made with float_feature()")

  tf$train$FeatureList(feature = l)
}

feature_lists <- function(...) {
  d <- as_dict(...)
  stopifnot(is_dict(d))
  tf$train$FeatureLists(feature_list = d)
}

example <- function(...) {
  if(is_features(..1)) {
    if(!missing(..2))
      stop("Only one features() per example()")
    f <- ..1
  } else
    f <- features(...)

  tf$train$Example(features = f)
}

sequence_example <- function(context = NULL, feature_lists = NULL) {
  stopifnot(is_features(context))
  stopifnot(is_feature_lists(feature_lists))

  # b()
  do.call(tf$train$SequenceExample, compact(nlist(context, feature_lists)))
}


# ------------- end constructors ------------------
# ------------- checkers ------------------

is_feature <- function(x) {
  inherits(x, "tensorflow.core.example.feature_pb2.Feature")
}

is_valid_feature <- function(x) {
  isTRUE(feature_type(x) %in% c("float_list", "int64_list", "bytes_list"))
}

feature_type <- function(x) {
  if(!is_feature(x))
    NULL
  else
    x$ListFields()[[1]][[1]]$name
}

is_features <- function(x) {
  inherits(x, "tensorflow.core.example.feature_pb2.Features")
}

is_feature_lists <- function(x) {
  inherits(x, "tensorflow.core.example.feature_pb2.FeatureLists")
}

# ----- helpers


is_dict <- function(x) inherits(x, "python.builtin.dict")

as_dict <- function(..., .convert = FALSE) UseMethod("as_dict")

as_dict.python.builtin.dict <- function(..., convert = FALSE) {
  stopifnot(missing(..2))
  ..1
}

all_unique <- function(x) {
  identical(length(unique(x)), length(x))
}

as_dict.list <- function(..., .convert = FALSE) {
  if (missing(..2)) {
    l <- ..1
    nl <- names(l)
    if(!all_unique(nl))
      stop("All keys in a dictionary must be unique")
    reticulate::py_dict(nl, unname(l), convert = .convert)
  } else
    dict2(..., .convert = .convert)
}

as_dict.default <- function(..., .convert = FALSE)
  dict2(..., .convert = .convert)



import::from(rlang, is_formula, f_lhs, f_rhs, names2)

dict2 <- function(..., .convert = FALSE) {
  dots <- list(...)
  calling_env <- parent.frame()

  keys <- as.list(names2(dots))
  values <- unname(dots)

  for (i in seq_along(dots)) {
    d <- dots[[i]]
    if (is_formula(d)) {
      warn_if_named_that_name_is_ignored(dots[i])
      keys[[i]] <- eval(f_lhs(d), calling_env)
      values[[i]] <- eval(f_rhs(d), calling_env)
    }
  }

  if(!all_unique(keys))
    stop("All keys in a dictionary must be unique")

  reticulate::py_dict(keys, values, convert = .convert)
}

warn_if_named_that_name_is_ignored <- function(x) {
  stopifnot(is_scalar(x))
  nx <- names(x)
  if(is.null(nx) || is.na(nx) || nx == "")
    return(invisible(FALSE))

  warning("Supplied name ", sQuote(nx), " is ignored", call. = FALSE)
  invisible(TRUE)
}

is_scalar <- function(x) identical(length(x), 1L)

#------- end helpers
# --- example usage
sig <- rnorm(10)

float_feature(sig[1:2])
feature_list(float_feature(sig[1:2]))
feature_list(float_feature(sig[1:2]),
             float_feature(sig[5:8]))

features(
  feat_a = float_feature(1),
  feat_b = bytes_feature("b")
)

feature_lists(
  a = feature_list(float_feature(sig[1:2])),
  b = feature_list(float_feature(sig[1:2])))


# literal translateion of
#  https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/kernel_tests/parsing_ops_test.py#L1030
value <- sequence_example(
  context = features(dict2(
    global_feature = float_feature(c( 1, 2, 3 )),
    global_feature2 = float_feature(c( 1, 2, 3 ))
  )),
  feature_lists = feature_lists(dict2(
    repeated_feature_2_frames =
      feature_list(list(bytes_feature(c("a", "b", "c")),
                        bytes_feature(c("a", "d", "e")))),
    repeated_feature_3_frames =
      feature_list(list(int64_feature(c(3, 4, 5, 6, 7)),
                        int64_feature(c(-1, 0, 0, 0, 0)),
                        int64_feature(c(1, 2, 3, 4, 5))))
  ))
)

value

# R-friendly translation
value <- sequence_example(
  context = features(
    global_feature = float_feature(1, 2, 3),
    global_feature2 = float_feature(1, 2, 3)
  ),
  feature_lists = feature_lists(
    repeated_feature_2_frames = feature_list(
      bytes_feature("a", "b", "c"),
      bytes_feature("a", "d", "e")),
    repeated_feature_3_frames = feature_list(
      int64_feature(3, 4, 5, 6, 7),
      int64_feature(-1, 0, 0, 0, 0),
      int64_feature(1, 2, 3, 4, 5))
    )
)

value

[kevinykuo]

@t-kalinowski yeah it's pretty rough... I'm putting together something that will try to make this a whole lot more intuitive for R users. Here's another helpful resource with some funny rants :wink: https://www.youtube.com/watch?v=oxrcZ9uUblI

[t-kalinowski]

a generic as_feature() works pretty well to simplify things.

as_feature <- function(...) UseMethod("as_feature")

as_feature.tensorflow.core.example.feature_pb2.Feature <- function(...) {
  if(!missing(..2))
    stop("Can't coerces multiple features into one feature")
  ..1
}

as_feature.numeric <- function(...) {
  x <- c(...)
  if(is_integerish(x))
    int64_feature(x)
  else
    float_feature(x)
}

as_feature.character <- bytes_feature
as_feature.raw <- bytes_feature
as_feature.default <- function(...)
  stop("as_feature must be a numeric, character, or raw")

enables usage like:

value <- sequence_example(
  context = features(
    global_feature = c(1, 2, 3),
    global_feature2 = c(1, 2, 3)
  ),
  feature_lists = feature_lists(
    repeated_feature_2_frames = feature_list(
      c("a", "b", "c"),
      c("a", "d", "e")),
    repeated_feature_3_frames = feature_list(
      c(3, 4, 5, 6, 7),
      c(-1, 0, 0, 0, 0),
      c(1, 2, 3, 4, 5))
  )
)

[kevinykuo]

I'm thinking of something like this, where users can pass in a familiar columnar list/tibble. We do schema inference in the background, and return a default parsing function they can pass to dataset_map(). We can expose arguments for passing "advanced" configs if necessary. Idea is to make it easy to port most existing workflows. Thoughts?

df <- tribble(
  ~age, ~movie, ~movie_ratings, ~suggestion,
  29, c("The Shawshank Redemption", "Fight Club"), c(9.0, 9.7), "Inception",
  18, "Titanic", 8.2, "Twilight"
)

my_parse_fun <- tfdatasets:::write_tfrecord(df, temp_file)
dataset <- tfrecord_dataset(temp_file) %>%
  dataset_map(my_parse_fun)
iter <- dataset$make_one_shot_iterator()
sess <- tf$Session()
# > sess$run(iter$get_next())
# $movie
# SparseTensorValue(indices=array([[0],
#                                  [1]]), values=array(['The Shawshank Redemption', 'Fight Club'], dtype=object), dense_shape=array([2]))
#
# $movie_ratings
# SparseTensorValue(indices=array([[0],
#                                  [1]]), values=array([9. , 9.7], dtype=float32), dense_shape=array([2]))
#
# $age
# [1] 29
#
# $suggestion
# [1] "Inception"
#
# > sess$run(iter$get_next())
# $movie
# SparseTensorValue(indices=array([[0]]), values=array(['Titanic'], dtype=object), dense_shape=array([1]))
#
# $movie_ratings
# SparseTensorValue(indices=array([[0]]), values=array([8.2], dtype=float32), dense_shape=array([1]))
#
# $age
# [1] 18
#
# $suggestion
# [1] "Twilight"

[jjallaire]

I agree with the idea of passing an example list/tibble for schema inference. That will be very nice for users. I don't think that the write function should return the parsing function though (as these operations may be done at different times). Rather, I think there should be a higher level parse function that takes the list/tibble spec as it's input.

[dfalbel]

I managed to make tfrecords support arrays.
Here is a small working example: https://github.com/dfalbel/tfrecords/blob/master/vignettes/mnist.Rmd

The idea is to be able to write lists of arbitrary data structures that have the same number of observations. It should work for arrays, matrices and sparse matrices ("dgMatrix"). Since we write directly from C++, the package should be faster then writing using tensorflow API.

Installation is not too simple yet since we need protobuff > 3.0 and we can't just apt-get this on ubuntu. But brew installs an updated version.

[kevinykuo]

@dfalbel Cool! Will take a look. Do we know how much performance gain is achieved vs. talking to Python? Also could we leverage RProtoBuf?

[dfalbel]

at my first benchmark it was 1500x faster then python. Here my benchmark code: https://github.com/dfalbel/tfrecords/blob/master/benchmark/benchmark.R

to test against RProtoBuf we would need to write the tfrecord serialization algorithm that's not too simple... but I think the problem is having to loop over the entire dataset in R - that we won't avoid by using RProtoBuf

[kevinykuo]

Yeah 1500x is indeed pretty bad and should justify having some external dependencies if there's no better way. There's quite a bit of overhead talking to Python it seems. brew install protobuf wasn't bad, but could we ship the necessary prereq binaries with the package so (windows?) users won't have to fudge with installation/environment variables?

[dfalbel]

Yes, I think we can use the same script they use in RProtoBuf: https://github.com/eddelbuettel/rprotobuf/blob/master/tools/winlibs.R
Not sure if we could do the same for ubuntu.

[kevinykuo]

@dfalbel I'm convinced the C++ approach is the way to go. Let me know if you were planning on supporting lists and sequence examples. I can also take a stab.

@jjallaire do you think this functionality should live in tfdatasets? Also, I'm thinking we can provide wrappers for the tf_ functions (and reexport in tfdatasets) commonly used to write parsing functions so users won't have to do tf-dollar-sign stuff.

[jjallaire]

I think it should either live in tfdatasets or in another tfrecord package.

Yes, we are definitely going to want wrappers for the tf_ functions just like we've made wrappers for the k_ functions in Keras which use R-native indexing, handle type conversions, etc.

Data preprocessing needs to be our main focus in TF-land for 2018. Ideally we can take a higher level look at the types of data people preprocess (images, text, vector-data, etc.) and write high level grammars in R for specifying pre-processing behavior. We also need to be aware of the relationship between this work and tfdeploy, as we will also ideally include the pre-processing code within the graph of deployed models.

This work will be a huge challenge and has an equally huge design space so we will need to do lots of iteration and review before we come up with the right answers. I'm therefore inclined to keep the current work in other packages (e.g. a tfrecord package) so that we can easily prototype and then discard designs as our understanding improves.

cc @javierluraschi

[dfalbel]

@kevinykuo Yes, I am planing on supporting lists and sequence examples, but I'm still not happy with my c++ code. Specifically, with this kind of implicit cast. Since it's done once per observation I think this making a lot of overhead. Before supporting other things I would like to solve this.

[dfalbel]

@kevinykuo I made some changes, and I think the code is much better now. Also added more docs for installation. It's also much faster now for sparse matrices.

Can you take a look? I would like to submit to CRAN this week.

I think before supporting lists and sequence examples we could implement the API for obtaining the parsing function to use with tfdatasets. Let me know what you think.

[jjallaire]

Awesome that you have made so much progress here! Agreed that the tfdatasets parsing function is a critical component as well.

In terms of submitting to CRAN I'd have these words of caution: I think this area (high performance ingest and preprocessing) is the most important thing for us to work on in 2018. I would like to put a lot of time/thinking/effort into this starting in a few weeks (@javierluraschi is in the same place). However, at least Jaiver and I don't have time right now for detailed review/iteration/contribution (@kevinykuo is slightly better off on this front). My guess is that once we all get working on this together that there will be changes, so I'm not sure you should commit to the level of interface stability required by a CRAN release.

So I'd say if you can keep it off CRAN for the time being that's better. If you have other reasons you really want to get to CRAN then let's discuss the name of the package and make sure we don't "use up" a name we want to use later for another iteration of the same concepts that are interface-incompatible.

[dfalbel]

Great! Sure, I can wait to submit to CRAN and agree that it's a good decision.

[jjallaire]

Okay, let's circle back on this again soon.

On Thu, May 3, 2018 at 8:29 AM, Daniel Falbel [e-mail]
wrote:

Great! Sure, I can wait to submit to CRAN and agree that it's a good
decision.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rstudio/keras/issues/355#issuecomment-386278889, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAGXx9WhQnPQvWSbXTyGPqeZLX5M_TvHks5tuvgZgaJpZM4TNKEb
.

[t-kalinowski]

@kevinykuo, have you had a chance to take a look the use-case described here and think about how it might be improved through the use of tfrecords? I'd like to pick this up again next week and am eager to see how you might approach this.

[kevinykuo]

@t-kalinowski I think you'll want to write the data to TFRecord format and pass it to keras via tfdatasets as JJ suggested. The more performant way would be to implement SequenceExample support in @dfalbel's C++ code, and the less performant way would be to do it via Python, which is more or less what you pieced together in https://github.com/rstudio/keras/issues/355#issuecomment-382080210. I also have some (R/Python) code with a bit more sugar that should suffice for smaller datasets; I'll polish it up a bit and share early next week. We can then start iterating on the API design.

[kevinykuo]

@t-kalinowski see this gist for an approach to the toy problem. It utilizes an obscenely slow implementation of write_tfrecord() here.

For this iteration, I've proposed some (somewhat opinionated) design choices that should be apparent by looking at the format of the input data. It'd be great if y'all can take a look and and share thoughts/potential alternatives. We should be OK focusing on the user-facing interface for now and worrying about the plumbing afterwards.

[t-kalinowski]

Thanks @kevinykuo, I'm still parsing this and i'll post a longer reply later. One initial bug I came up was that a sequenceExample dataset does not print properly. There error message is

Error in shape$as_list() : attempt to apply non-function

It can be traced to https://github.com/rstudio/tfdatasets/blob/711f322b7c9d1c3217f011b6ed3a090f48d411fa/R/dataset_methods.R#L534 , where it looks like for a sequence example, you need to index down one additional level in the list before you get to the shape tuple.

[t-kalinowski]

@kevinykuo, it's been pretty busy here and it'll be a few days before I can finish digesting your example; I'll post something more meaningful then.

In the meantime, you should checkout listarrays::split_along_rows(), it would make your data-munging life easier for the task of getting an R array into a dataframe

suppressPackageStartupMessages(library(tidyverse))       
library(listarrays)                                      

arr <- function(...) array(seq_len(prod(c(...))), c(...))
X <- arr(10, 3, 3, 3)                                    
Y_1 <- arr(10, 10)                                       
Y_2 <- arr(10)                                           

df <- lst(X, Y_1, Y_2) %>%                                     
   split_along_rows() %>%                                   
   as_tibble() %>%                                          
   mutate_if(~all(lengths(.x) == 1), simplify) 

df             
#> # A tibble: 10 x 3
#>    X                 Y_1          Y_2
#>    <list>            <list>     <int>
#>  1 <int [3 × 3 × 3]> <int [10]>     1
#>  2 <int [3 × 3 × 3]> <int [10]>     2
#>  3 <int [3 × 3 × 3]> <int [10]>     3
#>  4 <int [3 × 3 × 3]> <int [10]>     4
#>  5 <int [3 × 3 × 3]> <int [10]>     5
#>  6 <int [3 × 3 × 3]> <int [10]>     6
#>  7 <int [3 × 3 × 3]> <int [10]>     7
#>  8 <int [3 × 3 × 3]> <int [10]>     8
#>  9 <int [3 × 3 × 3]> <int [10]>     9
#> 10 <int [3 × 3 × 3]> <int [10]>    10

And the bind_as_rows() for the inverse

map_if(df, is.list, bind_as_rows)

[t-kalinowski]

@jjallaire, related to the original topic of this thread (before the thread turned to tfrecords):

Would it be possible to use https://pypi.org/project/prefetch_generator/ or something similar to get a speed-up for "free"? Essentially, queue up results from the R generators on a background thread so that when keras calls for the next mini-batch it's ready with no computation.

I'm know there are many dragons once move away from single-threaded execution, but if certain constraints can be reliably enforced (e.g, can't access any global variables), implementation might be mean just wrapping the supplied R functions with the@background decorator on the python side. (or do it all from R side with the processx package or something similar, though with that approach I don't know of an R equivalent to pythons Queue.Queue)

[jjallaire]

No, R code cannot be run outside of the thread where the R interpreter was initialized.

We could possibly do something with farming out generation to child processes, but I wonder if the serialization overhead that entails would wipe out any gains created by parallelism.

Our longer term goal is to allow you to write R pre-processing code that executes on the TF graph (which in turn is all a multi-threaded C++ runtime).