Keras: Slow Keras load: 20-40 seconds per model load using load_model_hdf5()

[leonjessen]

I'm facing issues with load_model_hdf5() load times. I have an ensemble of 20 models. The meta_data object holds file paths for trained models. I then want to attach the models, such that I may perform predictions like so predict(meta_data$model_01$model, X). Each model has 181,601 parameters and each hdf5 file is ~2.2MB. I'm using a new macbook pro and R-3.4.4 and keras_2.1.5.9001. Load times are as follows (weirdly there is seem to be an increasing load time with number of models loaded):

> meta_data = attach_models(meta_data)
model_01 loaded in 22.22 secs
model_02 loaded in 22.68 secs
model_03 loaded in 23.62 secs
model_04 loaded in 26.28 secs
model_05 loaded in 25.81 secs
model_06 loaded in 26.52 secs
model_07 loaded in 26.48 secs
model_08 loaded in 29.14 secs
model_09 loaded in 29.23 secs
model_10 loaded in 30.45 secs
model_11 loaded in 29.52 secs
model_12 loaded in 29.81 secs
model_13 loaded in 30.34 secs
model_14 loaded in 31.40 secs
model_15 loaded in 33.02 secs
model_16 loaded in 32.82 secs
model_17 loaded in 35.96 secs
model_18 loaded in 34.03 secs
model_19 loaded in 34.59 secs
model_20 loaded in 38.06 secs

The attach_models() function simply looks like so:

attach_models = function(meta_data){
  now = Sys.time()
  for( model_id in names(meta_data) ){
    model_file = meta_data[[model_id]]$model_file
    meta_data[[model_id]]$model = load_model_hdf5(filepath = model_file)
    cat(model_id, " loaded in ", round(Sys.time() - now, 2), " secs\n", sep = "")
    now = Sys.time()
  }
  return(meta_data)
}

and then as stated, I can do

predict(meta_data$model_01$model, X)

But, this means that currently loading the ensemble model will be ~10 minutes, before predictions can be made. I have tried using Hadley's save_rds() and read_rds() and then saving the models in a meta object after training, but that only stores reference to the model and not the model itself.

Is it really that slow or is something wrong?

[dfalbel]

I think your code is printing the accumulated loading time, not the loading time for each model. The first model loaded usually takes longer because it will also load Keras/TEnsorflow etc, but the other are quite fast.

[skeydan]

Could it be that at the time you're loading the models, your R session is already using a lot of memory, compared to RAM overall?
(E.g., you could have loaded images before, in case you wanted to classify them, etc...)

You could use lobstr::mem_used() to find how much is used.

[leonjessen]

Okay, so restarted the laptop and deleted the .RData file and these are the load times on a 'fresh' session:

model_01 loaded in 2.42 secs
model_02 loaded in 2.6 secs
model_03 loaded in 2.9 secs
model_04 loaded in 3.21 secs
model_05 loaded in 3.56 secs
model_06 loaded in 3.74 secs
model_07 loaded in 4.21 secs
model_08 loaded in 4.38 secs
model_09 loaded in 4.89 secs
model_10 loaded in 4.93 secs
model_11 loaded in 5.78 secs
model_12 loaded in 6.51 secs
model_13 loaded in 6.85 secs
model_14 loaded in 7.26 secs
model_15 loaded in 6.86 secs
model_16 loaded in 7.4 secs
model_17 loaded in 7.62 secs
model_18 loaded in 8.36 secs
model_19 loaded in 8.25 secs
model_20 loaded in 8.84 secs

Better load times, but still slow and with increasing load time. I can see that the list with attached models, does not take up much space like so:

library(pryr)
object_size(meta_data)
126 kB

It should be a lot faster right?

[leonjessen]

Ok, so the solution seem to be to include a k_clear_session(), before loading each model in my attach_models() function

attach_models = function(meta_data){
  now = Sys.time()
  for( model_id in names(meta_data) ){
    k_clear_session()
    model_file = meta_data[[model_id]]$model_file
    meta_data[[model_id]]$model = load_model_hdf5(filepath = model_file)
    cat(model_id, " loaded in ", round(Sys.time() - now, 2), " secs\n", sep = "")
    now = Sys.time()
  }
  return(meta_data)
}

Which results in load times like so:

model_01 loaded in 3.87 secs
model_02 loaded in 1.78 secs
model_03 loaded in 1.82 secs
model_04 loaded in 1.82 secs
model_05 loaded in 1.82 secs
model_06 loaded in 2.7 secs
model_07 loaded in 1.86 secs
model_08 loaded in 1.82 secs
model_09 loaded in 1.82 secs
model_10 loaded in 1.8 secs
model_11 loaded in 1.8 secs
model_12 loaded in 1.82 secs
model_13 loaded in 3.13 secs
model_14 loaded in 1.81 secs
model_15 loaded in 1.81 secs
model_16 loaded in 1.98 secs
model_17 loaded in 1.84 secs
model_18 loaded in 1.82 secs
model_19 loaded in 1.83 secs
model_20 loaded in 1.85 secs

I.e. roughly 2 seconds for each model