Keras: Cross validation for keras in R

[gundalav]

Python version of Keras allow interoperability with sklearn cross validation functions.
So that we can make the code like this for StratifiedKfold for example:

from sklearn.cross_validation import StratifiedKFold
from keras.models import Sequential
from keras.layers import Dense

def load_data():
    # load your data using this function

def create model():
    # create your model using this function
    model = Sequential()
    model.add(Dense(units=64, activation='relu', input_dim=100))
    model.add(Dense(units=10, activation='softmax'))
    model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])

    return model


def train_and_evaluate__model(model, data[train], labels[train], data[test], labels[test)):
    model.fit...
    # fit and evaluate here.

if __name__ == "__main__":
    n_folds = 10
    data, labels, header_info = load_data()
    skf = StratifiedKFold(labels, n_folds=n_folds, shuffle=True)

    for i, (train, test) in enumerate(skf):
            print "Running Fold", i+1, "/", n_folds
            model = None # Clearing the NN.
            model = create_model()
            train_and_evaluate_model(model, data[train], labels[train], data[test], labels[test))

What's the RStudio Keras pattern for the above? Does it has interop with any of other R package?

[jjallaire]

There is an example of this here: https://jjallaire.github.io/deep-learning-with-r-notebooks/notebooks/3.7-predicting-house-prices.nb.html

[Mrugankakarte]

@jjallaire Please let me know if I am wrong. Your example shows normal cross-validation right? It does not do stratified cross validation in case of class imbalance?
I did stratified cross validation using caret package createFolds function as below:

folds <- createFolds(y = raw_data[,target], k = 5, list = F)
raw_data$folds <- folds

  for(f in unique(raw_data$folds)){

        cat("\n Fold: ", f)
        ind <- which(raw_data$folds == f) 
        train_df <- data[-ind,]
        y_train <- as.matrix(raw_data[-ind, target])
        valid_df <- as.matrix(data[ind,])
        y_valid <- as.matrix(raw_data[ind, target])

        model_1 <- model %>% fit(
              x = as.matrix(train_df), y = y_train,
              batch_size = 256,
              epochs = 2,validation_data = list(valid_df, y_valid))
              # callbacks = list(callback_tensorboard(log_dir = "logs/cudnngru7"), 
              #                  callback_early_stopping(monitor = 'val_loss')))


        y <- predict(model ,tdata)
        write.csv(y, paste0("saves/bidirectional cudnngru/cv/",target,"_fold_",f,".csv"), row.names = F)

  }`

Please let me know your thoughts on this.

[jjallaire]

Yes, using createFolds is definitely a better way to go.

cc @topepo