Keras: Dropout during testing

[tbonne]

I'm having trouble trying to get dropout working during the prediction stage (i.e., trying to get an estimate of uncertainty around my predictions).

There seems to be some options for this in Keras through the 'training=True' option when adding a dropout layer: https://github.com/keras-team/keras/issues/9412#issuecomment-366487249. Though I haven't been able to get it to work with the keras package in R.

Is there a way to force dropouts when running 'predict'?

[jjallaire]

Yes, you can do this. Layers are essentially functions so if we use the Keras functional API we can call them with arbitrary parameters (like the training=TRUE one you need here). For example:

dropout_1 <- layer_dropout(rate = 0.4) 
dropout_2 <- layer_dropout(rate = 0.3)

input <- layer_input(shape = c(784))
output <- input %>% 
  layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>% 
  dropout_1(training = TRUE) %>% 
  layer_dense(units = 128, activation = 'relu') %>%
  dropout_2(training = TRUE) %>%
  layer_dense(units = 10, activation = 'softmax')

model <- keras_model(input, output)

[tbonne]

Thanks for clarifying. This is truly a great package.

Below is a small regression example in case it can be of use to anyone else:

```{r}
library(keras)
library(rsample)

SIMULATED data

x1 <- rnorm(1000,0,0.5)
x2 <- rnorm(1000,2,1)
y <- x1*x2 + x1^2
dat <- data.frame(y,x1,x2)
dat

get data into training and testing datasets

retrain_test_split <- initial_split(dat, prop = 0.8)
train_tbl <- training(retrain_test_split)
test_tbl <- testing(retrain_test_split)

get inputs and outputs for the networks

x_train_tbl <- as.matrix(train_tbl %>% dplyr::select(-y))
y_train_tbl <- as.matrix(train_tbl %>% dplyr::select(y))
x_test_tbl <- as.matrix(test_tbl %>% dplyr::select(-y))
y_test_tbl <- as.matrix(test_tbl %>% dplyr::select(y))

MODEL

layers

dropout_1 <- layer_dropout(rate = 0.4)
dropout_2 <- layer_dropout(rate = 0.3)

input <- layer_input(shape = c(2))

output <- input %>%
layer_dense(units = 32, activation = 'relu', input_shape=c(2)) %>%
dropout_1(training = TRUE) %>%
layer_dense(units = 32, activation = 'relu') %>%
dropout_2(training = TRUE) %>%
layer_dense(units = 1)

model1 <- keras_model(input,output)

Compile model

model1 %>% compile(
optimizer = 'rmsprop',
loss = 'mse'
)

FIT

history <- model1 %>% fit(
x = x_train_tbl,
y = y_train_tbl,
epochs=500,
batch_size=128,
validation_data = list(x_test_tbl,y_test_tbl )
)

plot(history)

EVALUATE

dist_scores <- vector()
for(i in 1:100){
score <- model1 %>% evaluate(
x = x_test_tbl,
y = y_test_tbl,
batch_size = 128)

dist_scores[length(dist_scores)+1]<-sqrt(score) #RMSE
}
hist(dist_scores, xlab = "Model RMSE")

PREDICT - predictions around one point as an example

pred.dat <- vector()
for(i in 1:100){
pred<-predict(model1,as.matrix(x_test_tbl))
pred.dat[length(pred.dat)+1] <- pred[1,]
}

hist(pred.dat) #predicted points
abline(v= as.matrix(y_test_tbl) [1], col="red") #actual point
```

[EhtashamBillah]

Hi,
Thanks for your comment. But how to obtain the distribution over all observations of y? Would really appreciate your help.

[EhtashamBillah]

Hi,
I have not been able to find documentation regarding applying Monte Carlo Dropout during test time, as suggested by Yarin Gal. However, I have tried to do so in R. I need your suggestions if I am doing it wrong.

As you can see, I am applying dropout with the ARGUMENT "training=TRUE".
I am doing the data augmentation and creating the train data with train generator function.I have 230 observations in my training data

I fit the model with dropout being active.I use fit generator function to fit the model

Finally, I want to predict the probability of y. I have 30 observations in test data and I want 100 probability value for each observation.I apply Monte Carlo dropout with predict generator function.

But I am not sure about the prediction part.I am not sure whether data is being passed 200 times through test data.Your suggestions will help a lot.

[aplamhden22]

for regression
@EhtashamBillah In order to get all distribution of y for lets say X_test[23] , just use
for i=1 to 500
y_pred <- model.predict(X_test[23])
y_pred_samples[i]<-y_pred

Now y_pred_samples [array] is the samples of the posterior distribution for X_test[23]. Use mean(y_pred_samples) for evaluation and get uncertainty of the model's prediction with std(y_pred_samples). If you want something quick, evaluate all X_test set with the mean. Otherwise, figure out how to use the uncertainty in the evaluation process.

sorry for the pseudo code in the for the loop