Keras: RNN - issue with input dimension

[fderyckel]

Hello,
I have tried to build a LSTM RNN in R following a tutorial given in Python with this dataset.
I managed to go through whole the steps, but I get stuck with the training part.

Here is my code:

library(tidyverse)
library(lubridate)
df <- read_csv("~/Keras/Bejing_Pollution.csv")
df2 <- df %>% 
  mutate(time = ymd_h(paste0(df$year, "-", df$month, "-", df$day, "-", df$hour))) %>% 
  select(-No, -year, -month, -day, - hour)

# Renaming variables
colnames(df2) <- c("pollution", "dew", "temperature", "pressure", "wind_dir", 
                   "wind_speed", "snow", "rain", "Date")

# Replacing all the NAs by 0
df2 <- df2 %>% replace_na(list(pollution = 0, temperature = 0, pressure = 0))

# Drop the first 24 rows as they are no output variables 
df2 <- df2[25:43824, ]
df3 <- df2
# We put into number the wind direction
df3$wind_dir <-  as.factor(df3$wind_dir) %>% as.numeric()

# We scale our df and shift our response variable by unit up  
df3 <- df3 %>% select(-Date) %>% 
  map_df(function(.x) as.numeric(.x)) %>% 
  map_df(function(.x) (.x - min(.x)) / (max(.x) - min(.x))) %>% 
  mutate(y_tplus1 = lead(pollution)) %>% 
  as_tibble()

# Splitting data and converting to matrix
train_input <- df3[1:32000, 1:8] %>% as.matrix()
train_output <- df3[1:32000, 9] %>% as.matrix()

So my input training matrix has 8 variables, and my output training matrix has 1 variable (the value to be predicted)

library(keras)
model_pollution <- keras_model_sequential()
model_pollution %>% 
  layer_lstm(units = 50, input_shape = c(1, NULL,7)) %>% 
  layer_dense(units = 1)

model_pollution %>% compile(loss = "mae", optimizer = "adam")

And I am getting this error:

Error in py_call_impl(callable, dots$args, dots$keywords) : ValueError: Error when checking input: expected lstm_6_input to have 3 dimensions, but got array with shape (32000, 8)

I know that in python they use the .reshape() function to add a third dimension into the array. reshape input to be 3D [samples, timesteps, features]
Now, how can we do this in R?

Thanks for any help and/or suggestions.

PS: I am aware that scaling on the whole dataset is a big NO NO as it introduce future data into the present data (data snooping biais, I will eventually change that )

[jjallaire]

In R you can use the array_reshape function

[jjallaire]

Also, in R the “c” function will remove the NULL. To preserve the NULL you need to use the “list” function to preserve the NULL

[fderyckel]

@jjallaire Thank you so much for such a quick reply.
So after a few trials, it seems like it would work this way:

train_input <- df3[1:32000, 1:8] %>% as.matrix()
train_input2 <- array_reshape(x = train_input, dim = list(32000, 1, 8))

test_input <- df3[32001:43800, 1:8] %>% as.matrix()
test_input2 <- array_reshape(x = test_input, dim = list(11800, 1, 8))

Let's see how the rest is working.

[fveronesi]

Hi @fderyckel
I am trying the same exercise, but once I start the model I get a NaN in the loss function computations.
Were you able to make it work?