Gpytorch: What GP module to use in reinforcement learning settings

[Akella17]

A little background: I am trying to implement a Bayesian RL algorithm for continuous environments. For this, I plan on replacing the last layer of the Critic network with a GP layer with the hope that this makes the algorithm significantly more sample-efficient.

For training, the network samples a fresh set of datapoints for each update. As I cannot get my hands on the training data beforehand, I don't think Exact inference is possible. For variational inference, what do I provide the num_data parameter as? Should I provide num_data as the size of the freshly sampled batch (as the previous samples are discarded) or the total number of datapoints (sample_size x num_samples)? I am asking this because varying this parameter considerably affects the performance of the network.

[jacobrgardner]

@Akella17 If I understand the question correctly, each time you obtain new data, you should probably recreate the VariationalELBO with the updated num_data. num_data is always the total amount of training data available. Another way to think of it is: in a single epoch of training, num_data is the total number of datapoints that the model will see.

[Akella17]

Yes exactly. And when you say recreate the VariationalELBO, should I be creating a new object each time and copying the learned mean and covariances or is it okay to reuse the previous object. In other words, would a single VariationalELBO object be trained on a newer batch of data on each time work?

When the num_data can be either sample_size or total_size, why not something in between (I understand it would not make sense theoretically)? The way I see it, the num_data parameter affects the performance which makes it a hyperparameter that needs to be tuned accordingly.

[jacobrgardner]

You could use the same object as long as you updated num_data appropriately.

I would not personally recommend viewing num_data as a hyperparameter to be tuned. The issue is that it's effectively controlling the normalization of the ELBO, which has a specific statistical interpretation.

If you modify that normalization, you may get better performance in the sense that you'll effectively be weighting the "model fit" term more or less depending. It would be hard to justify the change, however -- it's kind of like saying you get better performance by making this probability distribution sum to 3 instead of 1. At the end of the day, it's your model and your choice though 😄!

[Akella17]

Yeah, got it. Can we use exact inference in my RL settings (i.e. without providing the ExactGP object with the training data a priori)?

[jacobrgardner]

For exact GPs, you can pass None and None as the train inputs and train targets and then the GP you get in eval mode will be the prior. You could then use set_train_data(new_train_x, new_train_y, strict=False) to add data (although obviously the hyperparameters won't automagically update, you'll have to do some training each time you do this likely.

Or just create a new ExactGP object every time you need to add data. It's pretty lightweight to do, and you could just do new_model.load_state_dict(old_model.state_dict()) to copy the parameters in. Either way.

[Akella17]

When using gpytorch.mlls.VariationalELBO, can I get the output as a batch_size sized vector instead of a scalar overall loss over the input batch of data.

Also, does GPyTorch have any predefined mll for finding the loss between the output and target gpytorch.distributions.MultivariateNormal objects instead of an output gpytorch.distributions.MultivariateNormal object and a tensor of target values.