Gpytorch: Initialization of num_data parameter in mll.VariationalELBO

[Akella17]

In the Large-Scale Stochastic Variational GP Regression (CUDA) (w/ KISS-GP) notebook (https://gpytorch.readthedocs.io/en/latest/examples/05_Scalable_GP_Regression_Multidimensional/SVDKL_Regression_GridInterp_CUDA.html), shouldn't the num_data parameter be initialized with batch_size rather than train_y.size(0)?

mll = gpytorch.mlls.VariationalELBO(likelihood, model.gp_layer, num_data=train_y.size(0), combine_terms=False)

[gpleiss]

No - it should be train_y.size(0). The ELBO object needs to know the actual size of the dataset so it can scale terms appropriately to adjust for the minibatch size.

If n is the minibatch size and N is the total number of data, the stochastic ELBO is:

N/n expected_log_likelihood + KL_divergence.

The ELBO object gets n from the size of the data forwarded through the model, and gets N through the num_data argument

[Akella17]

In the case of reinforcement learning, the episode lengths are not constant. How can I use this model in RL framework? In other words, would providing a rough sample size have any detrimental effects on training?

[gpleiss]

What is "episode length" referring to in this context?

[Akella17]

Most reinforcement learning problems can be broken down into sequences in which an agent (the algorithm) interacts with its environment until it reaches a certain terminal state that initiates a reset to its initial state. "episode length" is the length of a sequence from its initial state to terminal state. This length (the duration/timesteps/actions elapsed by the agent in the environment) varies with episodes.

All I am trying to say is that the number of training points cannot be predetermined, i.e. we cannot know train_y.size(0) beforehand. I can set an upper bound to this value or iteratively calibrate it based on the mean episode length provided by the algorithm.

I wanted to know if a slightly inaccurate N could have detrimental effects on GP training or if it could be used as a mathematical hack?

[gpleiss]

The short answer is no

The long answer: I'm not sure I totally understand the setup. From the GP-based reinforcement l've done, here's the setup I used:

• Start out with no training data (num_data=0)
  For however long I want to train for:
  
  
  
  • Run a new trial/episode, where the model is conditioned on the data I've already collected. Gather new data
  
  
  • Create a new model/update the existing model with the new data (num_data = num_data + len_of_last_trial_epsiode). Train the model
  
  

Setting num_train to be the episode length (or approximate episode length) implies that your model is only being trained on a single episode, which is probably not what you want.

[Akella17]

The setup that I am using has the following routine:

1. 16 parallel processes that take 5 steps into the environment.
2. This data (batch_size = 5x16) is used to train the GP model for 4 epochs (32 randomized samples, mini_batch_size = 5x16//32).
3. Clear the data buffers and go to step 1 (The data collected previously in step 1 is discarded for collecting a fresher batch).

Should I be setting num_data = 80 and reuse the same GP model each time I sample a fresh batch?

[jacobrgardner]

@Akella17 If it helps, here's an explanation of the normalization that is happening:

The ELBO for stochastic optimization is something like num_data / num_batch E_{q(f)}[log p(y | f)] - KL[q(u)||p(u)]. We divide the ELBO by the total number of data points, leading to 1 / num_batch E_{q(f)}[log p(y | f)] - 1 / num_data KL[q(u)||p(u)].

num_data should be the total number of data points you would pass over in an epoch, where-as num_batch is the number of data points in a single minibatch.

[Akella17]

The data I am using to train the model keeps changing (this is because older data is being discarded). So the way I see it, I am training a model on num_data = 80 and then using this model as initialization for the next GP model which gets trained on the freshly collected 80 data points.

When I say num_data = 80, I am running 40 randomly sampled data points of batch_size = 2 for epochs = 4. Based on your previous comment, I think I am right at assuming num_data = 80.

[Akella17]

@gpleiss Could you share the code to the GP based RL experiment that you have worked on. It can be of great help in debugging the issues in my GP+PPO algorithm.

[gpleiss]

I'm sorry, I haven't worked on RL experiments.

Can you try changing the num_data property dynamically? It should just be a simple as mll.num_data = <new_num_data>

https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/mlls/variational_elbo.py#L20

[Akella17]

@gpleiss Setting the mll.num_data parameter is not fixing this problem. I am unable to isolate the issue as it only occurs in continuous control RL environments (the same network learns meaningful representations when trained in supervised learning settings). To illustrate this issue, I am sharing a sample code of DDPG algorithm implemented for Pendulum-v0.

• Actor_Critic.ipynb: Simple, working DDPG algorithm
• GP-AC.ipynb: Same as Actor_Critic.ipynb, except for the last linear layer of the critic network is replaced by GP regression

While Actor_Critic.ipynb learns a good policy in less than 100 updates, GP-AC.ipynb does not perform better than a random policy even after 600 updates.

https://drive.google.com/drive/folders/1ioE_T6MS-YVgLKNG8aLeKbgOERwUx2NT?usp=sharing

[cherepanovic]

Can you try changing the num_data property dynamically? It should just be a simple as mll.num_data =

is that the solution that works? I am going to try the SV-DKL model in an active learning scenario where the data also increase in further steps. Thanks!

[cherepanovic]

I think my question is answered by @jacobrgardner

If I understand the question correctly, each time you obtain new data, you should probably recreate the VariationalELBO with the updated num_data. num_data is always the total amount of training data available. Another way to think of it is: in a single epoch of training, num_data is the total number of datapoints that the model will see.

and

You could use the same object as long as you updated num_data appropriately.
I would not personally recommend viewing num_data as a hyperparameter to be tuned. The issue is that it's effectively controlling the normalization of the ELBO, which has a specific statistical interpretation.
If you modify that normalization, you may get better performance in the sense that you'll effectively be weighting the "model fit" term more or less depending. It would be hard to justify the change, however -- it's kind of like saying you get better performance by making this probability distribution sum to 3 instead of 1. At the end of the day, it's your model and your choice though 😄!

source: https://github.com/cornellius-gp/gpytorch/issues/595

süper! :)