Gpytorch: White noise kernel?

[stanbiryukov]

Hi there,

Thanks for the excellent library. Is it possible to either add a white noise kernel or have an option to include input variance to the diagonal of the kernel matrix during fitting? If I have known variance on my input data, it's not clear to me how I can incorporate that in the model examples.

[Balandat]

@darbour is working on implementing that. Pull request should be incoming soon.

[Balandat]

Ok, so I was trying to get this to work, but I'm having a hard time getting the correct behavior in all the different contexts (training, test, eval, forward, lazy, non-lazy, batch, non-batch, in usage with AdditiveKernel, etc.). @gpleiss, @jacobrgardner, any thoughts?

The common use case is to define a new AdditiveKernel, e.g. RBFKernel() + HeteroscedasticWhiteKernel(Y_train_target_var)

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

import torch

from gpytorch.kernels import Kernel
from gpytorch.lazy.diag_lazy_variable import DiagLazyVariable


class HeteroscedasticWhiteKernel(Kernel):

    def __init__(self, variances):
        super(HeteroscedasticWhiteKernel, self).__init__()
        self._variances = variances

    def forward(self, x1, x2, **params):
        if (
            x1.shape == x2.shape
            and x1.shape[0] == self._variances.shape[0]
            and torch.equal(x1, x2)
        ):
            return DiagLazyVariable(self._variances)
        else:
            return x1.new_zeros(x1.shape[0], x2.shape[0])

    def __call__(self, x1_, x2_=None, **params):
        if x2_ is None:
            x2_ = x1_
        return self.forward(x1_, x2_, **params)

[jacobrgardner]

@balandat I think I see two problems you are running in to:

1. Additive kernel needs to check if one output is Lazy and the other isn't, and wrap the non lazy thing in a NonLazyVariable. I have a fix for this up on a white_noise_kernel branch.

2. At test time, the way we get all the matrices we need is to concatenate the train and test data and call the kernel once (see exact_gp.py:106). Thus it is attempting to add a length n diagional to a (n+t) x (n+t) matrix, so needs to be padded with zeros.

Upon reflection, point 2 should change in my opinion for exact GPs. Particularly for BayesOpt getting the t x t component is wasteful. For Sparse GPs there's not really an issue because everything involved is Lazy. @gpleiss and I are discussing this now.

[Balandat]

Hmm so while #141 allows to subset the kernel matrix without fully computing it, I'm still not sure how to implement the logic for the heteroscedastic white kernel in a clean & efficient way.

With all the laziness and efficient transposing for diag computations there is a lot going on. It's also not super straightforward to debug if the error you're making somewhere doesn't surface until way later in the code b/c of the lazy evaluation :(

Anyway, here's is a very dumb indefficient barebones idea for the heteroschedastic white Kernel - this trains and evaluates, but will fail when evaluating .var() b/c @jacobrgardner was trying to be smart in #144 😄 (.covar().evaluate() works).

class HeteroscedasticWhiteKernel(Kernel):
    def __init__(self, variances):
        super(HeteroscedasticWhiteKernel, self).__init__()
        self.register_buffer("variances", variances)
        self._train_kern = torch.diag(self.variances).unsqueeze(0)

    def forward(self, x1, x2):
        if self.training:
            return self._train_kern.repeat(x1.shape[-3], 1, 1)
        else:
            out = x1.new_zeros(x1.shape[-3], x1.shape[-2], x2.shape[-2])
            ntrain = self._train_kern.shape[-1]
            out[:, : ntrain, : ntrain] = self._train_kern.repeat(out.shape[0], 1, 1)
            return out

I guess there are ways to be smarter here, like returning a DiagLazyVariable in training mode, or add some sort of block lazy variable (which does not necessarily require all elements to be square...).

Any thoughts are welcome.

[jacobrgardner]

This should be implemented, per #152.

Feel free to reopen if any issues arise!

[Balandat]

Awesome, thanks @jacobrgardner for doing this the right way!