Gpytorch: How to use a fixed noise Gaussian likelihood in a multi-task setting

[Galto2000]

Howdy folks,

GPyTorch provides Gaussian likelihood objects for fixed noise (FixedNoiseGaussianLikelihood) and for multi-task models (MultitaskGaussianLikelihood). I was wondering if someone could provide me some guidance on how to get a fixed noise multi-task Gaussian likelihood?

Thanks in advance

Galto

[jacobrgardner]

@Galto2000 I think we'd just need to implement FixedNoiseMultitaskGaussianLikelihood. Basically, you'd specify an n x t matrix of noises rather than a length n vector of noises, and the interface would otherwise be the same.

[Galto2000]

@jacobrgardner , would you please give a little more guidance, perhaps something like a high level recipe, on how I would go about implementing something like a FixedNoiseMultitaskGaussianLikelihood :) ?

Also, if you have some time, would you also please provide me with some clarification regarding my other issue (https://github.com/cornellius-gp/gpytorch/issues/890) - it's kind of related to this one.

I feel that the chips are starting to fall into place, but I just need an extra nudge from you and I think this would be a great challenge for me to wrap my head around some of the implementation details in GpyTorch.

Thanks in advance

Galto

[Balandat]

This is something that I'd like as well, let me see if I can find some time to work on this this week.

[Galto2000]

@Balandat, that would be great, thank you.

[Balandat]

@Galto2000 I'm assuming what you'd like to do here is provide the noise for the different tasks, but not the cross-task covariance - this should still be inferred. Is this correct?

[Galto2000]

@Balandat , yes, I think that is correct.

For instance, I am interested in doing multi-task, multi-sensor fusion; i.e. condition a model posterior on observations from different types of sensors (each of which has different noise) where the sensors output vector quantities, which makes it multi-task.

Thank you

Galto

[Balandat]

FWIW, I put up an early draft for this in 49e810b1a4f29fe1e0a102ad6f5963e90ae0dbdd - will have to do some cleaning up and testing before I make this a PR.

[Balandat]

Hmm, I'm realizing that the fact that MultitaskMultivariateNormal can be using either interleaved or non-interleaved representation significantly complicates things here. It'll take a little bit of work to iron this out.

[Balandat]

It seems that we should address #539 first in order to make this less of a pain to implement.

[Galto2000]

I'm trying to wrap my head around the interleaved concept as well on how multi-tasking is achieved in GPyTorch. Is there any relevant literature that I can reference ?

[Balandat]

yeah basically if you have n points and t tasks, gpytorch represents the joint covariance as an nt x nt matrix. You can represent that in different ways, either K_{data} \kron K_{task}, in which case you have n t x t matrices on the diagonal (i.e. "interleaved" w.r.t the data points), or K_{task} \kron K_{data}, in which case you have t n x n matrices on the diagonal. Depending on the use case one or the other representation may make more sense, hence the suggestion in #539.

Here K_{data} is the data covariance that depends on the hyperparameters, and K_{task} is a learned (often low-rank) correlation matrix. See #912 for some changes to the parameterization of that.

[Galto2000]

Hi @Balandat ,

One of my goals is to do "sensor fusion" (or data fusion) using GPs. In my case I have two different sensors measuring the same vector entity (a velocity in 2D). The sensors have different noise characteristics: sigma1 and sigma2.

I read a paper (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.296.1154&rep=rep1&type=pdf) that solves this from a heteroskedastic point of view.

I found your notebook with a heteroskedastic example: test_HadamardMultitaskMultiOutputHeteroskedasticLikelihood_universal.ipynb.txt

Firstly, I am trying to wrap my head around the example: mostly the last one where a GP is passed to the noise_covar of a likelihood - if you have some time could you perhaps explain this to me?

Secondly: if you have some time, would you perhaps have some suggestions on how I would be able achieve "sensor fusion" by treating my two different data sets as a heteroskedastic data set, where the noise levels are quantized: data with noise corresponding to the first sensor and data with noise corresponding to the second sensor?

Thanks in advance

Galto

[Galto2000]

actually I meant this one: test_MultitaskHeteroskedasticLikelihood.ipynb.txt
The very last example in the series

[Galto2000]

Hi @Balandat ,

In regards to the fixed noise Gaussian likelihood in a multi-task setting, I saw your draft - how do get these changes? I pip-installed GPyTorch - do I need to clone from github now?

Cheers

Galto

[Balandat]

@Galto2000 sorry I haven't gotten to work much on this - the draft isn't really in a usable state at this point, so unless you plan on actively developing it's probably not worth checking it out (which you would do by cloning the repo and checking out that branch). I'll try to get back to this soon-ish.

actually I meant this one: test_MultitaskHeteroskedasticLikelihood.ipynb.txt The very last example in the series.

Sorry what series do you mean exactly? Can you link to this?

Firstly, I am trying to wrap my head around the example: mostly the last one where a GP is passed to the noise_covar of a likelihood - if you have some time could you perhaps explain this to me?

It's pretty straightforward: if you have noise observations you can build a separate noise model. Typically this is fit on log-transformed data to ensure positivity and model multiplicative uncertainty. Then the prediction of that model at the input X is used as the noise level (rather than using fixed noises or a constant one). This has two benefits: (i) regularize the noise levels, in case these are themselves subject to observation noise [which they typically will be] and (ii) allow out-of-sample noise predictions, which is important for some more advanced acquisition functions in Bayesian Optimization. We have such a model checked in in BoTorch: https://github.com/pytorch/botorch/blob/master/botorch/models/gp_regression.py#L224

[Galto2000]

Thanks @Balandat for your reply.

I was referring to this little bit of code, that you posted some time ago:

train_x = torch.linspace(0, 1, 75)

sem_y1 = 0.05 + (0.75 - 0.05) * torch.linspace(0, 1, 75)
sem_y2 = 0.75 - (0.75 - 0.05) * torch.linspace(0, 1, 75)

train_y = torch.stack([
    torch.sin(train_x * (2 * math.pi)) + sem_y1 * torch.randn(train_x.size()),
    torch.cos(train_x * (2 * math.pi)) + sem_y2 * torch.randn(train_x.size()),
], -1)

train_y_log_var = torch.stack([(s ** 2).log() for s in (sem_y1, sem_y2)], -1)

log_noise_model = MultitaskGPModel(
    train_x,
    train_y_log_var,
    MultitaskGaussianLikelihood(num_tasks=2),
    num_tasks=2,
)

likelihood = _MultitaskGaussianLikelihoodBase(
    num_tasks=2,
    noise_covar=HeteroskedasticNoise(log_noise_model),
)
model = MultitaskGPModel(train_x, train_y, likelihood, num_tasks=2, rank=2)

I was wondering that I could do something analogous to get around the issue of not yet having MultitaskFixedGaussianNoise available.

In my case I have two observations of Y, over X, but at two different noise levels.
So I have observations y1 overx1 with known noise n1 and observationsy2 over x2 with known noise n2, and as such I concatenate or stack the tensors as follows
Y = [y1, y2], X=[x1,x2] and N = [n1, n2]

Now, pass N and X to a GP model with a linear kernel and pass that as noise_covar in a _MultitaskGaussianLikelihoodBase that will serve as the likelihood for a GP model that takes Y and X as their inputs. Multi-sensor fusion using GPs is my goal here.

It's going to be less computationally efficient than a MultitaskFixedGaussianNoise, but at this time that wouldn't bother me since the data is relatively small and it would be temporary until MultitaskFixedGaussianNoise comes online.

You see any issues with this approach?

Cheers

Galto

[Galto2000]

Hello, happy new year!

I was wondering if there is an ETA for the MultitaskFixedGaussianNoise ?

I tried the "heteroskedastic approach", in order to instil some fixed noise behavior in a multi-task setting, but there are many issues with doing it that way.

I am currently circumventing not having a MultitaskFixedGaussianNoise through using model-lists and (single task) FixedGaussianNoise and assuming the outcomes are independent, in order to make progress and in the hope that when MultiTaskFixedGaussianNoise comes available it would be relatively simple change at the end.

Cheers

Galto

[eytan]

I know that Max is out for the next week. BoTorch has support for MTGPs
with fixed noise... would something like
https://botorch.org/v/0.1.0/api/models.html#fixednoisemultitaskgp help?

On Mon, Jan 6, 2020 at 7:46 AM Galto2000 [e-mail] wrote:

Hello, happy new year!

I was wondering if there is an ETA for the MultitaskFixedGaussianNoise ?

I tried the "heteroskedastic approach", in order to instil some fixed
noise behavior in a multi-task setting, but there are many issues with
doing it that way.

I am currently circumventing not having a MultitaskFixedGaussianNoise
through using model-lists and (single task) FixedGaussianNoise and
assuming the outcomes are independent, in order to make progress and in the
hope that when MultiTaskFixedGaussianNoise comes available it would be
relatively simple change at the end.

Cheers

Galto

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cornellius-gp/gpytorch/issues/901?email_source=notifications&email_token=AAAW34IJJYUFRECHRW2BWM3Q4NG3TA5CNFSM4JA4PBV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIF2U5Q#issuecomment-571189878,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAW34LEUKWMDNZ7GCVQRQ3Q4NG3TANCNFSM4JA4PBVQ
.

[Galto2000]

@eytan

Thanks for pointing out the BoTorch fixed noise multitask - I'll check it out.