Gpytorch: [Docs]

[dialuser]

📚 Documentation/Examples

Hello, I'm trying to implement a Variational Multioutput model by following the example at https://docs.gpytorch.ai/en/v1.1.1/examples/04_Variational_and_Approximate_GPs/SVGP_Multitask_GP_Regression.html

In my case, the number of tasks is defined by nTasks. The input dimension is n x d and the output dimension is n x nTasks.
The motivation was mentioned in my question from yesterday--the outputs need to be constrained to [0,1] range. I was advised to use BetaLikelihood.

My question is what the shape of induce points should be for the multi-output case. Is it nTasks x nLatentSpace x d ?
The example documentation is not very clear. Also, is there a multitask version of Betalikelihood somewhere. Thank you in advance from a newbie.

    class MultitaskVarGPModel(gpytorch.models.ApproximateGP):
        def __init__(self, inducing_points):
        nTasks = N*N
        variational_distribution = gpytorch.variational.CholeskyVariationalDistribution(
            inducing_points.size(-2), batch_shape=torch.Size([nTasks])
        )
        variational_strategy = gpytorch.variational.MultitaskVariationalStrategy(
            gpytorch.variational.VariationalStrategy(
            self, inducing_points, variational_distribution, learn_inducing_locations=True
            ), num_tasks=nTasks
        )
        super().__init__(variational_strategy)
        self.mean_module = gpytorch.means.ConstantMean(batch_shape=torch.Size([nTasks]))
        self.covar_module = gpytorch.kernels.ScaleKernel(
            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([nTasks])),
            batch_shape=torch.Size([nTasks])
        )

        def forward(self, x):
        # The forward function should be written as if we were dealing with each output
        # dimension in batch
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

* Is there documentation missing? *

* Is documentation wrong? *

* Is there a feature that needs some example code? *

* Think you know how to fix the docs? * (If so, we'd love a pull request from you!)

• Link to GPyTorch documentation
• Link to GPyTorch examples

[gpleiss]

Is it nTasks x nLatentSpace x d ?

Yes

Is there a multitask version of Betalikelihood somewhere?

It should possibly work right out of the box with multitask models - if not give us a code example and a stack trace.

[dialuser]

Hi, all multitask likelihoods have this num_tasks parameter, right? I'm not clear when you said it should work out of the box. Anyway, I'd like to replace MultitaskGaussianLikelihood with Beta in the following,

likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=train_y.shape[1])

Thanks.

[gpleiss]

Hi, all multitask likelihoods have this num_tasks parameter, right?

This is only because the Gaussian likelihood has a parameter that's dependent on the number of tasks. This is not the case with Beta likelihoods, so it doesn't require a num_tasks argument.

What happens when you replace that line with

likelihood = gpytorch.likelihoods.BetaLikelihood()

[ZhiliangWu]

Hi @gpleiss, I am also reading this multitask SVGP tutorial and would like to ask what does it mean by batch dimension in Variational Strategies for Multi-Output Functions, as both task_dim in IndependentMultitaskVariationalStrategy and latent_dim in LMCVariationalStrategy referred it. Does it refer to the number of GPs in the batch GP as we defined in e.g., CholeskyVariationalDistribution through the batch_shape or something else? Where did we set this during building the multi-task SVGP? The default suggested value of -1 seems a bit abstract to me. I would really appreciate some simple examples to clarify the cases of different values of it. Thanks for your reply in advance.

[gpleiss]

@ZhiliangWu LMCVariationalStrategy and IndependentMultitaskVariationalStrategy both convert a batch of GPs into a multitask GP. For example, the IndependentMultitaskVariationalStrategy converts the output of d Gaussian processes into a n x d MultitaskMultivariateNormal.

Both IndependentMultitaskVariationalStrategy and LMCVariationalStrategy must operate on a batch of Gaussian processes, since multiple GPs are needed for multiple output dimensions. (See section 4.2 of this paper for more information).

However, it's possible that you might want to have a batch of c multi-output GPs, each with d dimensions. In this case, the IndependentMultitaskVariationalStrategy would have a batch shape of [c, d], where d would correspond to the multiple outputs, and c would correspond to the actual batch of multi-output GPs. In this case, the task_dim argument specifies that the d dimension corresponds to the GP outputs.

[ZhiliangWu]

Hi @gpleiss, thanks for the detailed explanation and pointers. Just to help my understanding:

the IndependentMultitaskVariationalStrategy converts the output of d Gaussian processes into a n x d MultitaskMultivariateNormal.

In this case, does it mean each task is actually a linear combination of the d GP processes? In the case of n=d=2, it is NOT the same as implementing two separated GPs, each for a different output as the following slide (from the author you pointed to at GPSS2017)

Then it looks quite like LMC to me, as these batch GPs work similar to latent functions. (I must have missed something. :exploding_head::exploding_head::exploding_head:)

it's possible that you might want to have a batch of c multi-output GPs, each with d dimensions.

In such a case, we set batch_shape=torch.Size([c, d]) in e.g., gpytorch.variational.CholeskyVariationalDistribution? I cannot imagine the cases where we might need such a setup, since directly setting batch_shape to the value of c x d makes more sense to me. Could you also kindly give some examples?