Gpytorch: SoftmaxLikelihood and building multiclass classification model

[mgarort]

Hi,

My issue is very similar to #994 by @cherepanovic, but I still don't understand multiclass classification after reading #994. I'm opening a new issue not to clutter #994 with my question.

I am trying to build a multiclass classification model using the SoftmaxLikelihood. However, I am not sure what the arguments of this likelihood function mean (the documentation on this function is very succint), and I haven't been able to figure it out by myself because I can't reproduce the only example I've found (SVDKL on CIFAR). Without any modification, I get the error

---> 50 from densenet import DenseNet
     51
     52 class DenseNetFeatureExtractor(DenseNet):

ModuleNotFoundError: No module named 'densenet'

If I define DenseNet as in your file densenet.py, I obtain the following error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-aa721b7a7db0> in <module>
    177 for epoch in range(1, n_epochs + 1):
    178     with gpytorch.settings.use_toeplitz(False):
--> 179         train(epoch)
    180         test()
    181     scheduler.step()

<ipython-input-8-aa721b7a7db0> in train(epoch)
    153             optimizer.zero_grad()
    154             output = model(data)
--> 155             loss = -mll(output, target)
    156             loss.backward()
    157             optimizer.step()

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/module.py in __call__(self, *inputs, **kwargs)
     22
     23     def __call__(self, *inputs, **kwargs):
---> 24         outputs = self.forward(*inputs, **kwargs)
     25         if isinstance(outputs, list):
     26             return [_validate_module_outputs(output) for output in outputs]

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/mlls/variational_elbo.py in forward(self, variational_dist_f, target, **kwargs)
     75         :return: Variational ELBO. Output shape corresponds to batch shape of the model/input data.
     76         """
---> 77         return super().forward(variational_dist_f, target, **kwargs)

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/mlls/_approximate_mll.py in forward(self, approximate_dist_f, target, **kwargs)
     55         # Get likelihood term and KL term
     56         num_batch = approximate_dist_f.event_shape.numel()
---> 57         log_likelihood = self._log_likelihood_term(approximate_dist_f, target, **kwargs).div(num_batch)
     58         kl_divergence = self.model.variational_strategy.kl_divergence().div(self.num_data / self.beta)
     59

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/mlls/variational_elbo.py in _log_likelihood_term(self, variational_dist_f, target, **kwargs)
     59
     60     def _log_likelihood_term(self, variational_dist_f, target, **kwargs):
---> 61         return self.likelihood.expected_log_prob(target, variational_dist_f, **kwargs).sum(-1)
     62
     63     def forward(self, variational_dist_f, target, **kwargs):

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/likelihoods/likelihood.py in expected_log_prob(self, observations, function_dist, *args, **kwargs)
     37
     38     def expected_log_prob(self, observations, function_dist, *args, **kwargs):
---> 39         likelihood_samples = self._draw_likelihood_samples(function_dist, *args, **kwargs)
     40         res = likelihood_samples.log_prob(observations).mean(dim=0)
     41         return res

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/likelihoods/likelihood.py in _draw_likelihood_samples(self, function_dist, sample_shape, *args, **kwargs)
     34             function_dist = base_distributions.Independent(function_dist, num_event_dims - 1)
     35         function_samples = function_dist.rsample(sample_shape)
---> 36         return self.forward(function_samples, *args, **kwargs)
     37
     38     def expected_log_prob(self, observations, function_dist, *args, **kwargs):

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/likelihoods/softmax_likelihood.py in forward(self, function_samples, *params, **kwargs)
     36         num_features, num_data = function_samples.shape[-2:]
     37         if num_features != self.num_features:
---> 38             raise RuntimeError("There should be %d features" % self.num_features)
     39
     40         if self.mixing_weights is not None:

RuntimeError: There should be 132 features

I may well be making a mistake, but I am copying and pasting everything in the example, without modifying any code myself.

My questions are:

1. What does the argument num_features to SoftmaxLikelihood mean? Is this the number of features of the inputs (e.g. if the input are MNIST images, is num_features 28*28=784? If so, why does the likelihood function need access to the number of features of the inputs, if it doesn't deal with the inputs themselves?

2. What does the argument num_classes to SoftmaxLikelihood mean? I'm assuming it is just the number of classes? E.g. in MNIST num_classes would be 10?

3. What are the mixing weights in a softmax function? The softmax that I know doesn't need any weights, but simply normalizes the logits by exponentiating them and dividing them by the sum of all the exponentiated logits.

4. What does the argument to num_tasks to MultitaskVariationalStrategy? This also has to be the number of classes, right?

5. What are the appropriate target dimensions when there are multiple classes? Should they be:

   • One-dimensional, with number of elements num_samples, and each target represented by a number? (e.g. 0-9 in MNIST)?

   • Two-dimensional, same as before, but with vertical vector, so dimensions are num_samples x 1?

   • Two-dimensional, but with a one-hot encoding, so the dimensions would be num_samples x num_classes?

Thanks a lot in advance.

[gpleiss]

What version of gpytorch are you using? Can you upgrade to the latest release (1.0, released a few hours ago) and see if you still have the same issue?
On Dec 20, 2019, 9:19 AM -0800, mgarort [e-mail], wrote:

Hi,
My issue is very similar to #994 by @cherepanovic, but I still don't understand multiclass classification after reading #994. I'm opening a new issue not to clutter #994 with my question.
I am trying to build a multiclass classification model using the SoftmaxLikelihood. However, I am not sure what the arguments of this likelihood function mean (the documentation on this function is very succint), and I haven't been able to figure it out by myself because I can't reproduce the only example I've found (SVDKL on CIFAR). Without any modification, I get the error
---> 50 from densenet import DenseNet
51
52 class DenseNetFeatureExtractor(DenseNet):

ModuleNotFoundError: No module named 'densenet'

If I define DenseNet as in your file densenet.py, I obtain the following error:

RuntimeError Traceback (most recent call last)
in
177 for epoch in range(1, n_epochs + 1):
178 with gpytorch.settings.use_toeplitz(False):
--> 179 train(epoch)
180 test()
181 scheduler.step()

in train(epoch)
153 optimizer.zero_grad()
154 output = model(data)
--> 155 loss = -mll(output, target)
156 loss.backward()
157 optimizer.step()

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/module.py in __call__(self, inputs, *kwargs)
22
23 def __call__(self, inputs, *kwargs):
---> 24 outputs = self.forward(inputs, *kwargs)
25 if isinstance(outputs, list):
26 return [_validate_module_outputs(output) for output in outputs]

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/mlls/variational_elbo.py in forward(self, variational_dist_f, target, *kwargs)
75 :return: Variational ELBO. Output shape corresponds to batch shape of the model/input data.
76 """
---> 77 return super().forward(variational_dist_f, target, *kwargs)

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/mlls/_approximate_mll.py in forward(self, approximate_dist_f, target, *kwargs)
55 # Get likelihood term and KL term
56 num_batch = approximate_dist_f.event_shape.numel()
---> 57 log_likelihood = self._log_likelihood_term(approximate_dist_f, target, *kwargs).div(num_batch)
58 kl_divergence = self.model.variational_strategy.kl_divergence().div(self.num_data / self.beta)
59

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/mlls/variational_elbo.py in _log_likelihood_term(self, variational_dist_f, target, *kwargs)
59
60 def _log_likelihood_term(self, variational_dist_f, target, *kwargs):
---> 61 return self.likelihood.expected_log_prob(target, variational_dist_f, *kwargs).sum(-1)
62
63 def forward(self, variational_dist_f, target, *kwargs):

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/likelihoods/likelihood.py in expected_log_prob(self, observations, function_dist, args, *kwargs)
37
38 def expected_log_prob(self, observations, function_dist, args, *kwargs):
---> 39 likelihood_samples = self._draw_likelihood_samples(function_dist, args, *kwargs)
40 res = likelihood_samples.log_prob(observations).mean(dim=0)
41 return res

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/likelihoods/likelihood.py in _draw_likelihood_samples(self, function_dist, sample_shape, args, *kwargs)
34 function_dist = base_distributions.Independent(function_dist, num_event_dims - 1)
35 function_samples = function_dist.rsample(sample_shape)
---> 36 return self.forward(function_samples, args, *kwargs)
37
38 def expected_log_prob(self, observations, function_dist, args, *kwargs):

~/housekeeping/virtualenv/ml2/lib/python3.6/site-packages/gpytorch/likelihoods/softmax_likelihood.py in forward(self, function_samples, params, *kwargs)
36 num_features, num_data = function_samples.shape[-2:]
37 if num_features != self.num_features:
---> 38 raise RuntimeError("There should be %d features" % self.num_features)
39
40 if self.mixing_weights is not None:

RuntimeError: There should be 132 features
I may well be making a mistake, but I am copying and pasting everything in the example, without modifying any code myself.
My questions are:

1. > What does the argument num_features to SoftmaxLikelihood mean? Is this the number of features of the inputs (e.g. if the input are MNIST images, is num_features 28*28=784? If so, why does the likelihood function need access to the number of features of the inputs, if it doesn't deal with the inputs themselves?
2. > What does the argument num_classes to SoftmaxLikelihood mean? I'm assuming it is just the number of classes? E.g. in MNIST num_classes would be 10?
3. > What are the mixing weights in a softmax function? The softmax that I know doesn't need any weights, but simply normalizes the logits by exponentiating them and dividing them by the sum of all the exponentiated logits.
4. > What does the argument to num_classes to MultitaskVariationalStrategy? This also has to be the number of classes, right?
5. > What are the appropriate target dimensions when there are multiple classes? Should they be:
   • > One-dimensional, with number of elements num_samples, and each target represented by a number? (e.g. 0-9 in MNIST)?
   • > Two-dimensional, same as before, but with vertical vector, so dimensions are num_samples x 1?
   • > Two-dimensional, but with a one-hot encoding, so the dimensions would be num_samples x num_classes?

Thanks a lot in advance.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[mgarort]

Hi @gpleiss

Thanks a lot for your quick reply! Upon upgrading I can reproduce the example, but now I am quite confused. In the SVDKL example, the output from the model (output = model(data)) has 132 dimensions, even though there are only 10 classes in CIFAR-10.

I think I am not understanding SoftmaxLikelihood and MultitaskVariationalStrategy properly. Do you think you could clarify the previous points? I think that will help me understand.

I'm copying them again here for convenience:

1. What does the argument num_features to SoftmaxLikelihood mean? Is this the number of features of the inputs (e.g. if the input are MNIST images, is num_features 28*28=784? If so, why does the likelihood function need access to the number of features of the inputs, if it doesn't deal with the inputs themselves?

2. What does the argument num_classes to SoftmaxLikelihood mean? I'm assuming it is just the number of classes? E.g. in MNIST num_classes would be 10?

3. What are the mixing weights in a softmax function? The softmax that I know doesn't need any weights, but simply normalizes the logits by exponentiating them and dividing them by the sum of all the exponentiated logits.

4. What does the argument num_tasks to MultitaskVariationalStrategy? This also has to be the number of classes, right?

5. What are the appropriate target dimensions when there are multiple classes? UPDATE From reproducing the example, it seems that the correct dimension of the target is one-dimensional, with number of elements being the number of samples and each target represented by a number.

[gpleiss]

In the SVDKL example, the output from the model (output = model(data)) has 132 dimensions, even though there are only 10 classes in CIFAR-10.

See equation 1 of the SVDKL paper. The output of the GP is a multi-output multivariate normal distribution of n x f, where f is num_features. The likelihood uses a linear mixing parameter (A in equation 1) to reduce this f dimensional output into a c dimensional output, where c is the number of classes num_classes.

1. num_features refers to f=132: the number of (independent) features that are output from the GP. If you look at the output of the GP layer, it will be a MultitaskMultivariateNormal distribution with event shape n x 132.

2. num_classes is c, the number of classification classes (e.g. 10 for mnist)

3. Again, see equation 1 of the paper

4. num_tasks here refers to f, the number of output dimension in the GP (so 132).

5. You are correct - the output of the likelihood is a Categorical distribution, which is represented by a c dimensional vector (one output for each class).

For more details, please refer to the SVDKL paper

[mgarort]

Thanks a lot @gpleiss! It is much clearer now. Sorry, I wasn't aware of the SVDKL paper.

[cherepanovic]

@gpleiss

Hello Geoff,

still one question, according to paper there are J Gaussian processes which will be applied to the features from the NN. What is J, how will it be built, from what does it depend?

In previous discussion, I thought that a separate GP will be learned for each output feature. If this is the case then J should be equal to C... isn't?

Thanks a lot for response!

In your description above
f = Q and n = J, right?

[gpleiss]

n is the number of data, and it shouldn't influence the number of neural network features or the dimensionality of the data.
f refers to the function that the GP learns - it is not a counting variable.

Q = J in our case - i.e. there is one Gaussian process for each neural network feature. (According to the experiments and the authors of the paper - this is the best setup to use).