Gpytorch: What is ard_num_dims useful for?

[eytan]

What is ard_num_dims useful for? Are there any use cases for when you'd want ARD for some kernel parameters but not others (at least within the context of a single constructor call)?. It's also not clear what the expected behavior is when ard_num_dims is not equal to the number of dimensions being fed into the kernel.

[gpleiss]

When ard_num_dims is not set, then we use the same scaling parameters for each dimension. This makes for a simpler model and can work when your dimensions operate at roughly the same scale (e.g. if you preprocess your data). This also simplifies the model by reducing the number of MLE-estimated parameters.

If it is set, and set to something that is not the dimensionality then there will be an error.

If you’re using multiple kernels, I can’t necessarily think of a case where you’d want some kernels with ard and some without, but it’s possible!

[eytan]

I guess to put it another way, why not just have a flag that says ard=True or False? :) it would get rid of a lot of boilerplate and room for error.

Sent from my iPhone

On Oct 4, 2018, at 7:47 AM, Geoff Pleiss [e-mail] wrote:

When ard_num_dims is not set, then we use the same scaling parameters for each dimension. This makes for a simpler model and can work when your dimensions operate at roughly the same scale (e.g. if you preprocess your data). This also simplifies the model by reducing the number of MLE-estimated parameters.

If it is set, and set to something that is not the dimensionality then there will be an error.

If you’re using multiple kernels, I can’t necessarily think of a case where you’d want some kernels with ard and some without, but it’s possible!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[Balandat]

The reason ARD is not just a flag is b/c we need to instantiate the lengthscale Parameter with the correct size. A flag wouldn’t give us that info.

No we could potentially change things to auto-instantiate the parameter upon the first kernel call, inferring the number of parameters, but that seems like too much magic (and complication of the kernel code internals)

[gpleiss]

We tried originally having the auto-instantiating parameters, and it’s a nightmare for saving/restoring models. I think users will have to supply the actual number of dimensions.

This mirrors what’s currently done in most PyTorch layers - the user must specify the appropriate number of dimensions for linear layers, etc.
On Oct 4, 2018, 10:58 AM -0400, Max Balandat [e-mail], wrote:

The reason ARD is not just a flag is b/c we need to instantiate the lengthscale Parameter with the correct size. A flag wouldn’t give us that info.
No we could potentially change things to auto-instantiate the parameter upon the first kernel call, inferring the number of parameters, but that seems like too much magic (and complication of the kernel code internals)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[eytan]

That makes sense. From a user-interface point of view it seems weird that you only need to specify the number of dimensions when using ARD. I don't know if that is good enough reason to force everybody to manually write out the number of dimensions that you are planning on using. Is it possible that there might be some downstream benefits / simplifications that could happen if you specified the number of dims in advance for all Kernels?

[Balandat]

Hmm while that would buy you consistency and a more explicit constructor, it would make things more verbose. Also, if you wanted to benefit from the pytorch broadcasting features under the hood and in fact share hyperparmeters, then you'd have to add another kwarg to the constructor to make that work...

[eytan]

@Balandat, can you clarify what you mean by your last comment about broadcasting features and sharing HPs? Requiring users to specify the number of dimensions in their kernels no matter what is a lot more consistent with what is expected of most PyTorch layers, as @gpleiss suggests.

While I recognize that a lot of GPyTorch was written as a platform for doing research on DKL, SKI(P), etc, where you generally wouldn't want to enable ARD, I suspect that almost all of the folks using GPyTorch for vanilla GPs will want ARD enabled. If we were to make all kernels require an input size and then have a boolean flag for ARD, we would then only really be (slightly) inconveniencing folks who are using DKL and are already having to keep careful track of their input/output dimensionality. The benefit is then clearer code and less consistency issues, such as #328.

[gpleiss]

@eytan that's a fair point. I think it would be reasonable to require ARD by default. However, I'm a bit hesitant to introduce a breaking change into GPyTorch so soon after our beta release...

[Balandat]

About the broadcasting, see https://github.com/cornellius-gp/gpytorch/issues/328#issuecomment-430989404

[eytan]

If this is something you wanted to change, I would recommend introducing new kwargs / phasing out the old API incrementally (e.g., introduce input_dims and an ard flag kwargs, and throw a warning that they should be more explicit, and then later take the old API away in one or two point releases). Personally, I find it to be a lot less magical and intuitive as a new user. Perhaps y'all will want to see if the ard continues to be a point of confusion, and if it does, come up with some plan to slowly move people to some required input dimension w/ a boolean flag for ARD.

[Balandat]

Sounds reasonable to me. I can put up a PR for that if @gpleiss and @jacobrgardner are ok with that soft deprecation.