Gpytorch: Training time comparison between GPytorch and GPflow.

[Hebbalali]

Hi all!

After reading the paper "GPyTorch: Blackbox Matrix-Matrix Gaussian
Process Inference with GPU Acceleration" i was interested to migrate from GPflow to GPytorch for obvious reasons of training speed.
Recently i began to do some playing with GPytorch. But the surprising result is that GPflow is faster in training than GPytorch in my experiments.
In an input space of dimension 25 and with 375 training points and with a Tesla P100 as GPU the GPflow training time using 500 Adam iterations takes approximatively 3 sec while the GPytorch training time using also 500 Adam iterations takes 11 sec.
When assessing the GPU usage via watch -n 2 nvidia-smi the tensorflow code takes all the memory usage (16280MiB) while the pytorch code takes only (800 MiB) which is expected since pytorch allocate GPU memory dynamically. However, in the training while GPflow uses 70% of the GPU (Volatile GPU-Util), GPytorch does not exceed 22%.

The following code is used for GPytorch

class ExactGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(ExactGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())
    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)
x_cuda=x_train.cuda()
y_cuda=y_train.cuda()
likelihood = gpytorch.likelihoods.GaussianLikelihood()  
model = ExactGPModel(x_cuda, y_cuda, likelihood1).cuda() 
model.train()
likelihood.train()
optimizerm = torch.optim.Adam([{'params': model.mean_module.parameters()},{'params': model.covar_module.parameters()},{'params': model.likelihood.parameters()}], lr=0.1)
def train(training_iter = 500):
    for k in range(training_iter):   
        optimizerm.zero_grad()
        outputm = model(x_cuda)
        loss = -mllm1(outputm1, y_cuda)
        loss.backward()
        #print('Iter %d/%d - Loss: %.3f' % (i + 1, training_iter, loss.item()))
        optimizerm.step()

with gpytorch.settings.use_toeplitz(False):
    train()

And the following code is used for GPflow:

likelihood = gpflow.likelihoods.Gaussian()
kernel = gpflow.kernels.RBF(dim)
model = gpflow.models.GPR(x_train,y_train,kern=kernel)
adam_action = AdamOptimizer(learning_rate=0.1).make_optimize_action(model)
actions=[adam_action]
Loop(actions, stop=5000)()

Thank you in advance.

[jacobrgardner]

The reason is simply that the ~300 data point regime is not what we've been focused on optimizing recently because the speed difference is somewhat uninteresting in that regime. As a result, we may have lost some advantage here, but we're willing to pay this for the performance we've gained at even slightly larger data sizes.

At the moment, we currently use Cholesky for inference on up to 256 data points by default which you can modify with the gpytorch.settings.max_cholesky_size context.

• If I run autompg (~352 data points) with GPyTorch Cholesky, it takes about 3s and roughly matches GPFlow. CG takes about 14s for me to complete 500 iterations.

• On the other hand, if I run skillcraft (~3000 data points), CG takes about 16s for me to complete 500 iterations, while Cholesky takes around 260s.

Looking at Table 1 here, we can run on 10x as much data as skillcraft (n=~30k) before CG starts consistently taking more than 100s.

[eytan]

@Hebbalali , I'd be curious how the performance compares if you use 255 training points instead of 300. AFAIK the choice in when we switch from Cholesky to CG (at 256 points) is somewhat arbitrary, so it's possible that we may wish to increase that threshold as the default.

[eytan]

Woops, didn't read @jacobrgardner's response carefully enough. Given your benchmark, Jake, perhaps we can consider upping the threshold from 256 to a higher number? Not sure how this might vary between CPU and GPU. cc @Balandat

[jacobrgardner]

@eytan good point. That threshold was chosen pretty arbitrarily, and it may move around. Let me do a benchmark in a little bit and see if it should be e.g. 512.

I think we should also maybe have a goal for the package where optimizations we do on large scale settings don't push it up too far. Maybe a good design goal would be to keep the threshold under 1000 for when CG starts becoming clearly faster?

[Hebbalali]

Thank you for your quick responses.
As you mentionned increasing the max_cholesky_size allows roughly the same time of training between GPflow and GPytorch.
I also increased the number of points to 2500, and in this case as expected the CG is faster (takes around 24 sec) while the cholesky decomp (takes around 27 sec).

[Balandat]

I would also assume there to be a large difference between CPU and GPU (with linear CG being faster relative to Cholesky on the GPU...)

[gpleiss]

@Balandat that's a good point - we should probably have different settings for CPU vs GPU.

[gpleiss]

i'm going to close this for now - due to updates like #928