Gpytorch: Parallel GPU training

[stanbiryukov]

Hi there,
Has anyone been able to train a GP model utilizing multiple GPUs via torch.nn.DataParallel?

[jacobrgardner]

It's not something we've specifically designed the package to support currently, but multi GPU support is definitely something worth considering. The most obvious place multi GPU support would be useful are the matrix multiplications we do in CG. This might not be super difficult to implement for exact GPs, but I expect it may require a significant amount of effort for the sparse GP methods.

[stanbiryukov]

Great - I've been attempting to implement multi-gpu training for an exact GP classification model. I think for larger models where exact GP is desired this will be a useful enhancement. Will loop back in if I make progress.

[eytan]

@stanbiryukov, any updates on this?

[stanbiryukov]

@eytan No updates yet - haven’t been able to spend more time with parallelism. For my particular problem I simply used a sparser dataset.

[andrewgordonwilson]

@stanbiryukov , Incidentally, sparse GPs, such as SKI, can be exact for all practical purposes, and would typically be preferable to using an exact GP on sparse data.

[stanbiryukov]

Good point, using KISSGP on all training data instead of exact GP on sparser data improved my out of sample RMSE. Thanks @andrewgordonwilson

[jacobrgardner]

@stanbiryukov #450 adds support for multiple GPUs!