Pytorch_geometric: topK in kernel benchmark script utilises <15% GPU

[ekagra-ranjan]

🐛 Bug

I was running the kernel benchmarks to evaluate the performances of the models. I started by running only topK by removing all the model name expect for topk in main.py. While running the script I found that topK is utilising only <15 % of the GPU (GTX 1080 Ti). It is taking 9 min for last config of hyperparam on MUTAG and >40 mins on last hyperparam on PROTEINS dataset. Is something wrong with topK implementation or is it normal for it to have such a low GPU utilisation?

To Reproduce

Steps to reproduce the behavior:

1.Remove all the names of models except topK here.

1. Run main.py

Environment

• OS: Ubuntu 16.04.5
• Python version: 3.7.3
• PyTorch version: 1.1.0
• CUDA/cuDNN version: 10.0
• GCC version: 5.4.0
• Any other relevant information:

[rusty1s]

I guess this depends on the batch_size, num_features and dataset which one uses, e.g. the MUTAG dataset is really small, so I think it is quite normal to expect low GPU utilisation?

[ekagra-ranjan]

Yes, that could be MUTAG. But for PROTEINS, the GPU util is not exceeding 15% and it takes total 40 min to run on all hyperparameters. In this case it will take around 2 days to run all 11 models in the list on 1 1080 Ti. Is the benchmark timing similar to what you must have done on your system?

[rusty1s]

Yes, it takes that long as well on my system. But in the end, we train around 8800 models (with 10-fold cross validation).

[ekagra-ranjan]

@rusty1s The benchmark is using stratified K-fold technique however I cannot see any mentions of it in the papers of diffpool, gPool or set2set. Wouldn't using stratified k fold increase the performance as there is a data leak from the test data to train and val due to the fact that we are considering the test distribution also before splitting to allow all the splits to have nearly equal proportion of classes?

[rusty1s]

Well, technically you are right, but personally I find this evaluation much more reasonable than hoping for some lucky splits. In the end all those datasets are quite balanced so this shouldn't affect the performance at all. I experienced this in my tests myself.

[ekagra-ranjan]

Thank you for the clarifications and help! I consider this as resolved and will now close this issue.