Ax: new UserWarning introduced by torch 1.7.0

[jieru-hu]

Our application started to get UserWarning from ax after the new torch 1.7.0 release

$ python example/banana.py -m 'banana.x=int(interval(-5, 5))' 'banana.y=interval(-5, 10.1)'
/private/home/jieru/.conda/envs/jh-test/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: Error instantiating/calling 'hydra_plugins.hydra_ax_sweeper.ax_sweeper.AxSweeper' : CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0

The torch warning was introduced in this torch PR and raise in ax in here

This warning is not relevant to my application and as a result it would be awesome if ax could suppress it.

[jieru-hu]

cc @shagunsodhani

[lena-kashtelyan]

@jieru-hu, @shagunsodhani, hi! You could probably leverage the warnings library to suppress these in your application?

Something like this could work (I also advise you to look into making it more granular so you are not just suppressing any UserWarning, but that's up to you):

import warnings
warnings.filterwarnings("ignore", category=UserWarning)

We'll also look into what is causing these, but you should be unblocked by the above in the meantime.

[ldworkin]

cc @2timesjay , can we remove the default here https://github.com/facebook/Ax/blob/master/ax/modelbridge/factory.py#L375 (or set default to None) and instead resolve in the body of the function? That way torch.cuda.is_available() would only be called if you're actually using this function (rather than by anybody who imports this module, which apparently is everybody)

[shagunsodhani]

@jieru-hu, @shagunsodhani, hi! You could probably leverage the warnings library to suppress these in your application?

Something like this could work (I also advise you to look into making it more granular so you are not just suppressing any UserWarning, but that's up to you):

import warnings
warnings.filterwarnings("ignore", category=UserWarning)

We'll also look into what is causing these, but you should be unblocked by the above in the meantime.

Hi @lena-kashtelyan

For the short-term, we can go with suppressing the pytorch warning. Though I think that since torch is a dependency on the side of Ax, Ax should be handling these warning and probably be raising an error (if required) which the Ax-plugin can handle.

Another thing I will mention is, this PR suggests that the warning should be thrown only when there is some issue with the gpu drivers. So the warning doesnt mean that a gpu wasnt found, it means that there was a driver issue. Maybe the warning could be avoided if Ax makes sure that drivers (if any) match the PyTorch version (as part of the setup process).

[lena-kashtelyan]

We definitely will look into this, there is a small fix we can put in to make the warning go away. Just suggesting that you are able to suppress the warnings on your side, and we on ours will deal with them appropriately too!

[lena-kashtelyan]

This warning is now removed on master and will be removed in the imminent stable version release.

[lena-kashtelyan]

The fix is now in the latest stable, 0.1.18.