Autogluon: Tabular Neural Network hanging non-deterministically

[Innixma]

Defect introduced recently sometime around Open Source. I have disabled NN by default in Tabular until we find the cause and fix it.

[zhanghang1989]

Thanks for opening the issue tracker! You mentioned it was a recent change causing this issue, right? Do you know which version was working okay? I can compare the diff to debug that.

[Innixma]

Defect introduced on PR #94 due to adding pynvml dependency. For some reason pynvml causes tabular NN to permanently hang. Removing pynvml.

[Innixma]

Opened #153 to fix

[Innixma]

Merged

[austinmw]

Hi, I'm also experiencing this NN hanging, in SageMaker using the MXNet DLAMI's as base images. I thought that a previous update fixed this hanging for me, but looks like it persisted. It doesn't look like these images have pynvml installed. This happens with both CPU and GPU images:

197237853439.dkr.ecr.us-east-2.amazonaws.com/autogluon-tabular-data-cpu-py36:latest and
197237853439.dkr.ecr.us-east-2.amazonaws.com/autogluon-tabular-data-gpu-py36-cu101:latest

To test, I've been trying different hyperparameter options. Any time I use NN only, the predict calls hang.

My local environment doesn't experience this issue, so I'm currently trying to tease out the difference(s). I'll update if I find anything. In case anyone gets a chance to look into the issue more, I'll attach the pip freeze of the cpu/gpu environments where it's failing.
cpu_freeze.txt
gpu_freeze.txt
Dockerfile.txt

[Innixma]

@austinmw This may be related to #345. Uncertain at this time though, as your issue is only on SageMaker. If it persists after #345 is merged, please make a new issue for this and we will look into it.

In your cpu_freeze.txt, you have LightGBM 2.3.0, but it should be 2.3.1 if you are using latest AutoGluon in mainline. Did you pip install (not fixed) or build from latest source (fixed)?

From your Dockerfile it looks like you are building from source, but then how are you not getting LightGBM 2.3.1?

[austinmw]

Oops, good catch, had LightGBM 2.3.0 in my requirements file. Same issue with latest versions though. Will wait for that merge and check again, thanks.

By the way, if I could ask, are there significant performance gains to be had from using a GPU with tabular? Thinking that maybe NN/lightgbm/catboost could use a GPU, and potentially other things like feature pruning, but not quite sure.

[Innixma]

I'd recommend considering removing your requirements file altogether to ensure AutoGluon is installed with its preferred dependencies.

As for GPU, we have #263 . Currently Tabular does not support GPU but it could likely achieve significant speedups in areas you mentioned if the functionality was added. Feature pruning is an interesting use-case, but would require some more thought on how it could leverage GPUs.

[austinmw]

Tested with latest master and still hanging with NN. May be out of scope here since it's only happening in sagemaker, but will continue trying to figure out what's causing it.