Autogluon: Hyperparameter tuning never completes for a tabluar task

[tigerhawkvok]

As stated.

Without tuning, the model runs in ~400s.

4 days later, no steps have completed on a hyperparameter_tune= True session:

The VM's pip freeze output:

absl-py==0.9.0
apt-clone==0.2.1
apturl==0.5.2
asn1crypto==0.24.0
astor==0.8.1
astroid==2.3.3
astropy==4.0
autogluon==0.0.4
bcrypt==3.1.7
bokeh==1.4.0
boto3==1.9.187
botocore==1.12.253
Brlapi==0.6.6
catboost==0.20.2
certifi==2018.1.18
cffi==1.13.2
chardet==3.0.4
Click==7.0
cloudpickle==1.2.2
command-not-found==0.3
ConfigSpace==0.4.10
cryptography==2.8
cupshelpers==1.0
cycler==0.10.0
Cython==0.29.14
dask==2.6.0
defer==1.0.6
dill==0.3.1.1
distributed==2.6.0
distro-info===0.18ubuntu0.18.04.1
docutils==0.15.2
fastparquet==0.3.2
gast==0.2.2
gluoncv==0.6.0
gluonnlp==0.8.1
google-pasta==0.1.8
gpg==1.10.0
graphviz==0.8.4
grpcio==1.26.0
h5py==2.10.0
HeapDict==1.0.1
httplib2==0.9.2
idna==2.6
isort==4.3.21
Jinja2==2.10.3
jmespath==0.9.4
joblib==0.11
Keras==2.3.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keyring==10.6.0
keyrings.alt==3.0
kiwisolver==1.1.0
language-selector==0.1
launchpadlib==1.10.6
lazr.restfulclient==0.13.5
lazr.uri==1.0.3
lazy-object-proxy==1.4.3
lightgbm==2.3.0
llvmlite==0.31.0
louis==3.5.0
macaroonbakery==1.1.3
Mako==1.0.7
Markdown==3.1.1
MarkupSafe==1.0
matplotlib==3.1.2
mccabe==0.6.1
msgpack==0.6.2
mxnet==1.5.1.post0
netifaces==0.10.4
numba==0.47.0
numpy==1.18.1
oauth==1.0.1
olefile==0.45.1
opt-einsum==3.1.0
packaging==20.0
PAM==0.4.2
pandas==0.25.3
paramiko==2.7.1
pexpect==4.2.1
Pillow==5.1.0
plotly==4.4.1
portalocker==1.5.2
protobuf==3.11.2
psutil==5.6.7
pycairo==1.16.2
pycparser==2.19
pycrypto==2.6.1
pycups==1.9.73
pygobject==3.26.1
PyICU==1.9.8
pylint==2.4.4
pymacaroons==0.13.0
PyNaCl==1.1.2
pyparsing==2.4.6
pyRFC3339==1.0
python-apt==1.6.5+ubuntu0.2
python-dateutil==2.6.1
python-debian==0.1.32
pytz==2018.3
pyxdg==0.25
PyYAML==3.12
reportlab==3.4.0
requests==2.22.0
requests-unixsocket==0.1.5
retrying==1.3.3
s3transfer==0.2.1
scikit-learn==0.22.1
scikit-optimize==0.5.2
scipy==1.4.1
SecretStorage==2.3.1
simplejson==3.13.2
six==1.14.0
sortedcontainers==2.1.0
system-service==0.3
systemd-python==234
tblib==1.6.0
tensorboard==1.15.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
thrift==0.13.0
toolz==0.10.0
tornado==6.0.3
tqdm==4.41.1
typed-ast==1.4.1
typing==3.7.4.1
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.22
usb-creator==0.3.3
wadllib==1.3.2
Werkzeug==0.16.0
wrapt==1.11.2
xkit==0.0.0
zict==1.0.0
zope.interface==4.3.2

[Innixma]

This looks like a major issue, investigating.

[Innixma]

Very interesting, we will have to take a look into this. I wasn't able to reproduce this trivially, so I will have to do a deeper dive by recreating the venv with your versions. It appears that it somehow stalled while training LightGBM.

Hyperparameter tuning may not be perfectly stable as of yet, especially if using a searcher other than random search. We are still dealing with some weird interactions dask has with our scheduler, and plan to improve hyperparameter tuning stability and effectiveness significantly in the next few months.

My suggestion to you for now would be to use auto_stack=True and hyperparameter_tune=False, we find that auto_stack provides much more benefit in less time than hyperparameter_tune in most problems, and is extremely stable (you won't find any weird stalling issues with auto_stack=True so long as hyperparameter_tune=False).

[tigerhawkvok]

@Innixma Thanks, I'll give it a try and let you know. If you have trouble replicating, let me know and I'll do what I can to assist.

[tigerhawkvok]

Confirming predictor = task.fit(train_data=task.Dataset(agTrainable), label="solutionLabel", auto_stack= True, hyperparameter_tune= False) runs to completion, taking ~15x longer than a standard run (as advertised)

I'd still love to do the full search and it's still hanging, so I'll leave this issue open, but thanks for the workaround!

[luyifanlu]

I also get same problem,but when i run the code as sugesstion ,i get a error:ValueError: Unknown keyword argument specified: auto_stack
predictor = task.fit(train_data=train_data[features], tuning_data=valid_data[features], label=label_column, output_directory=output_directory, time_limits=time_limits, num_trials=num_trials, auto_stack= True ,hyperparameter_tune=hp_tune, hyperparameters=hyperparameters, search_strategy=search_strategy)

[luyifanlu]

i tried
pip install tornado == 5. * pip install ipykernel
solve the problem.may it work for u

[Innixma]

@luyifanlu if auto_stack was not found as a valid argument, then you are using an old version of AutoGluon. auto_stack was added in 0.0.4.

Please try pip install --upgrade autogluon

[jwmueller]

This issue may be caused by the scheduler/searcher failing to handle errors during the individual training trial: https://github.com/awslabs/autogluon/issues/329

We have included a quick fix for TabularPrediction (thanks to @razmikmelikbekyan): https://github.com/awslabs/autogluon/pull/322, so can you try the full search again now on your dataset after pulling latest version of autogluon?

Note that if your goal is to maximize predictive accuracy, auto_stack = True will still likely be much better than hyperparameter_tune=True (due to potential overfitting). We recommend the latter for TabularPrediction only if your goal is to deploy one model only rather than an ensemble.

[Innixma]

@tigerhawkvok

I believe we have solved this issue. If you build from source on latest mainline, it should work as intended with HPO. Let us know if you get a chance to try it out!