Autogluon: Tabular: os.fork() called by MXNet NN model during prediction causing OOM

[Innixma]

This OOM error was caused during training of Albert OpenML dataset (Large) on m5.2xlarge. This OOM error has never been encountered before. (Happened 3 times on latest benchmark run on Albert and Airlines datasets).

The error happened during the setup for training the weighted ensemble with bagging and stacking disabled. Therefore, each model needs to do predictions on the validation data in turn (The amount of validation data is very small, less than 10,000, and therefore should not cause issues). For whatever reason, TabularNN is forking the process, which likely leads to a doubling of memory usage.

Question: Why are we running OOM during a stage that should require very minimal memory usage, and why are we forking the process?

Stack Trace, note that the memory check shortly prior to this indicates 32% memory usage:

[INFO] [amlb.utils.process:01:07:58.029] CPU Utilization: 41.7%
[INFO] [amlb.utils.process:01:07:58.029] Memory Usage: 32.1%
[INFO] [amlb.utils.process:01:07:58.029] Disk Usage: 15.7%
[ERROR] [amlb.benchmark:01:08:55.237] [Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/home/ubuntu/workspace/automlbenchmark/amlb/benchmark.py", line 391, in run
    meta_result = framework.run(self._dataset, task_config)
  File "/home/ubuntu/workspace/automlbenchmark/frameworks/autogluon_nobag/__init__.py", line 4, in run
    return run(*args, **kwargs)
  File "/home/ubuntu/workspace/automlbenchmark/frameworks/autogluon_nobag/exec.py", line 16, in run
    return exec_template.run(dataset=dataset, config=config, parameters=parameters)
  File "/home/ubuntu/workspace/automlbenchmark/frameworks/autogluon/exec_template.py", line 82, in run
    **fit_params
  File "/home/ubuntu/workspace/autogluon/autogluon/task/tabular_prediction/tabular_prediction.py", line 425, in fit
    hyperparameters=hyperparameters, time_limit=time_limits_orig, save_data=cache_data, verbosity=verbosity)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/learner/default_learner.py", line 99, in fit
    hyperparameters=hyperparameters)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/trainer/auto_trainer.py", line 39, in train
    self.train_multi_and_ensemble(X_train, y_train, X_test, y_test, models, hyperparameter_tune=hyperparameter_tune, feature_prune=feature_prune)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 507, in train_multi_and_ensemble
    self.train_multi_levels(X_train, y_train, X_test, y_test, models=models, hyperparameter_tune=hyperparameter_tune, feature_prune=feature_prune, level_start=0, level_end=self.stack_ensemble_levels)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 522, in train_multi_levels
    self.stack_new_level(X=X_train, y=y_train, X_test=X_test, y_test=y_test, models=models, level=level, hyperparameter_tune=hyperparameter_tune, feature_prune=feature_prune, time_limit_core=time_limit_core, time_limit_aux=time_limit_aux)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 533, in stack_new_level
    aux_models = self.stack_new_level_aux(X=X_test, y=y_test, fit=False, level=level+1, time_limit=time_limit_aux)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 568, in stack_new_level_aux
    X_train_stack_preds = self.get_inputs_to_stacker(X, level_start=0, level_end=level, fit=fit)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 797, in get_inputs_to_stacker
    X = dummy_stackers[level+1].preprocess(X=X, preprocess=False, fit=False, compute_base_preds=True)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/models/ensemble/stacker_ensemble_model.py", line 92, in preprocess
    y_pred_proba = base_model.predict_proba(X)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py", line 384, in predict_proba
    return self._predict_tabular_data(new_data=X, process=True, predict_proba=True)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py", line 398, in _predict_tabular_data
    new_data = self.process_test_data(new_data, batch_size=self.batch_size, num_dataloading_workers=self.num_dataloading_workers, labels=None)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py", line 486, in process_test_data
    problem_type=self.problem_type, labels=labels, is_test=True)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_dataset.py", line 150, in __init__
    self.generate_dataset_and_dataloader(data_list=data_list)
  File "/home/ubuntu/workspace/autogluon/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_dataset.py", line 158, in generate_dataset_and_dataloader
    num_workers=self.num_dataloading_workers)  # no need to shuffle test data
  File "/home/ubuntu/virtual/automlbenchmark/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py", line 642, in __init__
    initargs=[self._dataset, is_np_shape(), is_np_array()])
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/pool.py", line 174, in __init__
    self._repopulate_pool()
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/pool.py", line 239, in _repopulate_pool
    w.start()
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/popen_fork.py", line 66, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

[Innixma]

pip freeze output:

attrs==19.3.0
# Editable install with no version control (autogluon==0.0.7b11042020)
-e /home/ubuntu/workspace/autogluon
# Editable install with no version control (autogluon-utils==0.0.1)
-e /home/ubuntu/workspace/autogluon-utils
Babel==2.8.0
bcrypt==3.1.7
boto3==1.12.36
botocore==1.15.36
catboost==0.22
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
click==7.1.1
cloudpickle==1.3.0
colorama==0.4.3
ConfigSpace==0.4.10
cryptography==2.9
cycler==0.10.0
Cython==0.29.16
dask==2.6.0
deap==1.3.1
debtcollector==2.0.0
decorator==4.4.2
distributed==2.6.0
docutils==0.15.2
fasteners==0.15
fire==0.3.0
fsspec==0.7.1
future==0.18.2
gluoncv==0.7.0b20200405
gluonnlp==0.8.1
graphviz==0.8.4
h2o==3.30.0.1
HeapDict==1.0.1
idna==2.9
importlib-metadata==1.6.0
ipython-genutils==0.2.0
iso8601==0.1.12
jmespath==0.9.5
joblib==0.14.1
jsonschema==3.2.0
jupyter-core==4.6.3
kaggle==1.5.6
kiwisolver==1.2.0
liac-arff==2.3
lightgbm==2.3.1
matplotlib==3.2.1
mock==4.0.2
monotonic==1.5
more-itertools==8.2.0
msgpack==1.0.0
mxnet==1.6.0
nbformat==5.0.5
netaddr==0.7.19
netifaces==0.10.9
networkx==2.4
nose==1.3.7
numpy==1.18.2
openml==0.7.0
oslo.concurrency==4.0.1
oslo.config==8.0.1
oslo.i18n==4.0.0
oslo.utils==4.1.0
packaging==20.3
pandas==0.25.3
paramiko==2.7.1
pbr==5.4.4
Pillow==6.2.1
plotly==4.6.0
pluggy==0.13.1
portalocker==1.6.0
protobuf==3.11.3
protobuf3-to-dict==0.1.5
psutil==5.7.0
py==1.8.1
pyaml==20.4.0
pyarrow==0.16.0
pycparser==2.20
PyNaCl==1.3.0
pyparsing==2.4.6
pyrsistent==0.16.0
pytest==5.4.1
python-dateutil==2.8.1
python-slugify==4.0.0
pytz==2019.3
PyYAML==5.3.1
requests==2.23.0
retry==0.9.2
retrying==1.3.3
rfc3986==1.3.2
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
s3fs==0.4.2
s3transfer==0.3.3
sagemaker==1.55.0
scikit-learn==0.20.4
scikit-optimize==0.8.dev0
scipy==1.2.3
six==1.14.0
smdebug-rulesconfig==0.1.2
sortedcontainers==2.1.0
stevedore==1.32.0
stopit==1.1.2
tabulate==0.8.7
tblib==1.6.0
termcolor==1.1.0
text-unidecode==1.3
toolz==0.10.0
tornado==6.0.4
TPOT==0.11.1
tqdm==4.45.0
traitlets==4.3.3
typing==3.7.4.1
update-checker==0.16
urllib3==1.25.8
wcwidth==0.1.9
wrapt==1.12.1
xmltodict==0.12.0
zict==2.0.0
zipp==3.1.0

[jwmueller]

To clarify, you never saw this error previously show up on Albert/Airlines before this issue was posted, right? Or do you think that in the previous runs, the NN never got a chance to produce this error due to time limits, and it is now producing the error because of increased efficiency in the other models?

[Innixma]

I believe the latter given more thought. This run was on non-bagged, so every model trains 10x faster in a sense. It is also possible that it didn't run OOM on bagged version because it failed in the try/except part of the code during training (NN predicts on validation in fit() with bagging, which the exception can get caught and NN gets skipped).

[jwmueller]

One fix that should avoid the os.fork() / multiprocessing is to change the self.num_dataloading_workers in these lines:
https://github.com/awslabs/autogluon/blob/7e551a22921de4c2fdef594f99f7b06896a5ed6b/autogluon/utils/tabular/ml/models/tabular_nn/tabular_nn_model.py#L172-L174

to simply always have:
self.num_dataloading_workers = 0

This just supposes extra available threads will be utilized by the installed MXNet (MKL) as appropriate for the network computations rather than data-loading. It might be too unsafe for us to presume we can more aggressively ask for extra dataloading_worker threads for arbitrary heterogeneous datasets, especially when the available memory differs during training/validation/prediction phases based on outside context.

If num_dataloading_workers = 0 fixes the issue, we could discuss setting num_dataloading_workers > 0 during training and num_dataloading_workers = 0 during prediction. It's just unfortunately hard to come up with a single setting that exploits efficiency gains where possible but still works on any dataset.

[Innixma]

Do you know why it is running OOM? Is the fork causing memory doubling? Does it call fork for every core? Would it make sense to multi-thread instead? Forking and doubling memory would also be a significant time overhead, which would actually slow down most inference.

[zhreshold]

Discussed offline with @jwmueller , according to the existing use case, there's no reason to use multiprocessing workers, since each worker process has to keep a copy of original ndarray.

Two choices here:

• Stick with num_workers=0 for DataLoader, to avoid duplicate data in child processes
• Use num_workers>0, together with thread_pool=True, this will not bypass GIL, but can significantly reduce the memory usage, while potentially improve the IO

[jwmueller]

Thanks for your input! I think we should first try thread_pool=True to try and mitigate potential slowdowns.

Here is PR: https://github.com/awslabs/autogluon/pull/416

[Innixma]

I was able to demonstrate NN slowdown on the current setting by creating a large list in memory before calling task.fit():

long_lst = [i for i in range(200000000)]

On my Mac, this 7 GB list increased inference time by ~20% consistently for the NN. This would probably be much more severe on a larger machine with more cores and more data in memory. Note that the other ML models were not impacted when LongLst was in memory.

Inference Time by Scenario:
Normal: 0.48s
LongLst: 0.60s
ThreadPool: 0.37s (Not affected by LongLst)
NumWorkers0: 0.13s (Not affected by LongLst)

I will plan to run a full benchmark with ThreadPool and NumWorkers0 settings to see how they compare to the current settings.

[jwmueller]

Interesting... after delving into it a bit, I also found the DataLoader behaves in counterintuitive ways. Most of the public MXNet material regarding DataLoader practices is with regards to datasets stored on-disk rather than in-memory like the ArrayDataset we operate on. Older MXNet models used to usually use DataIterator rather than DataLoader, but this documentation specifically recommends DataLoader over DataIterator:

https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/data/datasets.html#Appendix:-Upgrading-from-Module-DataIter-to-Gluon-DataLoader

It seems we may end up in the strange situation where the NN could plausibly run at optimal performance using num_workers = 0 rather than > 0, and using mxnet (single-thread) rather than mxnet-mkl (multi-thread)...

[Innixma]

Benchmark Results:

                               framework  time_train_s  metric_error  time_infer_s       acc       auc   logloss  bestdiff  loss_rescaled      rank  rank=1_count  rank=2_count  rank=3_count  rank>3_count  error_count
0    autogluon_ag_nn_nostack_numwork0_4h    283.258974      0.330012      1.147583  0.831875  0.849475  0.562289  0.003097       0.025641  1.923077             4            34             1             0            0
1             autogluon_ag_nn_nostack_4h    244.100000      0.329844      1.204280  0.832035  0.849426  0.561840  0.002009       0.066691  2.025641             0            38             1             0            0
2  autogluon_ag_nn_nostack_threadpool_4h    883.158974      0.329747      1.817375  0.832004  0.849349  0.561519  0.001360       0.083982  2.051282             1            35             3             0            0

This suggests to use the current setting during training, and numwork0 for inference, or perhaps numwork0 for both if we want to lean towards memory stability. thread_pool=True should be avoided, at least for m5.2xlarge instances.

[Innixma]

I tested with an additional configuration (called _mixed in the benchmark): training set is kept as is, but validation set and test sets for inference use numworkers = 0. This achieves the best result with no downsides compared to the other methods:

                               framework  time_train_s  metric_error  time_infer_s       acc       auc   logloss  bestdiff  loss_rescaled      rank  rank=1_count  rank=2_count  rank=3_count  rank>3_count  error_count
0    autogluon_ag_nn_nostack_numwork0_4h    290.444737      0.338697      1.174810  0.827697  0.842307  0.562289  0.003178       0.026316  2.381579             4             0            33             1            0
1             autogluon_ag_nn_nostack_4h    250.284211      0.338524      1.229560  0.827861  0.842256  0.561840  0.002062       0.054820  2.513158             0             2            36             0            0
2       autogluon_ag_nn_nostack_mixed_4h    243.205263      0.338736      1.173114  0.827857  0.842221  0.562271  0.002979       0.089021  2.539474             0             2            35             1            1
3  autogluon_ag_nn_nostack_threadpool_4h    905.784211      0.338425      1.859594  0.827830  0.842176  0.561519  0.001396       0.082441  2.565789             1             1            33             3            0

Note that the inference speedup is much more significant than this data suggests, as the times are dominated by the large datasets. Most of the smaller datasets see upwards of a 400% inference speedup because of this change, and likely an even larger % improvement for online inference.

[Innixma]

Please see #422 for the proposed solution.