Autogluon: Pickling error with custom metrics

[tigerhawkvok]

For this code, bog standard except for a custom metric:

def median_absolute_percentage_error_raw(y_true, y_pred, multioutput='uniform_average'):
    import numpy as np
    output_errors = np.median(np.abs((y_pred - y_true) / y_true), axis=0)
    if isinstance(multioutput, str):
        if multioutput == 'raw_values':
            return output_errors
        elif multioutput == 'uniform_average':
            # pass None as weights to np.average: uniform mean
            multioutput = None
    return np.average(output_errors, weights=multioutput)

from autogluon.utils.tabular.metrics import make_scorer
median_absolute_percentage_error = make_scorer('median_absolute_percentage_error', median_absolute_percentage_error_raw, optimum=0, greater_is_better= False)

from autogluon import TabularPrediction as task
predictor = task.fit(train_data= task.Dataset(file_path= "myAutogluonTrainingSet.csv"), label= "myColumn", eval_metric= median_absolute_percentage_error, hyperparameter_tune= True)

Generating the fit seems to hang partway in, and shows no further progress

Num of Finished Tasks is 0
Num of Pending Tasks is 1000
  0%|                                                                                          | 0/1000 [00:00<?, ?it/s]Process Process-34:
Traceback (most recent call last):
  File "/home/pkahn/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/pkahn/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/scheduler/scheduler.py", line 125, in _worker
    ret = fn(**kwargs)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/core/decorator.py", line 58, in __call__
    output = self.f(args, **kwargs)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/core/decorator.py", line 139, in wrapper_call
    return func(*args, **kwargs)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/utils/tabular/ml/models/lgb/hyperparameters/lgb_trial.py", line 82, in lgb_trial
    trial_model_file = lgb_model.save(file_prefix=file_prefix, directory=directory, return_filename=True)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/utils/tabular/ml/models/abstract/abstract_model.py", line 166, in save
    save_pkl.save(path=file_name, object=self, verbose=verbose)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/utils/tabular/utils/savers/save_pkl.py", line 11, in save
    save_with_fn(path, object, pickle_fn, format=format, verbose=verbose)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/utils/tabular/utils/savers/save_pkl.py", line 24, in save_with_fn
    pickle_fn(object, fout)
  File "/home/pkahn/anaconda3/lib/python3.6/site-packages/autogluon/utils/tabular/utils/savers/save_pkl.py", line 10, in <lambda>
    pickle_fn = lambda o, buffer: pickle.dump(o, buffer, protocol=4)
_pickle.PicklingError: Can't pickle <function median_absolute_percentage_error_raw at 0x7f4f5af24378>: it's not the same object as __main__.median_absolute_percentage_error_raw

[Innixma]

Thanks for submitting the issue, we will take a look!

[tigerhawkvok]

The initial comment was via WSL and an anaconda dist, and I just confirmed it on Ubuntu 19.10 with a core python dist:

[larroy]

I spent some time looking into this but not able to find a fix yet. Root cause seems that the function is serialized along while the model while saving, but for the lgb model this doesn't work.

A small reproduction of a similar issue, is when redefining a function which has been captured by a class that is being pickled. But I didn't see the function mediand_absolute_percentage_error_raw here being redefined in the repro.

import pickle
def f():
    pass

#class D(object):
#    pass

class C(object):
    pass


if  __name__ == '__main__':
 #   d = D()
    c = C()
  #  d.f = f
    c.f = f
    def f():
        pass

  #  c.f = d.f
    s=pickle.dumps(c)

https://github.com/awslabs/autogluon/blob/master/autogluon/utils/tabular/ml/models/lgb/hyperparameters/lgb_trial.py#L22

[larroy]

I checked with the debugger, and for GBM seems the function is indeed redefined or has a different ID. I suspect it could be multiprocessing.

[larroy]

I would suggest to run fit with hyperparameter_tune=False I don't observe the crash with this, as it doesn't trigger multiprocessing. We will rework how we do multiprocessing and distributed training for hyperparameter tunning in the future.

[Innixma]

@tigerhawkvok

I've found a fix for this issue.

If you move your initialization of the custom metric outside of __main__, it will work.

For example:

from autogluon.utils.tabular.sandbox.ames.custom_metric import median_absolute_percentage_error

With the contents of custom_metric.py being:

import numpy as np
from autogluon.utils.tabular.metrics import make_scorer


def median_absolute_percentage_error_raw(y_true, y_pred, multioutput='uniform_average'):
    output_errors = np.median(np.abs((y_pred - y_true) / y_true), axis=0)
    if isinstance(multioutput, str):
        if multioutput == 'raw_values':
            return output_errors
        elif multioutput == 'uniform_average':
            # pass None as weights to np.average: uniform mean
            multioutput = None
    return np.average(output_errors, weights=multioutput)

median_absolute_percentage_error = make_scorer('median_absolute_percentage_error', median_absolute_percentage_error_raw, optimum=0, greater_is_better=False)

As to why this fixes things, I am uncertain. It is likely something strange to do with Python and Pickle. I don't believe it is an AutoGluon issue.

Marking this as resolved. Feel free to re-open if you are still having issues!