Autogluon: error in disrtribued fifo scheduler

[aaronkl]

If I run the distributed fifo scheduler across multiple instances I get the following error

autogluon.protocol.core - CRITICAL - Failed to deserialize 
Traceback (most recent call last): 
File "/usr/local/lib/python3.6/site-packages/distributed/protocol/core.py", line 106, in loads header = msgpack.loads(header, use_list=False, **msgpack_opts)
File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
ValueError: tuple is not allowed for map key
autogluon.utils_comm - INFO - Got an unexpected error while collecting from workers: tuple is not allowed for map key

It seem to be related to this issue
Is there a reason why we fix distributed==2.6.0 ?

[Innixma]

Is this for Tabular or is this generic? Tabular doesn't support distributed at the moment.

Could you provide a code sample that leads to this error?

[aaronkl]

It's the generic case. I will try to create a script that reproduces this behavior, but that might be a bit tricky with my distributed setup.

[mseeger]

Hi Aaron, this reminds me to something I had a while ago, in a different context.

JSON does not support "tuple" as datatype, you need to convert it to "list". Maybe the key is done by JSON in some way?

[mseeger]

Is there a stack trace, so we understand what part of the AutoGluon code is causing this?
For example, which method in FIFOScheduler?

[mseeger]

BTW: I am confused. I grep for any of these error messages, or parts of them, and I do not find anything! ??

[aaronkl]

the following code snippet should reproduce the error:

import autogluon as ag
import numpy as np

dist_ip_addrs = # add list of worker IP addresses here

@ag.args(p=ag.Real(0, 1),
         epochs=10,)
def train_fn(args, reporter, **kwargs):

   for e in range(args.epochs):
        epoch = e + 1
        r = np.random.binomial(1, p=args.p)

        reporter(epoch=epoch, result=r)

scheduler = ag.scheduler.FIFOScheduler(train_fn,
                                       dist_ip_addrs=dist_ip_addrs,
                                       resource={'num_cpus': 2, 'num_gpus': 0},
                                       num_trials=100,
                                       reward_attr='result',
                                       time_attr='epoch')
scheduler.run()
scheduler.join_jobs()

the versions of the relevant dependencies are:

asn1crypto 0.24.0
attrs 19.3.0
awscli 1.18.13
boto3 1.9.176
botocore 1.12.176
catboost 0.22
ConfigSpace 0.4.10
Cython 0.29.13
dask 2.6.0
decorator 4.4.2
distributed 2.6.0
docutils 0.14
gluoncv 0.6.0
gluonnlp 0.8.1
graphviz 0.8.4
mxnet 1.6.0
mxnet-mkl 1.4.1
numpy 1.18.1
onnx 1.4.1
terminado 0.8.3

let me know if you require further information

[zhanghang1989]

Hi @aaronkl , Thanks for reporting the issue!
Looks like maybe a bug recent introduced in master branch, or some system setting error.

I have created a AWS public AMI with id of ami-02bb25a14053a257b in US West (Oregon)us-west-2 zone with stable autogluon version.

You can start new p3.2xlarge instances and try it.

I have just tested both your code and the distributed tutorial. Both code works on that environment.

[aaronkl]

ok it's seems the problem went away with latest changes in the master branch