Autogluon: About EfficientNet and FIFOScheduler

[XWalways]

1.About search EfficientNet(width_coefficient, depth_coefficient) ,report error:'NoneType' object has no attribute 'initialize',maybe AutoGluonObject can't initialize directly?

2.FIFOScheduler seems can't run all trials, for example: num_trials=20，but it just run 3 trials, then shutdown

[zhanghang1989]

1.About search EfficientNet(width_coefficient, depth_coefficient) ,report error:'NoneType' object has no attribute 'initialize',maybe AutoGluonObject can't initialize directly?

2.FIFOScheduler seems can't run all trials, for example: num_trials=20，but it just run 3 trials, then shutdown

Regarding 2, what is the time_out value did you use? If time_out, no further trials are scheduled.

[XWalways]

1.About search EfficientNet(width_coefficient, depth_coefficient) ,report error:'NoneType' object has no attribute 'initialize',maybe AutoGluonObject can't initialize directly?
2.FIFOScheduler seems can't run all trials, for example: num_trials=20，but it just run 3 trials, then shutdown

Regarding 2, what is the time_out value did you use? If time_out, no further trials are scheduled.

i also notice that ,but time_out is None,i don't set time_out

[zhanghang1989]

1.About search EfficientNet(width_coefficient, depth_coefficient) ,report error:'NoneType' object has no attribute 'initialize',maybe AutoGluonObject can't initialize directly?
2.FIFOScheduler seems can't run all trials, for example: num_trials=20，but it just run 3 trials, then shutdown

Regarding 2, what is the time_out value did you use? If time_out, no further trials are scheduled.

i also notice that ,but time_out is None,i don't set time_out

Thanks for the feedback. Could you provide the code to reproduce it?

[XWalways]

1.About search EfficientNet(width_coefficient, depth_coefficient) ,report error:'NoneType' object has no attribute 'initialize',maybe AutoGluonObject can't initialize directly?
2.FIFOScheduler seems can't run all trials, for example: num_trials=20，but it just run 3 trials, then shutdown

Regarding 2, what is the time_out value did you use? If time_out, no further trials are scheduled.

i also notice that ,but time_out is None,i don't set time_out

Thanks for the feedback. Could you provide the code to reproduce it?

https://github.com/zhanghang1989/AutoGluon-Tutorial-CVPR2020/blob/master/6.pytorch_hpo.ipynb

try larger num_trials, it will shutdown early

[zhanghang1989]

1.About search EfficientNet(width_coefficient, depth_coefficient) ,report error:'NoneType' object has no attribute 'initialize',maybe AutoGluonObject can't initialize directly?
2.FIFOScheduler seems can't run all trials, for example: num_trials=20，but it just run 3 trials, then shutdown

Regarding 2, what is the time_out value did you use? If time_out, no further trials are scheduled.

i also notice that ,but time_out is None,i don't set time_out

Thanks for the feedback. Could you provide the code to reproduce it?

https://github.com/zhanghang1989/AutoGluon-Tutorial-CVPR2020/blob/master/6.pytorch_hpo.ipynb

try larger num_trials, it will shutdown early

What platform/operating system are you using?

[XWalways]

1.About search EfficientNet(width_coefficient, depth_coefficient) ,report error:'NoneType' object has no attribute 'initialize',maybe AutoGluonObject can't initialize directly?
2.FIFOScheduler seems can't run all trials, for example: num_trials=20，but it just run 3 trials, then shutdown

Regarding 2, what is the time_out value did you use? If time_out, no further trials are scheduled.

i also notice that ,but time_out is None,i don't set time_out

Thanks for the feedback. Could you provide the code to reproduce it?

https://github.com/zhanghang1989/AutoGluon-Tutorial-CVPR2020/blob/master/6.pytorch_hpo.ipynb
try larger num_trials, it will shutdown early

What platform/operating system are you using?
Ubuntu 16.04+cuda10.1+mxnet-cu101mkl-1.5.1.post0+gluoncv-0.5.0+torch-1.3.1

[XWalways]

import os
from tqdm import tqdm
import logging

import mxnet as mx
from mxnet import gluon
from mxnet.gluon.data.vision import transforms

from gluoncv.model_zoo import get_model
from gluoncv.utils import LRScheduler
from gluoncv.data import transforms as gcv_transforms

import autogluon as ag
from autogluon.model_zoo import *
from autogluon.core import Choice
from autogluon.utils import EasyDict

block_args = []
@ag.autogluon_function()
def mobilenet_block_args(kernel, num_repeat, channels, expand_ratio,
                         stride, se_ratio=0.25, in_channels=0):
    return EasyDict(kernel=kernel, num_repeat=num_repeat,
                    channels=channels, expand_ratio=expand_ratio,
                    stride=stride, se_ratio=se_ratio, in_channels=in_channels)

strides = [1, 2, 2]
for i in range(3):
    block_args.append(
        mobilenet_block_args(
        stride=strides[i],
        kernel=Choice(3, 5),
        num_repeat=Choice(1, 2, 3, 4),
        channels=Choice(*[32*i for i in range(1, 6)]),
        expand_ratio=Choice(1, 6)))
print(block_args)
@ag.autogluon_register_args(
        block_args=ag.space.List(block_args[0], block_args[1], block_args[2]),
        batch_size=64,
        num_gpus=8,
        num_workers=30,
        lr=1e-1,
        momentum=0.9,
        wd=0.0005,
        epochs=10
    )

def nas_train(args, reporter):
    net = EfficientNet(args.block_args, dropout_rate=0.2, num_classes=10, width_coefficient=1,
                       depth_coefficient=1, depth_divisor=8, min_depth=None, drop_connect_rate=0.2,
                       input_size=32)
    batch_size = args.batch_size
    num_gpus = args.num_gpus

    batch_size *= max(1, num_gpus)
    context = [mx.gpu(i) for i in range(num_gpus)] if num_gpus > 0 else [mx.cpu()]
    num_workers = args.num_workers

    transform_train = transforms.Compose([
        gcv_transforms.RandomCrop(32, pad=4),
        transforms.RandomFlipLeftRight(),
        transforms.ToTensor(),
        transforms.Normalize([0.4914, 0.4822, 0.4465], [0.2023, 0.1994, 0.2010])
    ])

    transform_test = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize([0.4914, 0.4822, 0.4465], [0.2023, 0.1994, 0.2010])
    ])
    train_data = gluon.data.DataLoader(
        gluon.data.vision.CIFAR10(train=True).transform_first(transform_train),
        batch_size=batch_size, shuffle=True, last_batch='discard', num_workers=num_workers)

    val_data = gluon.data.DataLoader(
        gluon.data.vision.CIFAR10(train=False).transform_first(transform_test),
        batch_size=batch_size, shuffle=False, num_workers=num_workers)

    def test(ctx, val_data):
        metric = mx.metric.Accuracy()
        for i, batch in enumerate(val_data):
            data = gluon.utils.split_and_load(batch[0], ctx_list=ctx, batch_axis=0)
            label = gluon.utils.split_and_load(batch[1], ctx_list=ctx, batch_axis=0)
            outputs = [net(X) for X in data]
            metric.update(label, outputs)
        return metric.get()

    def train(epochs, ctx):
        if isinstance(ctx, mx.Context):
            ctx = [ctx]
        net.initialize(mx.init.Xavier(), ctx=ctx)


        lr_scheduler = LRScheduler(mode='cosine', base_lr=args.lr,
                                   nepochs=args.epochs,
                                   iters_per_epoch=len(train_data))
        trainer = gluon.Trainer(net.collect_params(), 'sgd',
                                {'lr_scheduler': lr_scheduler, 'wd': args.wd, 'momentum': args.momentum})
        metric = mx.metric.Accuracy()
        train_metric = mx.metric.Accuracy()
        loss_fn = gluon.loss.SoftmaxCrossEntropyLoss()

        iteration = 0
        best_val_score = 0

        tbar = tqdm(range(epochs))
        for epoch in tbar:
            train_metric.reset()
            metric.reset()
            train_loss = 0
            num_batch = len(train_data)
            alpha = 1

            for i, batch in tqdm(enumerate(train_data)):
                data = gluon.utils.split_and_load(batch[0], ctx_list=ctx, batch_axis=0)
                label = gluon.utils.split_and_load(batch[1], ctx_list=ctx, batch_axis=0)

                with mx.autograd.record():
                    output = [net(X) for X in data]
                    loss = [loss_fn(yhat, y) for yhat, y in zip(output, label)]
                for l in loss:
                    l.backward()
                trainer.step(batch_size)
                train_loss += sum([l.sum().asscalar() for l in loss])

                train_metric.update(label, output)
                name, acc = train_metric.get()
                iteration += 1

            train_loss /= batch_size * num_batch
            name, acc = train_metric.get()
            name, val_acc = test(ctx, val_data)
            if reporter:
                reporter(epoch=epoch, accuracy=val_acc)
            else:
                net.save_parameters('final_models.params')
                tbar.set_description("epoch={epoch}, accuracy={accuracy}".format(epoch=epoch, accuracy=val_acc))

        if not reporter:
            net.save_parameters('final_models.params')

    train(args.epochs, context)

if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG)

    searcher = 'random'
    num_trials = 2
    args = nas_train.args
    if searcher == 'random':
        searcher = ag.searcher.RandomSampling(nas_train.cs)
        myscheduler = ag.scheduler.FIFOScheduler(nas_train, args,
                                                              {'num_cpus': 30, 'num_gpus': args.num_gpus}, searcher,
                                                              num_trials=num_trials,
                                                              checkpoint='./exp/checkerpoint.ag',
                                                              reward_attr="accuracy")
    elif searcher == 'grid':
        myscheduler = ag.scheduler.FIFOScheduler(nas_train, args,
                                                 resource={'num_cpus': 30, 'num_gpus': args.num_gpus},
                                                 searcher=searcher,
                                                 num_trials=20,
                                                 checkpoint='./grid_exp/checkerpoint.ag',
                                                 reward_attr="accuracy")
    else:
        raise NotImplemented

    # run experiments and get results
    myscheduler.run()
    myscheduler.join_jobs()
    myscheduler.get_training_curves(plot=True,use_legend=False)
    print('The Best Configuration and Accuracy are: {}, {}'.format(myscheduler.get_best_config(),
                                                                   myscheduler.get_best_reward()))
    #myscheduler.shutdown()
    ag.done()

in this code, FIFOScheduler can work well, but there is a little bug, while using grid_search, num_trials should be Not required, maybe 0.0.1 is better

[IsaacCheng1]

I encountered the same problem: FIFOScheduler does not run all trials=50, but stops at 2nd trial.
And I fixed this problem by setting max_reward to a large number, such as max_reward=1000.

As far as my understanding, FIFOScheduler will not schedule next task if the reporter reports an accuracy/reward larger than max_reward.
https://github.com/awslabs/autogluon/blob/master/autogluon/scheduler/fifo.py#L144

[jwmueller]

@zhanghang1989 Can we remove the default value max_reward = 1.0 in scheduler?
This will trigger unexpected behavior in many applications. Default should probably be Inf (or None).

[zhanghang1989]

I agree. This causes troubles.