Ignite: LR scheduling with Nvidia/APEX

[vfdev-5]

Probably, since https://github.com/NVIDIA/apex/pull/310 optimizer's param group is replaced and ignite's contrib lr schedulling is not applied to the actual param group.

lr_scheduler = PiecewiseLinear(optimizer, param_name='lr', milestones_values=milestones_values)
len(optimizer.param_groups), id(optimizer.param_groups), id(optimizer.param_groups[0])
> (1, 139701776927624, 139701613652440)

from apex import amp
model, optimizer = amp.initialize(model, optimizer, opt_level="O2", num_losses=1)

len(optimizer.param_groups), id(optimizer.param_groups), id(optimizer.param_groups[0])
> (1, 139701467050824, 139701465940400)

id(lr_scheduler.optimizer_param_groups)
> 139701776927624

[vfdev-5]

The problem is the following:

1) if lr_scheduler is setup on the original optimizer and which is then patched with Apex, using opt_level="O2", this case wont work

2) if lr_scheduler is setup on the patched optimizer using any type of opt_level, this case should work correctly

3) if lr_scheduler is setup on the original optimizer and which is then patched with Apex, using opt_level="O1", this case should work correctly

[vfdev-5]

Fixed by #690