Ignite: `ignite.metrics.Metric` for all use-cases: Expand Metric's arguments by attachment events

[DrStoop]

Issue

ignite.metrics.Metric currently only supports the calculation (and therefore also the visualization/tensorboard logging) of metrics at each completed epoch. In regular use-cases, e.g. to determine/visualize the optimum iteration for early stopping, it is required to return the loss metric after each iteration or at a customized event. This is currently always related to overriding Metric.attach(). So currently I experienced the number of use-cases to be very limited without overriding the source code all the time.

Solution

Add all events used to attach the methods of Metric to the arguments with default values referring to the current attachment events. The default values of the added event arguments will guarantee the backward compatibility of existing code and keep boilerplate code at the same level for "epoch-metrics". For any other desired/required use-case, the event arguments will provide updating, calculating and outputting the metric to engine.metrics at any desired/required event.
This would of course require modifying all inheriting classes of Metric (e.g. Loss, Accuracy, LambdaMetric and their base classes) but the modifications are simple and straight forward.

Code suggestion

Here a possible implementation:

from abc import ABCMeta, abstractmethod
from ignite._six import with_metaclass
from ignite.engine import Events
import torch


class Metric(with_metaclass(ABCMeta, object)):
    """
    Base class for all Metrics.

    Args:
        output_transform (callable, optional): a callable that is used to transform the
            :class:`~ignite.engine.Engine`'s `process_function`'s output into the
            form expected by the metric. This can be useful if, for example, you have a multi-output model and
            you want to compute the metric with respect to one of the outputs.
        started_event (<enum Events>): event from which on the metric should be calculated
        iteration_completed_event (<enum Events>): event at which intermediate metric calculations (`self.update()`)
            are executed from current model outputs, e.g. for a mean loss metric this would refer to adding
            the current model loss output after each iteration to a summed loss and increasing to count of losses added.
        completed_event (<enum Events>): event at which the metric value us calculated and written to
            `engine.state.metrics[metric_name]`. E.g. for mean loss metric calculation the summed output losses
            of each iterations is devided by the number of iterations and outputed to `trainer.state.metrics['loss'].
    """

    def __init__(self, output_transform=lambda x: x, started_event=Events.EPOCH_STARTED,
                 iteration_completed_event=Events.ITERATION_COMPLETED, completed_event=Events.EPOCH_COMPLETED):
        self._output_transform = output_transform
        self.started_event = started_event
        self.iteration_completed_event = iteration_completed_event
        self.completed_event = completed_event
        self.reset()

    @abstractmethod
    def reset(self):
        """
        Resets the metric to it's initial state.

        This is called at the start of each epoch.
        """
        pass

    @abstractmethod
    def update(self, output):
        """
        Updates the metric's state using the passed batch output.

        This is called once for each batch.

        Args:
            output: the is the output from the engine's process function.
        """
        pass

    @abstractmethod
    def compute(self):
        """
        Computes the metric based on it's accumulated state.

        This is called at the end of each epoch.

        Returns:
            Any: the actual quantity of interest.

        Raises:
            NotComputableError: raised when the metric cannot be computed.
        """
        pass

    def started(self, engine):
        self.reset()

    @torch.no_grad()
    def iteration_completed(self, engine):
        output = self._output_transform(engine.state.output)
        self.update(output)

    def completed(self, engine, name):
        result = self.compute()
        if torch.is_tensor(result) and len(result.shape) == 0:
            result = result.item()
        engine.state.metrics[name] = result

    def attach(self, engine, name):
        if not engine.has_event_handler(self.started, self.started_event):
            engine.add_event_handler(self.started_event, self.started)
        if not engine.has_event_handler(self.iteration_completed, self.iteration_completed_event):
            engine.add_event_handler(self.iteration_completed_event, self.iteration_completed)
        # `self.completed()` is always executed at the end, so it should be added to `engine` at the end
        engine.add_event_handler(self.completed_event, self.completed, name)

    def __add__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x + y, self, other)

    def __radd__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x + y, other, self)

    def __sub__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x - y, self, other)

    def __rsub__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x - y, other, self)

    def __mul__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x * y, self, other)

    def __rmul__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x * y, other, self)

    def __pow__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x ** y, self, other)

    def __rpow__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x ** y, other, self)

    def __mod__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x % y, self, other)

    def __div__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__div__(y), self, other)

    def __rdiv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__div__(y), other, self)

    def __truediv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__truediv__(y), self, other)

    def __rtruediv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x.__truediv__(y), other, self)

    def __floordiv__(self, other):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x, y: x // y, self, other)

    def __getattr__(self, attr):
        from ignite.metrics import MetricsLambda

        def fn(x, *args, **kwargs):
            return getattr(x, attr)(*args, **kwargs)

        def wrapper(*args, **kwargs):
            return MetricsLambda(fn, self, *args, **kwargs)
        return wrapper

    def __getitem__(self, index):
        from ignite.metrics import MetricsLambda
        return MetricsLambda(lambda x: x[index], self)

[vfdev-5]

@DrStoop thanks for the issue, definitely make sense to add this feature !

I was thinking to add such custom events to attach method as it is done here :
https://github.com/pytorch/ignite/blob/f5aad5f6d764cf5c131c953830d305c4d71dd1c6/ignite/contrib/handlers/tqdm_logger.py#L137-L140

Anyway, please send a PR and we could iterate on the code.

[DrStoop]

Hi @vfdev-5,

yes, me too was exactly between your idea with attach() and __init__(). I went for __init__() for the following reasons:

• The metric is strongly defined by the events at which it is started, updated and computed, meaning you have to answer these questions when you initialize it. Also for debugging it is important to have the events together with the other metric parameters in one line. As I am working with this code already, here an example how a more complex usecase can look like:

    def attach_metrics_to_engine(engine, metrics):
        for name, metric in metrics.items():
            metric.attach(engine=engine, name=name)

    metrics = {
        'trainer': {'lc_loss': Output(completed_event=Events.ITERATION_COMPLETED),
                    'lc_loss_running_average': RunningAverage(alpha=intern_params['alpha_running_average'],
                                                              output_transform=lambda x: x),
                    AverageOutput(started_event=optimizer_step_event.conditional_started_event,
                                  completed_event=optimizer_step_event.conditional_completed_event)},
        'xvalidator': {'lc_loss': AverageOutput(output_transform=linear_combination_loss_from_infer,
                                                completed_event=Events.ITERATION_COMPLETED),
                       'lm_accuracy': Accuracy(output_transform=lm_logits_and_labels_from_infer,
                                               completed_event=Events.ITERATION_COMPLETED),
                       'mc_accuracy': Accuracy(output_transform=mc_logits_and_labels_from_infer,
                                               completed_event=Events.ITERATION_COMPLETED)},
        'evaluator': {'lc_loss': AverageOutput(output_transform=linear_combination_loss_from_infer,
                                               completed_event=Events.ITERATION_COMPLETED),
                      'lm_accuracy': Accuracy(output_transform=lm_logits_and_labels_from_infer),
                      'mc_accuracy': Accuracy(output_transform=mc_logits_and_labels_from_infer)}

    # Attach all metric
    for engine, engine_name in zip([trainer, xvalidator, evaluator], ['trainer', 'xvalidator', 'evaluator']):
        attach_metrics_to_engine(engine=engine, metrics=metrics[engine_name])

• In almost all use-cases you should not attach one instance to multiple different events (and engines). Even if you could it is important for avoiding subsequent errors and for clarity to instantiate multiple Metrics for different metrics. E.g. metrics that update and compute on accumulated model outputs over time will go crazy when attached to multiple events or engines. Also when trying to tensorboard-log the different metrics - all held in this one instance - later on, this will be almost impossible. So suggesting the user the option to attach one metric-instance to multiple events means that this metric instance suddenly represents different metrics with different values and bring her/him into trouble.

So these are the main reasons why I suggest adding the events as argument to __init__() instead of to attach(). There were some more I first would have to remember...

I actually already implemented and use this code for different metrics from this proposed Metric in projects and so far everything runs smoothly. Any use-cases I could thing of were covered and compatible with the existing features (engines, TensorboardLogger, etc.).

I can also provide further inheriting metrics, e.g. Accuracy, Loss, RunningAverage, AverageOuput, etc.. And I would suppose modifying the whole package to this change is a matter of few hours.

[vfdev-5]

@DrStoop thanks for the explanation, yes I agree with your suggestions - let's setup events in __init__.

I can also provide further inheriting metrics, e.g. Accuracy, Loss, RunningAverage, AverageOuput, etc.. And I would suppose modifying the whole package to this change is a matter of few hours.

yes, a PR please :)

[DrStoop]

coming up... :)