Ignite: Accumulate output of .run()

[andreydung]

Thanks for the useful library.

Currently for validation I'm implementing the following for loop:

accumulator_eval = Evaluator()
for data, target in val_loader:
    score = model(data)
    accumulator_eval.insert(score, target)

metrics = accumulator_eval.get_metrics()
df_pred = accumulator_eval.get_prediction()

Evaluator is a custom class which receives pred/target in batches, accumulate them and then perform metric calculations and generate predictions as pandas dataframe.

I'm wondering how could I use ignite to represent this? My current approach is:

trainer = Engine(train_model)
tester = Engine(eval_model)

@trainer.on(Events.ITERATION_COMPLETED)
def log_testing(engine):
    n_iteration = engine.state.iteration

    if n_iteration % args.test_interval == 0:
        accumulator_eval = Evaluation(columns=COLUMNS)
        tester.run(val_loader)

I wasn't able to figure out how to pass accumulator_eval to the run loop, and how to get the output from them. Is attach() being the right approach?

Or would it be better to inherit Evaluator from Metric()? Note that for Evaluator, only y_pred and yis not enough as it also needs softmax proba, original data for generating predictions.

[vfdev-5]

@andreydung thanks for the feedback!

Your current implementation can be ported to ignite in several ways:

1) Rewrite your custom Evaluator as ignite.metrics.Metric.
If I understand correctly, your custom Evaluator class is a sort of metric computation class. In ignite we have Metric abstraction for that and it works as following: a) reset internal accumulator state, b) update((y_pred, y)) internal state of the metric class at each iteration, b) compute() calculates the final accumulated metric.
For example: Accuracy metric essentially cumulates _num_correct and _num_examples that are used to provide final accuracy as _num_correct / _num_examples.

So, you may want to wrap your Evaluator class or write a new one inheriting from ignite.metrics.Metric and just implement 3 methods:

• reset
• update
• compute

Once it is done, you can use it as in most of our examples when an evaluator engine computes metrics.
For example, https://github.com/pytorch/ignite/blob/3ffb81c55822dec14c8d04acff33b112b86caadd/examples/mnist/mnist.py#L57-L60

2) Use Evaluator as it is

If you want to keep your Evaluator class as it is without recoding, you can use as following inside ignite :

trainer = Engine(train_model)
tester = Engine(eval_model)

@trainer.on(Events.ITERATION_COMPLETED)
def log_testing(engine):
    n_iteration = engine.state.iteration - 1
    if n_iteration % args.test_interval == 0:
        accumulator_eval = Evaluation(columns=COLUMNS)
        tester.run(val_loader)

@tester.on(Events.ITERATION_COMPLETED)
def update_evaluator(engine):
    score, target = engine.state.output  # this is the output of  `eval_model` function
    accumulator_eval.insert(score, target)

@tester.on(Events.COMPLETED)
def compute_metrics(engine):
    metrics = accumulator_eval.get_metrics()
    df_pred = accumulator_eval.get_prediction()
    # now do whatever you would like with computed metrics

HTH

[vfdev-5]

@andreydung I close the issue as answered. Feel free to reopen if you still need a support this.

[andreydung]

Thanks! Just a quick question, how to obtain training n_iteration inside compute_metrics?

[vfdev-5]

@andreydung this can be done directly if compute_metrics is in the same scope as trainer :

trainer = Engine(train_model)

@tester.on(Events.COMPLETED)
def compute_metrics(engine):
    metrics = accumulator_eval.get_metrics()
    df_pred = accumulator_eval.get_prediction()
    # now do whatever you would like with computed metrics
    global_step = trainer.state.iteration

Otherwise, you can pass trainer as an argument to the handler. See here: https://pytorch.org/ignite/v0.2.0/engine.html#ignite.engine.Engine.add_event_handler