Ignite: How do I attach both train and validation metrics to evaluator engine?

[milongo]

Hi,
I'm using Comet-ml to do hyperparameter search. Comet-ml basically asks for a 'fit' function that registers a final training loss. But it also allows for multiple metric logging which I would like to do.
So I'm trying to get my evaluator engine to hold train and validation metrics. Something like this:

@trainer.on(Events.COMPLETED)
def register_final_loss(engine):
     evaluator.run(train_loader)
     train_metrics = evaluator.state.metrics
     train_avg_mse = train_metrics['mse']
     train_avg_nll = train_metrics['nll']
     evaluator.run(val_loader)
     val_metrics = evaluator.state.metrics
     val_avg_mse = val_metrics['mse']
     val_avg_nll = val_metrics['nll']

But I don't know if event handlers can return things. After running an epoch I see the evaluator engine's state holds the last metrics computed, but the above intermediate variables are lost. Can I return them? If so, where would I call register_final_loss?

Thanks!

[vfdev-5]

Hi @milongo
Maybe you can store them into another state's attribute:

@trainer.on(Events.STARTED)
def init_train_val_metrics(_):
    trainer.state.train_val_metrics = {
        "train": None,
        "val": None
    }

@trainer.on(Events.COMPLETED)
def register_final_loss(engine):
     evaluator.run(train_loader)
     train_metrics = evaluator.state.metrics
     trainer.state.train_val_metrics["train"] = train_metrics
     evaluator.run(val_loader)
     val_metrics = evaluator.state.metrics
     trainer.state.train_val_metrics["val"] = val_metrics

Otherwise, maybe you can directly log them to Comet-ML (based on this : https://pytorch.org/ignite/concepts.html#events-and-handlers and https://www.comet.ml/docs/python-sdk/pytorch/):

def log_metrics(evaluator, title):
    for name, value in evaluator.state.metrics.items():
        experiment.log_metric("{}_{}".format(title, name), value)

@trainer.on(Events.EPOCH_COMPLETED)
def evaluate(trainer):
    with experiment.test():
        with evaluator.add_event_handler(Events.COMPLETED, log_metrics, "train"):
             evaluator.run(train_loader)

        with evaluator.add_event_handler(Events.COMPLETED, log_metrics, "validation"):
            evaluator.run(validation_loader)

HTH

[vfdev-5]

@milongo can we close the issue as resolved ?

[milongo]

Yes! Even though I did not use the above solution, I ended up just logging them directly to comet doing

 with experiment.test():
            experiment.log_metric('mse', avg_mse, epoch=engine.state.epoch)
            experiment.log_metric('nll', avg_nll, epoch=engine.state.epoch)

and passing experiment as argument to a run function.