Ignite: Filter ignite stacktrace

[elanmart]

Should we consider filtering out the frames that come from ignite? Whenever there's a bug in my code I have to go through a lot of frames to find where ignite trace ends, and my code starts.

I think it is possible to do in Python.

E.g. I'd like to turn

/Mine/ml-mine/code.../lm/train.py in main(train, dev, nn_type, ninp, nhid, nlayers, dropout, dropouth, dropouti, dropoute, wdrop, tie_weights, bsz, bptt, max_epochs, lr, clip, adaptive, cutoffs, device, to_device, ckpt_embedding, lock_emb, kernel_size, num_levels, seed)
    238 
    239     logger.debug(f'Invoking trainer.run for {max_epochs} epochs')
--> 240     return trainer.run(data=trn_dset, max_epochs=max_epochs)

.../ignite/ignite/engine/engine.py in run(self, data, max_epochs)
    264         except BaseException as e:
    265             self._logger.error("Engine run is terminating due to exception: %s", str(e))
--> 266             self._handle_exception(e)
    267 
    268         return self.state

.../ignite/ignite/engine/engine.py in _handle_exception(self, e)
    222     def _handle_exception(self, e):
    223         if Events.EXCEPTION_RAISED in self._event_handlers:
--> 224             self._fire_event(Events.EXCEPTION_RAISED, e)
    225         else:
    226             raise e

.../ignite/ignite/engine/engine.py in _fire_event(self, event_name, *event_args)
    182             self._logger.debug("firing handlers for event %s ", event_name)
    183             for func, args, kwargs in self._event_handlers[event_name]:
--> 184                 func(self, *(event_args + args), **kwargs)
    185 
    186     def terminate(self):

.../lm/train.py in handle_exception(engine, e)
    178 
    179         else:
--> 180             raise e
    181 
    182     @validator.process_function()

.../ignite/ignite/engine/engine.py in run(self, data, max_epochs)
    251                 self.state.epoch += 1
    252                 self._fire_event(Events.EPOCH_STARTED)
--> 253                 hours, mins, secs = self._run_once_on_dataset()
    254                 self._logger.info("Epoch[%s] Complete. Time taken: %02d:%02d:%02d", self.state.epoch, hours, mins, secs)
    255                 if self.should_terminate:

.../ignite/ignite/engine/engine.py in _run_once_on_dataset(self)
    213         except BaseException as e:
    214             self._logger.error("Current run is terminating due to exception: %s", str(e))
--> 215             self._handle_exception(e)
    216 
    217         time_taken = time.time() - start_time

.../ignite/ignite/engine/engine.py in _handle_exception(self, e)
    222     def _handle_exception(self, e):
    223         if Events.EXCEPTION_RAISED in self._event_handlers:
--> 224             self._fire_event(Events.EXCEPTION_RAISED, e)
    225         else:
    226             raise e

.../ignite/ignite/engine/engine.py in _fire_event(self, event_name, *event_args)
    182             self._logger.debug("firing handlers for event %s ", event_name)
    183             for func, args, kwargs in self._event_handlers[event_name]:
--> 184                 func(self, *(event_args + args), **kwargs)
    185 
    186     def terminate(self):

.../lm/train.py in handle_exception(engine, e)
    178 
    179         else:
--> 180             raise e
    181 
    182     @validator.process_function()

.../ignite/ignite/engine/engine.py in _run_once_on_dataset(self)
    205                 self.state.iteration += 1
    206                 self._fire_event(Events.ITERATION_STARTED)
--> 207                 self.state.output = self._process_function(self, batch)
    208                 self._fire_event(Events.ITERATION_COMPLETED)
    209                 if self.should_terminate or self.should_terminate_single_epoch:

.../lm/train.py in step(engine, batch)
    107         output  = output.view(-1, output.size(2))
    108 
--> 109         loss = loss_fn(output, y)
    110         loss.backward()
    111 

into

/Mine/ml-mine/code.../lm/train.py in main(train, dev, nn_type, ninp, nhid, nlayers, dropout, dropouth, dropouti, dropoute, wdrop, tie_weights, bsz, bptt, max_epochs, lr, clip, adaptive, cutoffs, device, to_device, ckpt_embedding, lock_emb, kernel_size, num_levels, seed)
    238 
    239     logger.debug(f'Invoking trainer.run for {max_epochs} epochs')
--> 240     return trainer.run(data=trn_dset, max_epochs=max_epochs)

<PYTORCH-IGNITE STACKTRACE SKIPPED. USE `ignite.set_stacktrace(True)`>

.../lm/train.py in step(engine, batch)
    107         output  = output.view(-1, output.size(2))
    108 
--> 109         loss = loss_fn(output, y)
    110         loss.backward()
    111 

[alykhantejani]

that would be amazing! I did not know this was possible in python - @elanmart want to have a go?

[elanmart]

Sure, I'll give it a try then!

[vfdev-5]

@elanmart any updates on this ?

[elanmart]

So it turns out you cannot alter the stack trace object. You can only print the filtered version.

This doesnt matter in most cases, the only one where it matters is jupyter plugin for folding stacktraces, which would no longer work properly.

I have the branch with this, let me try to finish it this weekend.