Ignite: TimeLimit handler for competitions

[sdesrozis]

🚀 Feature

What do you think to be able to terminate a run after a time of computation ? This would be useful for using ignite on runtime environments like Colab or Kaggle which limit the time of session.

An implementation of such handler is available here and can be reused : https://github.com/Minyus/pipelinex/blob/master/src/pipelinex/ops/ignite/handlers/time_limit.py

It misses documentation, tests, etc.

[alxlampe]

Your link seems to be broken. But I think I've already something like that in use. Not well formatted, without help but perhaps a (more or less) good starting point. It measures time in hours. I can implement it with tests but I have not much time at the moment. Could take me around 4 weeks :-|

import time

from ignite.engine import Events


class TimeConstraint:
    def __init__(self, time_limit=0, time_formatter=lambda x: "{:.4f}h".format(x)):
        if not isinstance(time_limit, float) or not isinstance(time_limit, int):
            ValueError("time_limit must be of type float or int, got {} instead.".format(type(time_limit)))

        self._time_limit = time_limit
        self._time_formatter = time_formatter

    def _start_timer(self, engine):
        self._t0 = time.time()

    def _check_time_limit(self, engine):
        train_time = ((time.time() - self._t0) / 3600)
        if self._time_limit > 0 and (train_time > self._time_limit):
            engine.terminate()

            print("TimeConstraint: Training is terminated.")
            print("TimeConstraint: Maximum training time of {:} reached. Current training time is {}.".format(
                self._time_formatter(self._time_limit), self._time_formatter(train_time)))

    def attach(self, engine):
        if self._time_limit > 0:
            engine.add_event_handler(Events.STARTED, self._start_timer)
            engine.add_event_handler(Events.EPOCH_COMPLETED, self._check_time_limit)

[vfdev-5]

Here is a correct link : https://github.com/Minyus/pipelinex/blob/ca64d78e34added4cd7ff19865aa724e71423394/src/pipelinex/extras/ops/ignite/handlers/time_limit.py

@Bibonaut yes, if you want to give a try on this, please go ahead. We have no deadline for this issue.

[guptadhaval18]

@vfdev-5 I would like to contribute.

Here is a correct link : https://github.com/Minyus/pipelinex/blob/ca64d78e34added4cd7ff19865aa724e71423394/src/pipelinex/extras/ops/ignite/handlers/time_limit.py

@Bibonaut yes, if you want to give a try on this, please go ahead. We have no deadline for this issue.

I may be wrong. But I think the code given in link is already solving the issue without any modification.

[sdesrozis]

@guptadhaval18 Thank you.

The code is correct but it has to be modified because we expect an handler (see https://pytorch.org/ignite/concepts.html#events-and-handlers).

Documentation and tests are needed too.

Tell me if you want to be assigned on that issue.

HTH

[guptadhaval18]

@sdesrozis so basically

• [x] the code has to be written in new file created at ignite/handlers
• [x] test has to be written at tests/ignite/handlers
• [x] docs has to be written in docs/source/handlers.rst

If yes, please assign me the task, else correct me if I am wrong.
Thanks for your help

[sdesrozis]

@guptadhaval18 That is correct. Thank you for the help !

The code https://github.com/pytorch/ignite/issues/936#issuecomment-628019910 could help too.

[vfdev-5]

@guptadhaval18 are you still working on the issue or I can unassign it ?