Ignite: Running trainer for more than one epoch with torch.utils.data.IterableDataset

[bhedayat]

🐛 Bug description

Hello,
I don't know if I am missing something but I am currently trying to run my trainer for more than 1 epoch where each epoch has 5 iterations. However there is not a new iterator initialized over the dataloader and you get the warning below.

tensor([1])
Epoch [1/3]: [1/5]  20%|█████████████████████████▊                                                                                                        [00:00<00:00]/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ignite/contrib/handlers/base_logger.py:124: UserWarning: Provided metric name 'loss' is missing in engine's state metrics: []
  "in engine's state metrics: {}".format(name, list(engine.state.metrics.keys()))
tensor([4])
Epoch [1/3]: [1/5]  20%|█████████████████████████▊                                                                                                        [00:00<00:00]tensor([9])
Epoch [1/3]: [2/5]  40%|███████████████████████████████████████████████████▌                                                                              [00:00<00:00]tensor([16])
Epoch [1/3]: [3/5]  60%|█████████████████████████████████████████████████████████████████████████████▍                                                    [00:00<00:00]tensor([25])
Epoch [1/3]: [5/5] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ [00:00<00:00]
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ignite/engine/engine.py:465: UserWarning: Data iterator can not provide data anymore but required total number of iterations to run is not reached. Current iteration: 5 vs Total iterations to run : 15
  self.state.iteration, self.state.epoch_length * self.state.max_epochs

Now if I do not use the ignite package and write the loop to iterate over the number of epochs it works fine. Here is a code sample I have pertaining to my use case.

import torch
from ignite.engine import Engine, Events
from ignite.contrib.handlers import ProgressBar
import pdb

class DatasetUtils:
    def __init__(self, datapoints):
        self.datapoints = datapoints

    def load_data(self):
        for d in self.datapoints:
            yield d

    def create_examples(self, data):
        for datapoint in data:
            yield datapoint**2

    def __len__(self):
        return len(self.datapoints)


class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, helper):
        self.helper = helper

    def __iter__(self):
        data_iter = self.helper.load_data()
        for example in self.helper.create_examples(data_iter):
            yield example

    def __len__(self):
        return len(self.helper)

datapoints = [1, 2, 3, 4, 5]
helper = DatasetUtils(datapoints)

ds = MyIterableDataset(helper)

data_loader = torch.utils.data.DataLoader(ds, num_workers=1, batch_size=1)


def update(engine, batch):
    print(batch)

# Using ignite 
trainer = Engine(update)
pbar = ProgressBar(persist=True)
pbar.attach(trainer, metric_names=["loss"])
trainer.run(map(lambda x: x, data_loader), epoch_length=5//1, max_epochs=3)


# Not using ignite
for epoch in range(3):
    data_iter = map(lambda x: x, data_loader)
    while True:
        try:
            batch = next(data_iter)
            print(batch)
        except StopIteration:
            break

Thank you

Environment

• PyTorch Version (e.g., 1.2):
• Ignite Version (e.g., 0.4.0.dev20200420'): Used this version to solve this issue https://github.com/pytorch/ignite/issues/932#issue-602172119
• OS (e.g., Linux):
• How you installed Ignite (pip):
• Python version: 3.6.5
• Any other relevant information:
  I printed self._dataloader_iter and it is empty
  https://github.com/pytorch/ignite/blob/10fd6022ef88f7bcff970775e9eb0920720d4d85/ignite/engine/engine.py#L472

[sdesrozis]

@bhedayat Thank you for the report ! I can reproduce. Let me check.

[sdesrozis]

It seems your codes w/wo ignite are not similar. The code wo ignite should be the following

data = map(lambda x: x, data_loader)
for epoch in range(3):
    data_iter = iter(data)
    while True:
        try:
            batch = next(data_iter)
            print(batch)
        except StopIteration:
            break

The issue occurs in both cases.

[vfdev-5]

@bhedayat you can add a restart iterator in epoch completed handler as in your code snippet:

# Using ignite 
trainer = Engine(update)
pbar = ProgressBar(persist=True)
pbar.attach(trainer, metric_names=["loss"])

@trainer.on(Events.ITERATION_COMPLETED(every=5))
def restart_dataloader():
    print(trainer.state.iteration, "restart_dataloader")
    trainer.state.dataloader = map(lambda x: x, data_loader)

trainer.run(map(lambda x: x, data_loader), epoch_length=5, max_epochs=3)

[sdesrozis]

@vfdev-5 You are so fast :)

[bhedayat]

It works! Thank you for your help