Keras: Batch_size scheduler: successful update of batch_size value in params, but no effect on training

[iexarch]

I want to create a batch_size scheduler using callback on keras (in a manner similar to the learning rate scheduler, which I try to imitate in my code) but, although I'm able to change the batch_size in self.params, this does not seem to affect the actual batch_size implemented during training. Here is what I have tried:

class BatchSizeScheduler(Callback):
      """Batch size scheduler.
      # Arguments
        schedule: a function that takes an epoch index as input
        (integer, indexed from 0) and current batch size
        and returns a new batch size as output (int).
        verbose: int. 0: quiet, 1: update messages.
     """
    def __init__(self, schedule, verbose=1):
         super(BatchSizeScheduler, self).__init__()
         self.schedule = schedule
         self.verbose = verbose

     def on_epoch_begin(self, epoch, logs=None):


         print(self.params)
         bs = self.params['batch_size']

         try:  # new API
            bs = self.schedule(epoch, bs=bs)
         except TypeError:  # old API for backward compatibility
            bs = self.schedule(epoch)
         if not isinstance(bs, (int, np.int32, np.int64)):
            raise ValueError('The output of the "schedule" function '
                         'should be integer.')

         self.params['batch_size'] =  bs 

         if self.verbose > 0:
             print('\nEpoch %05d: BatchSizeScheduler increasing batch '
                 'size to %s.' % (epoch + 1, self.params['batch_size']))

The problem is, this does not seem to update the batch size during training. I get no errors, the print(self.params) shows an increasing batch size, but I can tell from the training progress bar that the algorithm keeps using the same, initial batch size, despite the increased value in self.params. What's the cause of this? I suspect the way I update the parameter (self.params['batch_size']=bs) is wrong?

[zachmayer]

I just want to ping this issue: It'd be awesome to have a callback I could use to change batch sizes

[Dref360]

You should implement a keras.utils.Sequence with a variable batch size.

class MySequence(keras.utils.Sequence):
    def __init__(self):
        self.batch_size = 10
        ...
    def on_epoch_end(self):
        self.batch_size = 15

    def __getitem__(self,idx):
        ...
    def __len__(self):
        ...

[zachmayer]

So if I wanted to say, double my batch size every epoch, I could do something like this?

class DoubleBatchSize(keras.utils.Sequence):
    def __init__(self, x, y, batch_size, shuffle=True):
        self.x = x
        self.y = y
        self.batch_size = batch_size
        self.shuffle = shuffle

    def on_epoch_end(self):
        self.batch_size *= 2
        if self.shuffle:
            np.random.shuffle(self.idx)

    def __len__(self):
        return int(np.ceil(len(self.x) / float(self.batch_size)))

    def __getitem__(self, idx):
        batch_x = self.x[idx * self.batch_size:(idx + 1) * self.batch_size]
        batch_y = self.y[idx * self.batch_size:(idx + 1) * self.batch_size]
        return np.array(batch_x), np.array(batch_y))

training_generator = DoubleBatchSize(x, y, batch_size=32, shuffle=T)
my_model.fix_generator(training_generator)

?

[Dref360]

Yeah kinda.

An issue would be that the number of steps per epoch would stay the same. So the epoch would progressively get longer.

[zachmayer]

Could I halve the number of steps each epoch too?

Sent from my iPhone

On Dec 18, 2018, at 9:32 AM, Frédéric Branchaud-Charron [e-mail] wrote:

Yeah kinda.

An issue would be that the number of steps per epoch would stay the same. So the epoch would progressively get longer.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[Dref360]

This is not currently possible.

---

De : Zach Mayer [e-mail]
Envoyé : 18 décembre 2018 09:35:26
À : keras-team/keras
Cc : Frédéric Branchaud-Charron; Comment
Objet : Re: [keras-team/keras] Batch_size scheduler: successful update of batch_size value in params, but no effect on training (#10143)

Could I halve the number of steps each epoch too?

Sent from my iPhone

On Dec 18, 2018, at 9:32 AM, Frédéric Branchaud-Charron [e-mail] wrote:

Yeah kinda.

An issue would be that the number of steps per epoch would stay the same. So the epoch would progressively get longer.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/keras-team/keras/issues/10143#issuecomment-448240735, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIj4on8ZjMtsNOAfCG6I3VJWU6WTt-GTks5u6P0ugaJpZM4T3BkW.

[zachmayer]

Ok thanks!

[moorejee]

So, how can I change the batch size in training? For example, in the beginning 10 epochs, using a smaller batch size, in the remaining epochs, using a larger batch size. Is there a solution?

[zachmayer]

@moorejee train 1 epoch at a time, in a loop. e.g.:

python for i in range(10): model.fit(x, y, batch_size = i*10001)

[moorejee]

@zachmayer OK, that is a trade-off method, thanks

@moorejee train 1 epoch at a time, in a loop. e.g.:

for i in range(10):
    model.fit(x, y, batch_size = i*10001)

[maxpv]

This is not currently possible.

How can I help with this?

Two papers supporting this method: Don't Decay the Learning Rate, Increase the Batch Size and On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima.

[Dref360]

This is quite simple, we just need to recompute the number of step after each epoch. (len(sequence) if the data is a Sequence otherwise stay the same.)