Keras: Multiprocessing ImageDataGenerator - Performance Boost

Created on 1 Feb 2017 · 6Comments · Source: keras-team/keras

I've cleaned up my work to add multiprocessing.Pool support to ImageDataGenerator. It is available on this branch: https://github.com/stratospark/keras/tree/multiprocess-image-data-generator. There are some preliminary benchmark results that I plan to blog about here: https://github.com/stratospark/keras-multiprocess-image-data-generator. I'm basically getting a 3-4x speed up while training on my system, for both images in memory and images streamed from disk.

I'd like to get some feedback on whether this is a good design and whether it should potentially be merged into Keras or be a stand alone module.

One current drawback with this implementation is that zca whitening has slowed down considerably due to having to pass the principal_components array to each worker. Some kind of shared memory may help here. I also have not tested whether results can be reproduced by setting seeds.

Thanks for any feedback!

@fchollet @ncullen93

stale

Source

stratospark

👍7

Most helpful comment

Is there still work being done in this direction? Does the Keras version implement it's own multi processing version?

I can't find any info on this in the web. I did some experimentation with the multiprocessing version, but I'm not sure, if I get any speed ups.

Nimi42 on 9 Feb 2018

👍4

All 6 comments

This is so awesome! Looks very promising. Since sampling has no dependencies on the core keras code, if I were you I'd release it as a a stand-alone file in a separate repository for now. People will use this.

One suggestion for the blog - add in a section where you discuss the actual code you added/used. This is a good opportunity for you to elucidate us on how multiprocessing in python works and transcends keras, might apply to sampling for DL models in tensorflow, etc. From a quick glance, it seems clever yet straight-forward to implement this.. so it might help people make their own samplers.