Imageai: Checkpointing uses a lot of free space (Google Drive)

[Combinacijus]

TLDR

While training, custom model files are saved every epoch which takes a LOT of space. If you train for many epochs it's easy to run out of space especially on Google Drive.

Setup on Google Colab

So I'm running example code on custom object recognition model training. Only thing I have changed is that I cd to my Google Drive (GDrive) so that Colab won't delete trained model after some time. Yesterday I have left training over night without GDrive and I guess 12h have passed and all trained models and files disappeared (newbie mistake).
Code (required files already downloaded to GDrive):

# cd to GDrive
from google.colab import drive
drive.mount('/content/drive')
%cd "/content/drive/My Drive/Colab Content/ImageAILearning"

# Training custom model code
from imageai.Detection.Custom import DetectionModelTrainer

trainer = DetectionModelTrainer()
trainer.setModelTypeAsYOLOv3()
trainer.setDataDirectory(data_directory="hololens")
trainer.setTrainConfig(object_names_array=["hololens"], batch_size=4, num_experiments=200, train_from_pretrained_model="pretrained-yolov3.h5")
trainer.trainModel()

Problem:

• Single YoloV3 model size: 236MB
• _trainer.trainModel()_ - saves every model for every epoch which is really helpful because Google Colab might stop at any time. But _num_experiments=200_ therefore there will be 200 models on GDrive after full training. So 200 x 236MB = 47.2GB
• Free GDrive account only have 15GB minus already used space for personal files. So I think we see the problem.

Proposed solution

• _trainer.trainModel()_ function could have parameter of how many latest model we want to save. Let say we want to save 3 latest models so at 4th epoch we could save new model and delete oldest one (first model). There might be problem when you train for to long and overfit the model and can't get older model before overfitting but atleast it won't run out of free space on GDrive or computer.

• Another solution could be by saving every x number of epochs

Workarounds

• Manually deleting old models on GDrive.
• Automate
  
  
  
  • Setting _num_experiments_ let's say to 3
  
  
  • Change _train_from_pretrained_model_ to the oldest trained model on GDrive
  
  
  • Delete old unused models
  
  
  • Call _trainer.trainModel()_
  
  
  • Put everything in a loop
    
    (Maybe I have just described the solution ha ha)
  
  

Any other solutions?

[OlafenwaMoses]

Hello, thanks for your comments. On the contrary, the models are not stored on your Google Drive. Google Colab assigns a temporary 350GB of storage for your Notebook which is where the models are stored. Learn more via the link below.

https://stackoverflow.com/questions/47805170/whats-the-hardware-spec-for-google-colaboratory

Also keeping the latest 3-4 models sounds like a good idea. However, the latest models aren't always the most accurate in detection, despite the lower loss. Which is why we provided the .evaluateModel() function to retrieve the mAP for each model, which is the best method to obtain the most accurate model. Learn more on this via the link below.

https://github.com/OlafenwaMoses/ImageAI/blob/master/imageai/Detection/Custom/CUSTOMDETECTIONTRAINING.md#evaluatingmodels

[Combinacijus]

Thanks for reply that actually gave me some ideas.

On the contrary, the models are not stored on your Google Drive

Yes it's not stored there by default but I have mounted GDrive to store my files there. And only reason for that is that after 12h you loose all data on Colab temporary storage. And for example if you left training over night or etc. you might loose models before downloading them. And GDrive doesn't have enough space to store all models generated over night.

_trainer.trainModel()_ is blocking function so I can't evaluate models while training. I should probably make a thread to handle evaluation and file moving if GPU can handle that.

Also this library is just amazing. Now it's really easy to start with ML, Thanks!

[OlafenwaMoses]

Hello please see an urgent bug fix update here #278