Imageai: Object Detection training problem

[KamilLysik]

Hi I have a problem with training yolo to object detection.
I have done this:
trainer = DetectionModelTrainer() trainer.setModelTypeAsYOLOv3() trainer.setDataDirectory(data_directory="../headset") trainer.setTrainConfig(object_names_array=["oculus", "hololens"], batch_size=2, num_experiments=200, train_from_pretrained_model='../yolo.h5') trainer.trainModel()
but when I run this code on google colab or jupyter it looks like:
Generating anchor boxes for training images and annotation... Average IOU for 9 anchors: 0.78 Anchor Boxes generated. Detection configuration saved in ../headset/json/detection_config.json Training on: ['hololens', 'oculus'] Training with Batch Size: 4 Number of Experiments: 200 WARNING:tensorflow:From /srv/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /srv/conda/lib/python3.6/site-packages/imageai/Detection/Custom/yolo.py:24: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. Training with transfer learning from pretrained Model /srv/conda/lib/python3.6/site-packages/keras/callbacks.py:1335: UserWarning:epsilonargument is deprecated and will be removed, usemin_deltainstead. warnings.warn('epsilonargument is deprecated and ' WARNING:tensorflow:From /srv/conda/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead.

and after above nothings happend. It looks like it freezes.

[KamilLysik]

Do you have any support?

[rola93]

Those messages are just warnings, and is due to mismatch between keras + tensorflow + numpy versions, don't worry, they are common.

training should start however. What ImageAI version are you using? upgrade it and try again

[KamilLysik]

Thanks for replay. I use 2.1.4 version.
Okey, I understund that this is only warning but I can't train because it's start and nothing happend.
I see only Epoch 1/200 and nothing else. What does it mean? I don't have any error message. I was trying on google colab and jupyter and I have there the same sitiuation.

[OlafenwaMoses]

@rola93 Thanks for the comment.

@KamilLysik The model is actually training in the background. It does takes some minutes (depending on the size of the dataset) for each experiment (epoch) to be done.

The latest update v.2.1.5 now displays progress bar when you train. Try performing the training again. You should see details like the one before:

Epoch 1/200 480/480 [==============================] - 395s 823ms/step - loss: 36.9000 - yolo_layer_1_loss: 3.2970 - yolo_layer_2_loss: 9.4923 - yolo_layer_3_loss: 24.1107 - val_loss: 15.6321 - val_yolo_layer_1_loss: 2.0275 - val_yolo_layer_2_loss: 6.4191 - val_yolo_layer_3_loss: 7.1856

[KamilLysik]

Okey, I see the progress now but how much time averagely should take training "oculus" and "hololens"? (I know that depend )
My training take for example 20 minutes and the progress is:
Epoch 1/1 25/960 [..............................] - ETA: 7:17:58 - loss: 136.5593 - yolo_layer_1_loss: 18.2737 - yolo_layer_2_loss: 40.2584 - yolo_layer_3_loss: 78.0272 - (Google Colab)

[rola93]

Did you set up correctly the GPU usage on Colab?

[OlafenwaMoses]

YOLOv3 takes time to train. However, the time it takes for each epoch depends on

• your training batch_size
• the GPU memory

Also, does your Colab Notebook have the option to use a High GPU ? If it does, you can train with batch size of 8 without memory issues.

[KamilLysik]

Yes, I use High GPU RAM on Google Colab but I have an error message after 1 hour: Your session crashed after using all available RAM. View runtime logs

[OlafenwaMoses]

Did it crash with a batch size of 8 ? If it did, that's unfortunate as I have trained up to 50 experiments on Colab using batch size of 8.

You may have to work with batch size of 4.

[KamilLysik]

Is this normal that the 1 experiment takes 5 hours with batch size 4 on Google Colab?

[OlafenwaMoses]

• Are you sure you are have the Runtime set to GPU?
• If yes, are you training on a new dataset apart from the Hololens and Headset samples?
• If yes, how many images are in your dataset?

Training speed will depend on the number of images in your dataset.

[ghost]

Did it crash with a batch size of 8 ? If it did, that's unfortunate as I have trained up to 50 experiments on Colab using batch size of 8.

You may have to work with batch size of 4.

Same problem but I have to work with batch size of 8 It's work for me.

[Yejing-Lai]

Hello, how do you use your own data set for object detection?

[rola93]

The documentation explains how to do it step by step.