Tflearn: Computer freezes as soon as training starts

[SebastianKlein]

Hi,
I'm currently have 2 hidden layers, one convolutional and one fully connected, with 200 neurons each. The input are 640x120x1 images with a batch size of 8. But as soon as the training starts, the whole computer freezes. I have waited about 30 min but nothing happens and I have to do a power reset (no kernel log is saved).

    #Input
    network = input_data(shape=[None, 640, 120, 1], name='input')

    # Hidden 1
    network = conv_2d(network, 200, 3, activation='relu', regularizer="L2")
    network = max_pool_2d(network, 2)
    network = local_response_normalization(network)

    # Hidden 2
    network = fully_connected(network, LayerNeuron, activation='tanh')
    network = dropout(network, 0.8)

    # Output
    network = fully_connected(network, 2, activation='softmax')
    network = regression(network, optimizer='adam', learning_rate=0.01,
                       loss='categorical_crossentropy', name='target')

    # Training
    model = tflearn.DNN(network, tensorboard_verbose=0,)
    model.fit({'input': train_X}, {'target': train_Y}, n_epoch=1,
              validation_set=0.1,
              snapshot_step=500, show_metric=True, batch_size=8)

The server is running ubuntu has about 32GB ram and is using about 28Gb before the crash. Tensorflow is running in cpu mode. How can I fix this? Shouldn't tensorflow catch OOM errors?

Thanks!

[aymericdamien]

How large are your data? Maybe you are trying to load too much data in memory. You can try to use 'image_preloader' http://tflearn.org/data_utils/#image-preloader, to load your data from disk.

When you say it freezes, can you check CPU usage, to know if it really freeze or if it is just computing... Also try a smaller batch size (such as 1 just for test).

[SebastianKlein]

Thanks for your answer. My dataset has 7400 images(90% test and 10% validation). I can't do anything when it freezes. But until it freezes each CPU is running at 90% - 97%. I've tested different amounts of neurons and as soon as as the ram is about to hit it's limit the who system freezes.
I've tested the model on my pc and there I get an OOM error, on the server however not :/

I'm currently planing to reduce the image size and I'll try a batch size of 1.

[jid1]

Don't know if this has anything to do with this issue but hopefully could be useful to some people.

Problem:
PC freezing following CPU training required a hard reboot. Even though I tried multiple configurations, my model consisted of a few LSTM layers, adam optimizer, a BatchNormalization layer and softmax.

Solution:
I updated the motherboard BIOS "ASUS Z87-K" and until now it seems to be working fine.
Update: Following the update this issue occurred again but in lower frequency. I have turned off "turbo mode" on my motherboard and stressed my PC for 2 days and all seems to be working fine.

Sorry for the long comment, but this should help people searching...

Extra info:
My PC was completely freezing, running on CPU. My CPU utilisation was 100% and my CPU working above the 3.5GHz at about 3.64GHz. After checking temperature (ok), forcing CPUs to work slower the issue persisted.

I installed docker assuming it could help but I got exactly the same.

The freeze tended to happen after approx. 1736 epochs and even if I run in 4 training batches (repeatedly calling fit) it would freeze at about the same number of epochs (regardless of time between runs).

System:
Windows 10 Pro (latest update)
32GB RAM
Intel i7-4770K @3.5GHz
ASUS Z87-K