Keras: The training accuracy is very high, while the validation accuracy is very low?

[newzhx]

I got a strange question. I train a two layers CNN using .flow_from_directory(), the training accuracy is very high, while the validation accuracy is very low. following is my code ,very simple.
from keras.preprocessing.image import ImageDataGenerator

from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.callbacks import EarlyStopping

model=Sequential()
model.add(Convolution2D(32, 5,5, input_shape=(3,28,28)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Convolution2D(32, 3,3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))

model.add(Dense(10))
model.add(Activation('softmax'))

train_datagen=ImageDataGenerator(rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen=ImageDataGenerator(rescale=1./255)

train_generator=train_datagen.flow_from_directory(
r'C:\Users\zhx\Desktop\mnist\train',
target_size=(28,28),
classes=['0','1','2','3','4','5','6','7','8','9'],
batch_size=60,
class_mode='categorical',
shuffle=True)

validation_generator=test_datagen.flow_from_directory(
r'C:\Users\zhx\Desktop\mnist\test',
target_size=(28, 28),
classes=['0','1','2','3','4','5','6','7','8','9'],
batch_size=100,
class_mode='categorical',
shuffle=True)

model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
early_stopping =EarlyStopping(monitor='val_loss', patience=2)
model.fit_generator(train_generator, samples_per_epoch=60000, nb_epoch=10, validation_data=validation_generator, callbacks=[early_stopping],nb_val_samples=10000)

json_string=model.to_json()
open(r'C:\Users\zhx\Desktop\mnistmodel\mnistcnn_arc.json','w').write(json_string)
model.save_weights(r'C:\Users\zhx\Desktop\mnistmodel\mnistcnn_weights.h5')
score=model.evaluate_generator(validation_generator, 10000)

print('Test score:', score[0])
print('Test accuracy:', score[1])

Here is the log(only the last )

Using Theano backend.
Found 60000 images belonging to 10 classes.
Found 10000 images belonging to 10 classes.
Epoch 1/10

60/60000 [..............................] - ETA: 812s - loss: 2.2756 - acc: 0.1667
120/60000 [..............................] - ETA: 690s - loss: 2.2897 - acc: 0.1667
180/60000 [..............................] - ETA: 688s - loss: 2.2728 - acc: 0.1833
240/60000 [..............................] - ETA: 721s - loss: 2.2692 - acc: 0.1792
300/60000 [..............................] - ETA: 768s - loss: 2.2633 - acc: 0.1800
360/60000 [..............................] - ETA: 833s - loss: 2.2561 - acc: 0.1750
420/60000 [..............................] - ETA: 884s - loss: 2.2437 - acc: 0.1786
Warning (from warnings module):
File "C:\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\keras\callbacks.py", line 67
% delta_t_median)
UserWarning: Method on_batch_end() is slow compared to the batch update (0.484438). Check your callbacks.

480/60000 [..............................] - ETA: 1002s - loss: 2.2285 - acc: 0.2042
540/60000 [..............................] - ETA: 978s - loss: 2.2102 - acc: 0.2241 
600/60000 [..............................] - ETA: 979s - loss: 2.1920 - acc: 0.2317
660/60000 [..............................] - ETA: 1013s - loss: 2.1655 - acc: 0.2439
Warning (from warnings module):
File "C:\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\keras\callbacks.py", line 67
% delta_t_median)
UserWarning: Method on_batch_end() is slow compared to the batch update (0.578522). Check your callbacks.

720/60000 [..............................] - ETA: 1057s - loss: 2.1447 - acc: 0.2514
780/60000 [..............................] - ETA: 1038s - loss: 2.1188 - acc: 0.2577
840/60000 [..............................] - ETA: 1029s - loss: 2.1058 - acc: 0.2631

.........

59400/60000 [============================>.] - ETA: 11s - loss: 0.1535 - acc: 0.9568
59460/60000 [============================>.] - ETA: 10s - loss: 0.1537 - acc: 0.9567
59520/60000 [============================>.] - ETA: 9s - loss: 0.1537 - acc: 0.9568
Warning (from warnings module):
File "C:\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\keras\callbacks.py", line 67
% delta_t_median)
UserWarning: Method on_batch_end() is slow compared to the batch update (0.705150). Check your callbacks.

59580/60000 [============================>.] - ETA: 8s - loss: 0.1536 - acc: 0.9567
Warning (from warnings module):
File "C:\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\keras\callbacks.py", line 67
% delta_t_median)
UserWarning: Method on_batch_end() is slow compared to the batch update (0.606548). Check your callbacks.

59640/60000 [============================>.] - ETA: 6s - loss: 0.1536 - acc: 0.9567
59700/60000 [============================>.] - ETA: 5s - loss: 0.1535 - acc: 0.9568
59760/60000 [============================>.] - ETA: 4s - loss: 0.1535 - acc: 0.9568
59820/60000 [============================>.] - ETA: 3s - loss: 0.1535 - acc: 0.9568
Warning (from warnings module):
File "C:\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\lib\site-packages\keras\callbacks.py", line 67
% delta_t_median)
UserWarning: Method on_batch_end() is slow compared to the batch update (0.461924). Check your callbacks.

59880/60000 [============================>.] - ETA: 2s - loss: 0.1535 - acc: 0.9567
59940/60000 [============================>.] - ETA: 1s - loss: 0.1535 - acc: 0.9567
60000/60000 [==============================] - 1189s - loss: 0.1535 - acc: 0.9567 - val_loss: 8.0909 - val_acc: 0.1783
Test score: 8.09085823536
Test accuracy: 0.178299999908

Can any one give me a suggestion? thanks.

[keunwoochoi]

Please wrap your code with ``` for readability.
Your model is overfitting and there can be too many reasons. One of them is because your dataset is too small compared to the model capacity.

[newzhx]

@keunwoochoi
I ran it again, but the val_acc was still very low.
The dataset is mnist, I used 60000 training pictures and 10000 validation pictures.
I used the same model as the example mnist-cnn.py in keras, the only difference is the data input method. what is wrong with my code or my data?
Here is the log:
Found 60000 images belonging to 10 classes.
Found 10000 images belonging to 10 classes.
Epoch 1/10
60000/60000 [==============================] - 154s - loss: 0.0226 - acc: 0.8690 - val_loss: 0.1159 - val_acc: 0.1716
Epoch 2/10
60000/60000 [==============================] - 152s - loss: 0.0085 - acc: 0.9559 - val_loss: 0.1242 - val_acc: 0.1542
Epoch 3/10
60000/60000 [==============================] - 154s - loss: 0.0063 - acc: 0.9673 - val_loss: 0.1230 - val_acc: 0.1543
Test score: 0.122995681057
Test accuracy: 0.154300000757

Here is my code:
from keras.preprocessing.image import ImageDataGenerator

from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.callbacks import EarlyStopping

from keras.optimizers import SGD

model=Sequential()
model.add(Convolution2D(32, 5,5, border_mode='valid',input_shape=(1,28,28)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))#model.add(Dropout(0.5))
model.add(Convolution2D(32, 3,3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))

model.add(Dense(10))
model.add(Activation('sigmoid'))

train_datagen=ImageDataGenerator(rescale=1./255)
test_datagen=ImageDataGenerator(rescale=1./255)

train_generator=train_datagen.flow_from_directory(
r'E:\zhx\mnist\train',
target_size=(28,28),
classes=['0','1','2','3','4','5','6','7','8','9'],
color_mode='grayscale',
batch_size=10,
class_mode='categorical',
shuffle=True)

validation_generator=test_datagen.flow_from_directory(
r'E:\zhx\mnist\test',
target_size=(28, 28),
classes=['0','1','2','3','4','5','6','7','8','9'],
batch_size=100,
color_mode='grayscale',
class_mode='categorical',
shuffle=False)

model.compile(loss='mean_squared_error', optimizer='adadelta', metrics=['accuracy'])
early_stopping =EarlyStopping(monitor='val_loss', patience=1)
model.fit_generator(train_generator, samples_per_epoch=60000, nb_epoch=10, validation_data=validation_generator, callbacks=[early_stopping],nb_val_samples=10000)

json_string=model.to_json()
open(r'E:\zhx\mnist\model\mnistdir2.json','w').write(json_string)
model.save_weights(r'E:\zhx\mnist\model\mnistdir2_weight.h5')

score=model.evaluate_generator(validation_generator, 10000)

print('Test score:', score[0])
print('Test accuracy:', score[1])

[baorave]

Hello! Have you solved the problem? I have met the same problem. I got very high training accuracy and very low validation accuracy.

[newzhx]

I changed all the activation relu functions to sigmoid one and solved the overfitting problem. I still do not know what the real problem. You can try it as well. May it help you. @baorave

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[ChenWentai]

Hello, please check if you have shuffled the data before training. Because the validation splitting in keras is performed before shuffle, so maybe you have chosen an unbalanced dataset as your validation set, thus you got the low accuracy.

[pwarriors]

@ChenWentai thanks. It helped.

[alyato]

@ChenWentai
I met the same issue.
i shuffled the dataset,then split the dataset into train_data,val_data.
and the dataset are unbalanced.
do you give some advices. thanks

[yinseng2001]

did you try to increase batch_size? example 50 - 100 - 200.
and disable rescale=1./255 on ImageDataGenerator.

[NamburiSrinath]

How to shuffle the data. As in I have used validation_split = 0.2 but it simply means that the last 20% of data is used for validation. Instead of that, can I give some arguments so that the data shuffles before splitting?

I am using fit_generator and ImageDataGenerator for training

[Houd1ny]

@ChenWentai thank you!. I spend a lot of augmentation and other tricks when the problem was so easy)

[ShikharGhimire]

Hey there. How do I shuffle the data to train my CNN? I have split the training and testing pictures in two folders inside my directory.

[NamburiSrinath]

Hey there. How do I shuffle the data to train my CNN? I have split the training and testing pictures in two folders inside my directory.

Hi ShikharGhimire,

The functions in keras(evaluate, fit etc) do have an argument called "Shuffle" Set the Booloean value to true and check if that works. It worked for me.

Inform the updates regarding this
Thank you
Srinath

[saurabh2811]

@ChenWentai
I met the same issue.
i shuffled the dataset,then split the dataset into train_data,val_data.
and the dataset are unbalanced.
do you give some advices. thanks

Sir can u help me out i am facing the same problem of validation accuracy being less and testing accuracy being very high