Keras: Good accuracy but really bad predictions

[dvirsamuel]

I am new to machine learning and deep learning, and for learning porpuses I tried to play with Resnet. I tried overfit over small data (3 different images) and see if I can get almost 0 loss and 1.0 accuracy - and I did.

The problem is that prediction on the training images(those 3 images) are not correct..

Training Images: 0.jpg 1.jpg 2.jgp

Their corresponding labels are: [1,0,0],[0,1,0],[0,0,1]

Here is part of the keras code:

#loading 3 images and resizing them
imgs = np.array([np.array(Image.open("./Images/train/" + fname)
                          .resize((140, 140), Image.ANTIALIAS)) for fname in
                 os.listdir("./Images/train/")]).reshape(-1,140,140,1)
# creating labels
y = np.array([[1,0,0],[0,1,0],[0,0,1]])
# create resnet model
model = ResNet50(input_shape=(140, 140,1),classes=3,weights=None)

# compile & fit model
model.compile(loss='categorical_crossentropy', optimizer='adam',metrics=['acc'])

model.fit(imgs,y,epochs=5,shuffle=True)

# predict on training data
print(model.predict(imgs))

The model does overfit the data:

3/3 [==============================] - 22s - loss: 1.3229 - acc: 0.0000e+00
Epoch 2/5
3/3 [==============================] - 0s - loss: 0.1474 - acc: 1.0000
Epoch 3/5
3/3 [==============================] - 0s - loss: 0.0057 - acc: 1.0000
Epoch 4/5
3/3 [==============================] - 0s - loss: 0.0107 - acc: 1.0000
Epoch 5/5
3/3 [==============================] - 0s - loss: 1.3815e-04 - acc: 1.0000
but predictions are:

[[ 1.05677405e-08 9.99999642e-01 3.95520459e-07]
[ 1.11955103e-08 9.99999642e-01 4.14905685e-07]
[ 1.02637095e-07 9.99997497e-01 2.43751242e-06]]
which means that all images got label=[0,1,0]

why? and how can that happen?

[maoshengyang]

I've encountered the similar issue. Strange but not got the solution yet.

[dvirsamuel]

@maoshengyang Check the thread I opened in stackoverflow: https://stackoverflow.com/questions/47157526/resnet-100-accuracy-during-training-but-33-prediction-accuracy-with-the-same

someone gave his answer.. have not checked it yet but you can verify..

[maoshengyang]

@dvirsamuel Awesome! That's the answer that I need. I'll try the ideas!

[GarlandZhang]

In case anyone has a similar problem, my image classification model said it had 100% classification accuracy but this was not the case when I tried to predict with the model. As it turns out, I was loading the images using opencv, which follows a BGR format but my model was trained on RGB format. In addition, my preprocessing step when testing was different than training, which was another source of problem.