Keras: Each time I run the Keras, I get different result.

[spraveengupta]

Each time I run the Keras, I get inconsistent result.
Is there any way that it converges to the same solution as we have 'random_state' in sklearn which helps us getting the same solution how many ever times we run it.
Please help me to get out of this issue

[amirothman]

by default Keras's model.compile() sets the shuffle argument as True. You should the set numpy seed before importing keras. e.g.:

import numpy as np
np.random.seed(1337) # for reproducibility
from keras.models import Sequential

most of the provided Keras examples follow this pattern.

[spraveengupta]

Hi amirothman,

Thanks for the prompt response.
even after setting the seed, it is giving me inconsistent result.

model.compile is not taking any shuffle argument. whereas, model.fit takes shuffle argument. So, I passed 'False' value to the argument. Even after that it is giving me inconsistent result. I will be highly grateful, if there is any other way to obtain consistent result.

FYI, please find my code

np.random.seed(1337)
print('Building model...')

model = Sequential()
model.add(Dense(2048, init = 'glorot_normal', input_shape=(dims,),activation='sigmoid'))
model.add(PReLU())
model.add(BatchNormalization())

model.add(Dense(2048))
model.add(PReLU())
model.add(BatchNormalization())

model.add(Dense(nb_classes))
model.add(Activation('softmax'))

model.compile(loss='binary_crossentropy', optimizer='sgd')
from sklearn.metrics import log_loss
from numpy import savetxt
#roc_auc_score(y, y_score)
from sklearn.cross_validation import StratifiedKFold
cv=[]
cvscore=[]
kf=StratifiedKFold(labels, n_folds=5, shuffle=True, random_state=111)
for train_index,test_index in kf:
    X_train, X_test = X[train_index],X[test_index]
    y_train, y_test = y[train_index],y[test_index]
    ID_test=X_ID[test_index]
    print("#############################")
    model.fit(X_train, y_train, nb_epoch=10, batch_size=1024,shuffle=False)
    y_score = model.predict_proba(X_test)
    cvscore.append(log_loss(y_test, y_score))
    print(log_loss(y_test, y_score))
    TEMP = (ID_test,y_score[:,1])
    TEMP2=zip(*TEMP)
    cv=cv+TEMP2

[AntreasAntoniou]

The model weights are initialised randomly according to the initialization
type. In general stochastic optimisation is not known to yield the exact
same result each time. That's why people like to use ensembles of models to
give more accurate predictions.

On Wed, 18 May 2016 00:18 spraveengupta, [email protected] wrote:

Hi amirothman,

Thanks for the prompt response.
even after setting the seed, it is giving me inconsistent result.

model.compile is not taking any shuffle argument. whereas, model.fit takes
shuffle argument. So, I passed 'False' value to the argument. Even after
that it is giving me inconsistent result. I will be highly grateful, if
there is any other way to obtain consistent result.

FYI, please find my code

np.random.seed(1337)
print('Building model...')

model = Sequential()
model.add(Dense(2048, init = 'glorot_normal', input_shape=(dims,),activation='sigmoid'))
model.add(PReLU())
model.add(BatchNormalization())

model.add(Dense(2048))
model.add(PReLU())
model.add(BatchNormalization())

model.add(Dense(nb_classes))
model.add(Activation('softmax'))

model.compile(loss='binary_crossentropy', optimizer='sgd')
from sklearn.metrics import log_loss
from numpy import savetxt

roc_auc_score(y, y_score)

from sklearn.cross_validation import StratifiedKFold
cv=[]
cvscore=[]
kf=StratifiedKFold(labels, n_folds=5, shuffle=True, random_state=111)
for train_index,test_index in kf:
X_train, X_test = X[train_index],X[test_index]
y_train, y_test = y[train_index],y[test_index]
ID_test=X_ID[test_index]
print("#############################")
model.fit(X_train, y_train, nb_epoch=10, batch_size=1024,shuffle=False)
y_score = model.predict_proba(X_test)
cvscore.append(log_loss(y_test, y_score))
print(log_loss(y_test, y_score))
TEMP = (ID_test,y_score[:,1])
TEMP2=zip(*TEMP)
cv=cv+TEMP2

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2743#issuecomment-219882332

[spraveengupta]

Hi Antreas,

Thank you for the response.
So, don't we have any way to obtain consistent result?
I did not optimize my model yet so, I am ready to change my initialization type and the optimizer as well. Can you please let me know choosing which types would give me consistent results.
Now, the variation being shown in every run is very high. So, I am looking for the parameters which would give me consistent result.

Thanks in advance.

[amirothman]

i also read somewhere (the link escapes me right now), that your version of Theano may ignore your seed. Make sure you have the latest version of Theano.

[spraveengupta]

Hi @amirothman

Thanks for the information.
However, I have installed the bleeding edge version of Theano.

[joelthchao]

@spraveengupta Possibly relate to this #2479?

[spraveengupta]

Hi Joel,

I see the issue #2479 is closed. However, my theano is a bleeding edge version and I have latest version of Keras installed on my machine.
Can you please check and let me know what else could be the reasons for the inconsistent outputs. (FYI, I have posted my code in my previous comment)

Regards,
Praveen

[braingineer]

you could arbitrarily recreate the same result by saving the model weights immediately upon initalization and then loading them for future learning experiments. aka: initialize once, reuse.

[mbollmann]

@spraveengupta In your example, you don't include the Keras import statements. Have you really set np.random.seed(1337) _before_ importing any Keras modules, like @amirothman suggested?

[spraveengupta]

Hi @mbollmann,

Thanks for the suggestion. I have set the seed before importing the Keras libraries. Here, just to show that I have put the seed, I have edited in that way.
After a lot of prolonged analysis I found that, to get the consistency in results, we need to shutdown the ipynb file, restart once again and run the code.
If I just interrupt and rerun the code once again, it is giving me inconsistent results. (However, I expect that the results should be consistent even if I forcefully interrupt and rerun the code once again. Please let me know your comments).

Anyway, as there is a way to produce the consistent result, I am closing the issue.

Thanks once again for all your valuable suggestions.

[GuokaiLiu]

It seems that I solved this problem in this way:
http://blog.csdn.net/qq_33039859/article/details/75452813
step1: fix the numpy random seed at the top of code
step2: be sure that model.fit(shuffle=False)

[mrgloom]

I have fixed random seed like:

import numpy as np
np.random.seed(2017)

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "3"

import sys
import random
import math

import cv2
import pandas as pd

from keras.models import Model
from keras.layers import Input, Conv2D, MaxPooling2D, UpSampling2D, Conv2D, Reshape
from keras.layers import concatenate
from keras.layers.normalization import BatchNormalization
from keras.layers.core import Dropout, Activation
from keras.optimizers import Adadelta, Adam
from keras.callbacks import ModelCheckpoint, EarlyStopping
from keras import backend as K

Disable shuffle like this:

    history = model.fit_generator(
        generator=batch_generator(batch_size),
        nb_epoch=epochs,
        samples_per_epoch=batch_size,
        validation_data=batch_generator(batch_size),
        nb_val_samples=batch_size,
        verbose=1,
        callbacks=callbacks,
    shuffle=False)

I don't use ipython notebook (I use python2) and I'm using Tensorflow backend, but still have slightly different results at each run if I look at loss and val_loss at some iteration.

Also running in CPU only mode via os.environ["CUDA_VISIBLE_DEVICES"] = "" I get same issue.

Also setting tensorflow seed not helped:

import numpy as np
np.random.seed(2017)

from tensorflow import set_random_seed
set_random_seed(2017)

[milinddeore]

All the above answers look perfect but i have given snippet of complete code.

This code is for tensorflow backend.

import tensorflow as tf
import random as rn

os.environ['PYTHONHASHSEED'] = '0'

# Setting the seed for numpy-generated random numbers
np.random.seed(37)

# Setting the seed for python random numbers
rn.seed(1254)

# Setting the graph-level random seed.
tf.set_random_seed(89)

from keras import backend as K

session_conf = tf.ConfigProto(
      intra_op_parallelism_threads=1,
      inter_op_parallelism_threads=1)

#Force Tensorflow to use a single thread
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)

K.set_session(sess)

# Rest of the code follows from here on ...

[oseiskar]

I addition to everything that has already been mentioned here, keep multiprocessing=False in methods like fit_generator

[jruivo-dev]

Hey everyone, I've tried all the settings from this post but I still cannot get the same results, I don't know what's wrong or what am I missing.
I've set up the seeds on the top of the file, exact code from the Keras faq, and I've set shuffle=False on the fit_generator.
Here's the code:

import numpy as np
import tensorflow as tf
import random as rn
import os
os.environ['PYTHONHASHSEED'] = '0'
np.random.seed(42)
rn.seed(12345)
session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
from keras import backend as K
tf.set_random_seed(1234)
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
K.set_session(sess)

from keras.layers import Input, Dropout, Flatten, Conv2D, MaxPooling2D, Dense, Activation, Lambda,GlobalAveragePooling2D
from keras.optimizers import RMSprop , SGD, Adam,Nadam
from keras.callbacks import ModelCheckpoint, Callback, EarlyStopping, History
from keras.preprocessing.image import ImageDataGenerator
from keras.applications import VGG16, VGG19, ResNet50, Xception
from keras.models import Model

batch_size = 32
num_channels = 3
img_size = 512
img_full_size = (img_size, img_size, num_channels)
num_classes = 2
seed = 1 # for image transformations
train_path = 'keras_folders/train/'
validation_path = 'keras_folders/val/'
test_path = 'keras_folders/test/'

train_datagen = ImageDataGenerator(
    rescale=1./255,
    horizontal_flip=True)

validation_datagen = ImageDataGenerator(
    rescale=1./255)

test_datagen = ImageDataGenerator(
    rescale=1./255)

train_generator = train_datagen.flow_from_directory(
    train_path,
    target_size=(img_size, img_size),
    batch_size=batch_size,
    class_mode='categorical', 
    seed=seed)

validation_generator = validation_datagen.flow_from_directory(
    validation_path,
    target_size=(img_size, img_size),
    batch_size=batch_size,
    shuffle=False,
    class_mode='categorical',
    seed=seed)

from collections import Counter
counter = Counter(train_generator.classes)
max_val = float(max(counter.values()))
class_weights = {class_id : max_val/num_images for class_id, num_images in counter.items()}  

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=img_full_size)
conv_base.trainable=True
for layer in conv_base.layers[:4]:
    layer.trainable = False
x = Flatten()(conv_base.output)
x = Dense(256, activation='relu')(x)
x = Dropout(0.218)(x)
predictions = Dense(num_classes, activation='softmax')(x)
model = Model(inputs = conv_base.input , outputs=predictions)

adam = Adam(lr=0.0001)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

train_samples = train_generator.samples
validation_samples = validation_generator.samples
model.fit_generator(
    train_generator,
    class_weight=class_weights,
    steps_per_epoch= train_samples // batch_size,
    epochs=1,
    validation_data= validation_generator,
    validation_steps= validation_samples // batch_size,
    shuffle=False)

[fast-reflexes]

@jruivo-dev If you use SGD instead of Adam, do you get reproducible results? My preliminary results indicate that Adam, RMSProp etc... contains some unknown random source which I have not yet locked down and with SGD I get reproducible results which I don't get if I use any of the more complex optimizers instead ...

[Stancoo]

Yes, SGD seems to be indeed more stable, i.e. I get "much more reproducible" results with SGD, but sadly they are still not identical.

[RamyaAV]

write the below line just before the model.fit line
np.random.seed(0)

[sri9s]

Could anyone figure out how to correct this? I get different accuracies and loss values for every run.

[RamyaAV]

it is because every time Keras layers use different values as
initialization weights to different parameters.
in order to fix it, use the below line just before model.add() line
np.random.seed(0)

On Thu, Dec 27, 2018 at 10:38 PM sri9s [e-mail] wrote:

Could anyone figure out how to correct this? I get different accuracies
and loss values for every run.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/2743#issuecomment-450115143,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Arn7WulBQpH9dW-Y3a4o2Xh1g--wpDQxks5u9JULgaJpZM4IgXvU
.

--
Thanks with Regards
Ramya A V

[sri9s]

it is because every time Keras layers use different values as initialization weights to different parameters. in order to fix it, use the below line just before model.add() line np.random.seed(0)
…
On Thu, Dec 27, 2018 at 10:38 PM sri9s @.*> wrote: Could anyone figure out how to correct this? I get different accuracies and loss values for every run. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2743 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/Arn7WulBQpH9dW-Y3a4o2Xh1g--wpDQxks5u9JULgaJpZM4IgXvU .
-- Thanks with Regards Ramya A V

Like you said I tried it, but I still get different scores for each run. Here is my code:

np.random.seed(0)
model = keras.Sequential([keras.layers.Flatten(input_shape=(1,9000)),
                     keras.layers.Dense(200, activation=tf.nn.relu),            
                     keras.layers.Dense(4, activation=tf.nn.softmax)])



model.compile(optimizer=tf.train.AdamOptimizer(), 
          loss='sparse_categorical_crossentropy',
          metrics=['accuracy'])



history = model.fit(training_set,training_set_label, validation_split=0.20, epochs=10)

[InChristAlone]

when I use model.save I also encounter the strange problem--yield random result when predict., so then I save the model architecture to .yaml and weights to .h5. then reload the model, but also face the same problem

but an exciting thing happen when I change from
model_from_yaml(loaded_model_yaml)
to
tf.keras.models.model_from_yaml(loaded_model_yaml)

it works, yield the same prediction results!

[machengcheng2016]

This solved the problem:
https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development

Seems works for python2 and python3.

still don't work for me :(

[mrgloom]

Seems in pytorch docs they say that reproducibility is not guaranteed Furthermore, results need not be reproducible between CPU and GPU executions, even when using identical seeds.
https://pytorch.org/docs/stable/notes/randomness.html

[nvinayvarma189]

You need to set a seed value for numpy, Tensorflow/Theano and any 3rd party libraries that you are using. Here is a nice blog post covering all the cases. Also, please check this.

[machengcheng2016]

@mrgloom @nvinayvarma189 Thanks for your help. Following your instruction, I've set the random seeds of Numpy, Tensorflow, and Random at the beginning of my script test.py. Then run CUDA_VISIBLE_DEVICES="" PYTHONHASHSEED=0 python test.py, and I can finally get same result every time. (repeat almost 100 times)

[mrgloom]

Seems reproducible results is not guaranteed in case of GPU:
https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development
Moreover, when using the TensorFlow backend and running on a GPU, some operations have non-deterministic outputs, in particular tf.reduce_sum(). This is due to the fact that GPUs run many operations in parallel, so the order of execution is not always guaranteed. Due to the limited precision of floats, even adding several numbers together may give slightly different results depending on the order in which you add them.

[mrgloom]

Simple test that shows results are not reproducible:

wget https://raw.githubusercontent.com/keras-team/keras/master/examples/mnist_cnn.py
time CUDA_VISIBLE_DEVICES="1" PYTHONHASHSEED=0 python mnist_cnn.py

Run 1:
Test loss: 0.027183168271976227
Test accuracy: 0.9923

Run 2:
Test loss: 0.025196429155063833
Test accuracy: 0.9928

V2 with added fixed random seed before other imports (for tensorflow, numpy, python random):
time CUDA_VISIBLE_DEVICES="1" PYTHONHASHSEED=0 python mnist_cnn_v2.py

Run 1:
Test loss: 0.033751709432680944
Test accuracy: 0.9895

Run 2:
Test loss: 0.03416693595497927
Test accuracy: 0.9894

For single epoch on CPU:
V1(multi core and not fixing random seeds):
time CUDA_VISIBLE_DEVICES="" PYTHONHASHSEED=0 python mnist_cnn.py

Run 1:
Test loss: 0.05565065107960254
Test accuracy: 0.9821

Run 2:
Test loss: 0.06051686334293336
Test accuracy: 0.9811

V2 (single core and fixed random seeds):
time CUDA_VISIBLE_DEVICES="" PYTHONHASHSEED=0 python mnist_cnn_v2.py

Run 1:
Test loss: 0.055917373919580134
Test accuracy: 0.9822

Run 2:
Test loss: 0.055917373919580134
Test accuracy: 0.9822

V3(multi core and fixed random seeds):
time CUDA_VISIBLE_DEVICES="" PYTHONHASHSEED=0 python mnist_cnn_v2_multi_cpu.py

Run 1:
Test loss: 0.06009488845057785
Test accuracy: 0.9806

Run 2:
Test loss: 0.058704034704808145
Test accuracy: 0.9815

So result is only reproducible for single CPU.

[vamshi-1]

I found a solution from https://github.com/keras-team/keras/issues/2280. the following worked for me

import os
os.environ['PYTHONHASHSEED'] = '0'
os.environ['CUDA_VISIBLE_DEVICES']='-1'
os.environ['TF_CUDNN_USE_AUTOTUNE'] ='0'

import numpy as np
import random as rn
import tensorflow as tf

rn.seed(1)
np.random.seed(1)
from tensorflow import set_random_seed
set_random_seed(1)

from keras import backend as k
config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1,
allow_soft_placement=True, device_count = {'CPU': 1})
sess = tf.Session(graph=tf.get_default_graph(),config=config)
k.set_session(sess)

[jozi98]

@vamshi-1 can I ask what os.environ['CUDA_VISIBLE_DEVICES']='-1' is used for.

Does -1 assign the load to all GPU's, since typically you see a non-negative value such as 0 or 1 for example .

[RayneChiang]

@vamshi-1 can I ask what os.environ['CUDA_VISIBLE_DEVICES']='-1' is used for.

Does -1 assign the load to all GPU's, since typically you see a non-negative value such as 0 or 1 for example .

To disable the use of GPU

[NajwaLahly]

I had the same problem, I solved it with fixing seeds before importing keras+ adding suffle = false to model.fit. The result it the same if you run the code many times, if you restart kernel, to reinitialize weights

import numpy as np
np.random.seed(90)
import tensorflow as tf
tf.set_random_seed(96)
from tensorflow import keras
import tensorflow.keras.backend as K

history = model.fit(X_train, dummy_Y_train, batch_size=1200, epochs=5, verbose=2, shuffle=False)