Keras: not expected images from autoencoder with sparsity constraint

Created on 16 Feb 2017 · 3Comments · Source: keras-team/keras

Hello all,

This problem might be not so good to post here, but please help me.
I follow the examples from 'Building Autoencoders in Keras' (https://blog.keras.io/building-autoencoders-in-keras.html), but encounter a problem when I try to run a sparse autoencoder. The simplest case works well, however, when I add a regularizers.activity_l1(10e-5) in encoder layer, the images predicted from decoder model are very fuzzy.

here is the source code:

=================================================================================================

Adding a sparsity constraint on the encoded representations

=================================================================================================

from keras.layers import Input, Dense
from keras.models import Model
from keras.datasets import mnist
from keras import regularizers

import numpy as np
import matplotlib.pyplot as plt

encoding_dim = 32
input_img = Input(shape=(784,))

add a Dense layer with a L1 activity regularizer

encoded = Dense(encoding_dim, activation='relu',
activity_regularizer=regularizers.activity_l1(10e-5))(input_img)
decoded = Dense(784, activation='sigmoid')(encoded)

autoencoder = Model(input=input_img, output=decoded)

create a separate encoder and decoder model

encoder = Model(input=input_img, output=encoded)
encoded_input = Input(shape=(encoding_dim,))
decoder_layer = autoencoder.layers[-1]
decoder = Model(input=encoded_input, output=decoder_layer(encoded_input))

-------------------------------------------------------------------------------------------------

Let's train this model for 100 epochs (with the added regularization the model is less likely

to overfit and can be trained longer). The models ends with a train loss of 0.11 and test loss of 0.10.

The difference between the two is mostly due to the regularization term being added to the loss

during training (worth about 0.01).

-------------------------------------------------------------------------------------------------

autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

Get train and test data

(x_train, _), (x_test, _) = mnist.load_data()
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
print(x_train.shape)
print(x_test.shape)

autoencoder.fit(x_train, x_train,
nb_epoch=100,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))

encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)

n = 10 # how many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
# display original
ax = plt.subplot(2, n, i + 1)
plt.imshow(x_test[i].reshape(28, 28))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)

# display reconstruction
ax = plt.subplot(2, n, i + 1 + n)
plt.imshow(decoded_imgs[i].reshape(28, 28))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)

plt.show()

Source