Keras: No AutoEncoder in keras code ??

[saj1919]

I am getting following error

ImportError                               Traceback (most recent call last)
<ipython-input-2-02e63576e9b0> in <module>()
----> 1 from keras.layers.core import Dense, AutoEncoder
ImportError: cannot import name AutoEncoder

There is no AutoEncoder class in https://github.com/fchollet/keras/blob/master/keras/layers/core.py
Previously it was there as I discussed some problem based on it here - https://github.com/fchollet/keras/issues/365

Can somebody confirm the same ?

[TeunKrikke]

Unfortunately I can confirm the same. It seems to have been removed in the Keras 1.0 preview but why?? It was working fine and how can we now develop the AutoEncoder?

[fchollet]

The functional API removes the need for a dedicated Autoencoder layer. The
best way to implement autoencoders in Keras at this point is via the
functional API.

On 11 April 2016 at 23:42, Swapnil Jadhav [email protected] wrote:

I am getting following error

---

ImportError Traceback (most recent call last)
in ()
----> 1 from keras.layers.core import Dense, AutoEncoder
ImportError: cannot import name AutoEncoder

There is no AutoEncoder class in
https://github.com/fchollet/keras/blob/master/keras/layers/core.py
Previously it was there as I discussed some problem based on it here -

365 https://github.com/fchollet/keras/issues/365

Can somebody confirm the same ?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2269

[TeunKrikke]

To be complete and to check this should resemble a small AutoEncoder

    inputs = Input(shape=(45,))
    encoder = Dense(110)(inputs)
    decoder = Dense(45)(encoder)
    model = Model(input=inputs, output=decoder)

[thenomemac]

Wouldn't this autoencoder implemented without shared weights for the encoder and decoder layers? I'm trying to implement an autoencoder such as referenced in the literature where the encoder layer is $W$ and the decoder layer is $W^{T}$. I believe this is how the old Autoencoder layer in keras worked, but I'm unclear how I could implement this shared weight scheme in the functional API?

[dancivitarese]

I used to stack auto encoders by creating each one as:

ae = Sequential()
auto_e = AutoEncoder(
    encoder=containers.Sequential([Dense(n_out, input_shape=(n_in,),
                                         activation='sigmoid')]), 
    decoder=containers.Sequential([Dense(n_in,
                                         input_shape=(n_out,),
                                         activation='sigmoid')]),
    output_reconstruction=True)
ae.add(auto_e)
ae.compile(loss='binary_crossentropy', optimizer=optimizers.RMSprop())
ae.fit(labels_tmp, labels_tmp, batch_size=batch_size,
       nb_epoch=nb_epoch, show_accuracy=True, verbose=1)

where n_in and n_out is, respectively, the input and the output dimentions. In order to train inner auto encoders, I used to create a temporary dataset by calling the last one without reconstructing the output:

auto_e.output_reconstruction = False
ae.compile(loss='binary_crossentropy', optimizer=optimizers.RMSprop())
encoders.insert(0, ae.layers[0].decoder)
labels_tmp = ae.predict(labels_tmp)

buy now, with this new format, I tried to do:

ae = Sequential()
inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation="tanh")(inputs)
decoder = Dense(n_in)(encoder)
auto_e = Model(input=inputs, output=decoder)
ae.add(auto_e)
ae.compile(loss='binary_crossentropy', metrics=["binary_accuracy"],
           optimizer=optimizers.RMSprop())
ae.fit(x_train_tmp, x_train_tmp, batch_size=batch_size, nb_epoch=nb_epoch,
       verbose=1)
auto_e.output_reconstruction = False
ae.compile(loss='binary_crossentropy', metrics=["binary_accuracy"],
           optimizer=optimizers.RMSprop())
encoders.append(encoder)
x_train_tmp = ae.predict(x_train_tmp)

without success, because network output is still equal to the input. Does any one know how do I get the output of the new encoder (without reconstruction)?

[matrix0415]

I faced the same problem for autoencoders without reconstruction.
How to implement Stacked autoencoders on the newer version of Keras?
Thanks!

[dancivitarese]

I managed to have partial outputs to stack them. Unfortunately, I didn't have time to stack them yet. If you do it, please let me know.

Consider a list encoders for storing the pre-trained encoders.

inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation="tanh")(inputs)
decoder = Dense(n_in, activation="tanh")(encoder)
auto_e = Model(input=inputs, output=decoder)
auto_e.compile(loss='binary_crossentropy', metrics=["binary_accuracy"],
               optimizer=optimizers.RMSprop())
auto_e.fit(x_train_tmp, x_train_tmp, batch_size=batch_size, nb_epoch=nb_epoch,
           verbose=1)

auto_e = Model(input=inputs, output=encoder)
auto_e.compile(loss='binary_crossentropy', metrics=["binary_accuracy"],
               optimizer=optimizers.RMSprop())

x_train_tmp = auto_e.predict(x_train_tmp)
encoders.append(encoder)

Now it's missing the way to stack them in a Sequential or Model. My first attempt using Sequential was unsuccessful.

[mzola]

Referring to @dchevitarese first sample code.
I think that there are two possible ways to do that.

The first is to use keras backend (in my example I'm using theano):

inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation="tanh")
decoder = Dense(n_in)(encoder(inputs))
auto_e = Model(input=inputs, output=decoder)
ae.compile(loss='binary_crossentropy', metrics=["binary_accuracy"], optimizer=optimizers.RMSprop())
ae.fit(x_train_tmp, x_train_tmp, batch_size=batch_size, nb_epoch=nb_epoch, verbose=1)

After you fit the model you can take intermediate output tensor encoder and create a function that can be evaluated on new data. The model weights remain the same of the original model. That correspond to the encoding of previous Autoencoders methods.

from keras import backend as K
encoder_func = K.function([ae.layers[0].input], [ae.layers[1].output])
encoder_output = encoder_func([x_train_tmp])[0]

Another method is to use the functional approach:
encoder_output = encoder(x_train_tmp).eval()
I don't know if there is a better way to retrieve the encoder output without calling the eval() function.

[braingineer]

I think the mistake here is that you aren't saving the functions as their own entities. Instead, you are using them right away to retrieve the tensors.

Try this code:

n_in, n_out, depth = 10, 5, 2
inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation='tanh')
decoder = Dense(n_in)

state = inputs

for _ in range(depth):
    state = decoder(encoder(state))

model = Model(input=inputs, output=state)
model.compile('SGD', 'MSE')

or, with this compose function

def compose(*layers):
    def func(x):
        out = x
        for layer in layers[::-1]:
            out = layer(out)
        return out
    return func

n_in, n_out, depth = 10, 5, 2
inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation='tanh')
decoder = Dense(n_in)

ae = compose(decoder, encoder)

state = inputs

for _ in range(depth):
    state = ae(state)

model = Model(input=inputs, output=state)
model.compile('SGD', 'MSE')

I haven't verified this with data as I'm not familiar with the typical data in AE situations. If you could please verify, that would be great. If it doesn't work, I'd be happy to help debug.

Edit: things might not be set up for layer recursion like this. in principle, you would think it would create a new instance of the layer with a different name but the same weights. When you try to do (1) and (2), things get a little weird. And (3) won't even plot.

### 1
n_in, n_out, depth = 10, 5, 1
inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation='tanh')
decoder = Dense(n_in)

state = decoder(encoder(inputs))

model = Model(input=inputs, output=state)
model.compile('SGD', 'MSE')

SVG(model_to_dot(model).create(prog='dot', format='svg'))

### 2 
n_in, n_out, depth = 10, 5, 1
inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation='tanh')
decoder = Dense(n_in)

state = encoder(decoder(encoder(inputs)))

model = Model(input=inputs, output=state)
model.compile('SGD', 'MSE')

SVG(model_to_dot(model).create(prog='dot', format='svg')) 

### 3
n_in, n_out, depth = 10, 5, 1
inputs = Input(shape=(n_in,))
encoder = Dense(n_out, activation='tanh')
decoder = Dense(n_in)

state = decoder(encoder(decoder(encoder(inputs))))

model = Model(input=inputs, output=state)
model.compile('SGD', 'MSE')

SVG(model_to_dot(model).create(prog='dot', format='svg'))

[braingineer]

note: i have no idea if this next one works.. but it compiles (famous last words?). in theory, you could reuse your model in this way:

new_input = Input(shape=(n_in,))
new_depth = 10
new_output = new_input
for _ in range(new_depth):
    new_output = model(new_output)
insane_model = Model(input=new_input, output=new_output)
insane_model.compile('SGD', 'MSE')

[theosem]

Hi All,
I am new to keras and trying to learn how things are done.
did you manage to build an autoencoder with this approach? If you could post a complete example it would be great!
thanks

[braingineer]

Hi @theosem

There is a recent keras blog post on this:

http://blog.keras.io/building-autoencoders-in-keras.html

[dancivitarese]

They show to stack auto encoders and train them automatically. Great find! I think someone should close this issue.

[GeorgesAlkhouri]

Somebody knows if the hidden layers (like in the example ) will be pre trained or not? From my understanding deep auto encoders with no pre trained hidden layers tend to are very inefficient during training.
See https://www.cs.toronto.edu/~hinton/science.pdf

[SwordPL]

Would the layers of the autoencoder be trained end-to-end or layer-wise in Deep Autoencoder from http://blog.keras.io/building-autoencoders-in-keras.html?