Keras: jointly train autoencoder and classifier

[FrancisYizhang]

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• [Y] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

• [Y] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [N] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [Y] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

I want to jointly train classifier and autoencoder but I don't kmow how to build the model?

For example, I use VGG16 as classifier, and I want add a autoencoder following the last second convolutional layer since I want to analyse the kernel of last second convolutional layer, but the kernel is too large, hence I try to apply autoencoder to reduce its dimensionality.

I know how to deal with the problem by training classifier and autoencoder independently, but, my model and data is too large.

Could anyone help me?

Thanks in advance!

[FrancisYizhang]

The code is showed below,

import keras
from keras.datasets import mnist
from keras.models import Model
from keras.layers import Conv2D, MaxPooling2D, Dense, BatchNormalization, Activation, Input, Flatten


# download the mnist to the path '~/.keras/datasets/' if it is the first time to be called
(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(x_train.shape[0], 28, 28, 1)
x_test = x_test.reshape(x_test.shape[0], 28, 28, 1)

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, 10)
y_test = keras.utils.to_categorical(y_test, 10)

# data pre-processing
x_train = x_train.astype('float32') / 255. - 0.5       # minmax_normalized
x_test = x_test.astype('float32') / 255. - 0.5         # minmax_normalized

# classifier (main stream)
input_image = Input(shape=(28, 28, 1), dtype='float32')
x_c = Conv2D(32, kernel_size=(3, 3), padding='same')(input_image)
x_c = BatchNormalization()(x_c)
x_c = Activation(activation='relu')(x_c)
x_c = MaxPooling2D(pool_size=(2, 2))(x_c)

x_c = Conv2D(64, kernel_size=(3, 3), padding='same')(x_c)
x_c = BatchNormalization()(x_c)
x_c = Activation(activation='relu')(x_c)
x_c = MaxPooling2D(pool_size=(2, 2))(x_c)

x_c = Flatten()(x_c)

x_c_e = Dense(256, name='x_c_e')(x_c)  # link autoencoder from here  !!!!!!!!!!!
x_c = BatchNormalization()(x_c_e)
x_c = Activation(activation='relu')(x_c)

y_c = Dense(10, activation='softmax', name='y_c')(x_c)


# autoencoder (auxiliary stream)
x_e = Dense(64)(x_c_e)
x_e = BatchNormalization()(x_e)
x_e = Activation(activation='relu')(x_e)
x_e = Dense(10)(x_e)
x_e = BatchNormalization()(x_e)
x_e = Activation(activation='relu')(x_e)
x_e = Dense(2)(x_e)
x_e = BatchNormalization()(x_e)
x_e = Activation(activation='relu')(x_e)

x_e = Dense(10)(x_e)
x_e = BatchNormalization()(x_e)
x_e = Activation(activation='relu')(x_e)
x_e = Dense(64)(x_e)
x_e = BatchNormalization()(x_e)
x_e = Activation(activation='relu')(x_e)
y_e = Dense(256, activation='tanh', name='y_e')(x_e)  # y_e = x_c_e  !!!!!!!!!!!!!!

model = Model(input=input_image, outputs=[y_c, y_e])

model.compile(optimizer='adam', loss=['categorical_crossentropy', 'mse'])

model.summary()

# training
model.fit(x=x_train,
          y=[y_train, model.get_layer('x_c_e').output],
          nb_epoch=100,
          batch_size=32,
          shuffle=True)

I really don't know how to assign x_c_e to y_e and how to compile the model.
Can anyone help me? Thanks in advance!

But, the error is showed below from model.fit(.........

# AttributeError: 'Tensor' object has no attribute 'ndim'

[FrancisYizhang]

My problems are list below:

1. The output (y_e) and input (x_c_e) of autoencoder are the middle layer output (x_c_e) of classifier;
2. There are multiple loss, that is, autoencoder needs mse, but classifier needs categorical_crossentropy.

[FrancisYizhang]

@fchollet
I am sorry to bother you, could you help me?

[brge17]

First: "If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel"

Second: I'll answer your question and give you the code because it is simple/quick.

This is a straightforward adaptation of the mnist classification/autoencoder tutorials:

import keras

from keras.datasets import mnist
from keras.models import Model
from keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, UpSampling2D

batch_size = 128
num_classes = 10
epochs = 12

# input image dimensions
img_rows, img_cols = 28, 28

# Data 
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1).astype('float32') / 255
x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1).astype('float32') / 255
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

# Convolutional Encoder
input_img = Input(shape=(img_rows, img_cols, 1))
conv_1 = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
pool_1 = MaxPooling2D((2, 2), padding='same')(conv_1)
conv_2 = Conv2D(8, (3, 3), activation='relu', padding='same')(pool_1)
pool_2 = MaxPooling2D((2, 2), padding='same')(conv_2)
conv_3 = Conv2D(8, (3, 3), activation='relu', padding='same')(pool_2)
encoded= MaxPooling2D((2, 2), padding='same')(conv_3)

# Classification
flatten = Flatten()(encoded)
fc = Dense(128, activation='relu')(flatten)
softmax = Dense(num_classes, activation='softmax', name='classification')(fc)

# Decoder
conv_4 = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
up_1 = UpSampling2D((2, 2))(conv_4)
conv_5 = Conv2D(8, (3, 3), activation='relu', padding='same')(up_1)
up_2 = UpSampling2D((2, 2))(conv_5)
conv_6 = Conv2D(16, (3, 3), activation='relu')(up_2)
up_3 = UpSampling2D((2, 2))(conv_6)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same', name='autoencoder')(up_3)

model = Model(inputs=input_img, outputs=[softmax, decoded])

model.compile(loss={'classification': 'categorical_crossentropy', 
                    'autoencoder': 'binary_crossentropy'},
              optimizer='adam',
              metrics={'classification': 'accuracy'})

model.fit(x_train, 
          {'classification': y_train, 'autoencoder': x_train},
          batch_size=batch_size,
          epochs=epochs,
          validation_data= (x_test, {'classification': y_test, 'autoencoder': x_test}),
          verbose=1)