keras mtl

[tarekiraki]

Hello,

I would like to implement the Multi task learning approach via keras.

There is a paper with a good explanation: http://people.eecs.berkeley.edu/~russell/classes/cs294/f05/papers/caruana-1997.pdf ; pdf-doc-page 4, paper-page 44.

My problem looks like:
My input consist of one data structure with three components:
1) picture (450x450 px),
2) label age (1dimensional vector), over 50 or not.
3) label gender (1dimensional vector), men/women.

It's like the mnist example, but here i have two labels instead of one.

So I have one Input and would like to have two outputs. The neural network reuses the inner layers for common parts, and has a specific part at top.

So I would like to implement this, but I dont know how to build the separate specific parts at top. I would use a mlp or an cnn.
Does anyone have an idea or an code example?

Thanks.

[farizrahman4u]

This should be pretty straight forward using the functional api. First you add the common layers to a Sequential model:

feature_extractor = Sequential()
feature_extractor.add(....)
feature_extractor.add(....)

Define your input:

image = Input((450, 450))

Pass the image through the sequential model to get feature vector:

features = feature_extractor(image)

Create models for age and gender classification:

age_classifier = Sequential()
age_classifier.add(...)
age_classifier.add(...)
age_classifier.add(Dense(2, activation='sigmoid'))

gender_classifier = Sequential()
gender_classifier.add(...)
gender_classifier.add(...)
gender_classifier.add(Dense(2, activation='sigmoid'))

Build the final model and compile:

age = age_classifier(features)
gender = gender_classifier(features)

model = Model(input=image, output=[age, gender])

model.compile(loss='binary_crossentropy', optimizer='sgd')

[tarekiraki]

Thank you very much, I think it works well.
But how does the fit-method and the evalute-method looks like with three input? Something like this:

model.fit(X_train, Y_train, Z_train,
batch_size=batch_size, nb_epoch=nb_epoch,
verbose=1, validation_data=(X_test, Y_test, Z_test))

Or just like this:
model.fit(X_train, Y_train,
batch_size=batch_size, nb_epoch=nb_epoch,
verbose=1, validation_data=(X_test, Y_test,))
-> with Y as gender and age?

[farizrahman4u]

model.fit(images, [ages, genders])

[tarekiraki]

I have to say, that I flattened the image-vector via a library-method. So I have a 1-d vector with 450*450 = 202500. So my input looks like this:

Define your input:

image = Input(shape=(202500,))

So when I try to call the 'model.fit' method I get an exception: "Expected to see 1 arrays but instead got the following list of 633 arrays:". So how can I change to have a number of input-pics, not just 1.

Thanks.

[farizrahman4u]

Your image X should be of shape (batch_size, 202500).

[tarekiraki]

Thank you very much for your answers. Your idea works very well. But my score is not that good as I thought:
sec2_acc: 0.77380952097120737
sec3_acc: 0.66666666855887757

Do you have some ideas for improvement? My code is below.
Thanks.

import data_set

from keras.models import Sequential, Model
from keras.layers import Input
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import SGD, Adam, RMSprop
from keras.utils import np_utils

nb_epoch = 2
batch_size = 100

common layers to a Sequential model

feature_extractor = Sequential()
feature_extractor.add(Dense(450, input_shape=(228962,)))
feature_extractor.add(Activation('relu'))
feature_extractor.add(Dropout(0.5))
feature_extractor.add(Dense(450))
feature_extractor.add(Activation('relu'))
feature_extractor.add(Dropout(0.5))
feature_extractor.summary()

Define your input:

image = Input(shape=(228962,))

Pass the image through the sequential model to get feature vector:

features = feature_extractor(image)

Create models for age and gender classification:

age_classifier = Sequential()
age_classifier.add(Dense(450, input_shape=(450,)))
age_classifier.add(Dense(2, activation='sigmoid'))

gender_classifier = Sequential()
gender_classifier.add(Dense(450, input_shape=(450,)))
gender_classifier.add(Dense(2, activation='sigmoid'))

Build the final model and compile:

age = age_classifier(features)
gender = gender_classifier(features)
model = Model(input=image, output=[age, gender])
model.compile(loss='binary_crossentropy', optimizer='sgd')

trainImage, trainAge, trainGender, testImage, testAge, testGender = data_set.read_data_set()
history = model.fit(trainImage, [trainAge, trainGender],
batch_size=batch_size, nb_epoch=nb_epoch,
verbose=1, validation_data=(testImage, [testAge, testGender]))

score = model.evaluate(testImage, [testAge, testGender], verbose=0)
print('score: ', score)

[farizrahman4u]

Try convnet for feature extraction.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.