Keras: i want to use SVM as last layer of CNN any suggestion?

[myk1603]

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• [x] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [ ] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [ ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[unrealwill]

Hello,
Have you tried this ? -> https://github.com/fchollet/keras/issues/2588

[myk1603]

@unrealwill #2588 does not have any information about how to use as SVM last layer of CNN

[unrealwill]

In #2588 it suggest to end your model with :

model.add(Dense(nb_classes), W_regularizer=l2(0.01))
model.add(Activation('linear'))

model.compile(loss='hinge',
              optimizer='adadelta',
              metrics=['accuracy'])

You can adapt it to CNN by using a kernel size of 1 for conv1d or (1,1) for conv2d, use the appropriate regularizer and adding a flattening layer afterwards and you should be good to go :

(untested not even runned )

model.add(ConvxD(nb_classes,kernel_size=(1,...) '), kernel_regularizer=l2(0.01), bias_regularizer=l2(0.01))
#### model.add(Activation('linear')) Not necessary because by default Conv use linear activation by default 
model.add(Flatten())
model.compile(loss='hinge',
              optimizer='adam',
              metrics=['accuracy'])

[myk1603]

i want to train a neural network, then select one of the first fully connected one, run the neural network on my dataset, store all the feature vectors, then train an SVM with a different library (e.g sklearn).

[unrealwill]

Ok, then you might find this useful
https://keras.io/getting-started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer

Additional hint, you can have two models : one Model for training and one Model for exporting the intermediate layer. As long as the models share their weights, the training will modify the the second model use for prediction of the intermediate layer.

[keunwoochoi]

That's what people do a lot in transfer learning including me in this work, take a look on codes. Basically, 1) do what @unrealwill linked, 2) store all of them 3) do SVM using scikit-learn.

[myk1603]

Thanks @unrealwill @keunwoochoi

How can i solve this issue
when i try to store features i got error:
get_features = K.function([K.learning_phase(), model.layers[0].input], [model.layers[9].output])

X is the data of all the input samples

layer_output = get_features([X])[0]
Traceback (most recent call last):

File "", line 3, in
layer_output = get_features([X])[0]

File "C:\Users\user\Anaconda2\lib\site-packages\keras\backend\theano_backend.py", line 959, in __call__
return self.function(*inputs)

File "C:\Users\user\Anaconda2\lib\site-packages\theano\compile\function_module.py", line 795, in __call__
allow_downcast=s.allow_downcast)

File "C:\Users\user\Anaconda2\lib\site-packages\theano\tensor\type.py", line 178, in filter
data.shape))

TypeError: Bad input argument to theano function with name "C:\Users\user\Anaconda2\lib\site-packages\keras\backend\theano_backend.py:955" at index 0 (0-based).
Backtrace when that variable is created:

File "C:\Users\user\Anaconda2\lib\site-packages\ipykernel\zmqshell.py", line 501, in run_cell
return super(ZMQInteractiveShell, self).run_cell(args, *kwargs)
File "C:\Users\user\Anaconda2\lib\site-packages\IPython\core\interactiveshell.py", line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\user\Anaconda2\lib\site-packages\IPython\core\interactiveshell.py", line 2821, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\user\Anaconda2\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 7, in
from keras.models import Model
File "C:\Users\user\Anaconda2\lib\site-packages\keras__init__.py", line 2, in
from . import backend
File "C:\Users\user\Anaconda2\lib\site-packages\keras\backend__init__.py", line 64, in
from .theano_backend import *
File "C:\Users\user\Anaconda2\lib\site-packages\keras\backend\theano_backend.py", line 23, in
_LEARNING_PHASE = T.scalar(dtype='uint8', name='keras_learning_phase') # 0 = test, 1 = train
Wrong number of dimensions: expected 0, got 2 with shape (2515L, 120000L).

[myk1603]

Hello @keunwoochoi @unrealwill
data input for CNN
X_train = (2012L, 3L, 200L, 200L)
Y_train = (2012L, 2L) #label after encoding for CNN

Store features after CNN training

get_activations = K.function([model.layers[0].input, K.learning_phase()], [model.layers[9].output,])
feature = get_activations([X_train,0])

Now I have feature vector

features = float (2012L, 128L)

y_train = (2012L,)

I want to use this feature vector with SVM anyone can help me?

currently, I am trying this code

svcClf = SVC(C=1.0,kernel="rbf",cache_size=975)
scaler = MinMaxScaler() #transform features by scaling each features to a given range
feature = scaler.fit_transform(feature)
svcClf.fit(feature, y_train)

But I got error

with scaler
ValueError: Found array with dim 3. MinMaxScaler expected <= 2.
Without scaler
ValueError: Found array with dim 3. Estimator expected <= 2.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.