Keras: Removing layers with layers.pop() doesn't work?

[aristid]

I'm trying to start https://github.com/iamaaditya/VQA_Demo which uses pre-trained VGG16 model (https://github.com/iamaaditya/VQA_Demo/blob/master/models/CNN/VGG.py). The last 2 layers are removed with layers.pop(). This doesn't seem to work, however. The message "ValueError: could not broadcast input array from shape (1000) into shape (4096)" is displayed, and I see the layers were not removed when I do "plot(image_model, to_file='model_vgg.png', show_shapes=True)". Thank you.

[joelthchao]

If you want to fine-tune this model, you should add new dense layer after popping the original one.

[aristid]

The original author does that; here is the notebook: https://github.com/iamaaditya/VQA_Demo/blob/master/Visual_Question_Answering_Demo_in_python_notebook.ipynb

Image features are supposed to be merged with language features, but it doesn't come to that because of the error I mentioned. The issue is, the model still uses the popped layers, which can be verified by plotting it.

If I comment out "model = model.model" (line 21 of keras/utils/visualize_util.py), the right model without the last layers is plotted. But I don't know what it means or how to let Keras know the layers have changed.

Additionally, I just verified it doesn't happen with Keras 0.3.1, the layers were dropped correctly there.

[somewacko]

I actually ran into this recently, and I believe that this is an issue with how Model was rewritten to be more graph-based. In addition to the layers property models also have an outputs property, so when you pop layers off your last layer is left dangling in the model's output (which is why your model still thinks its output size is 1000).

A method to pop layers off and manage all the links correctly is needed, but until then I believe this would work: (my model compiles, but I haven't tried training it yet)

# ... Load pre-trained VGG16 model

model.layers.pop() # Get rid of the classification layer
model.layers.pop() # Get rid of the dropout layer
model.outputs = [model.layers[-1].output]
model.layers[-1].outbound_nodes = []

[aristid]

Thank you. I downgraded to 0.3.1 where it works, so can't try your fix right now. I'm not sure it's all there is, because if I'm not mistaken plot() would draw all the popped layers (I tried popping many layers which all re-appeared), so maybe the copy of model.layers is stored somewhere.

As a side note, model.layers.pop() looked like a hack anyway. We did not push the layers there and should not expect to pop them without side effects. Maybe something like model.remove() would make more sense as a pendant to model.add().

[albertomontesg]

@fchollet It could be possible to add into a Sequential model the possibility to pop the last layer and updating the output and outbound_nodes?

[joelthchao]

@albertomontesg like this?

def pop(self):
    '''Removes a layer instance on top of the layer stack.
    '''
    if not self.outputs:
        raise Exception('Sequential model cannot be popped: model is empty.')
    else:
        self.layers.pop()
        if not self.layers:
            self.outputs = []
            self.inbound_nodes = []
            self.outbound_nodes = []
        else:
            self.layers[-1].outbound_nodes = []
            self.outputs = [self.layers[-1].output]
        self.built = False

[albertomontesg]

@joelthchao Yes. That was exactly what I was thinking.

[aristid]

@joelthchao seems to work fine, indeed. Thanks!

P.S. This particular example runs into #2386 after that, so it works under 0.3.1 only.

[iamaaditya]

@joelthchao Thanks for the idea.
@aristid I have changed the code to work with Keras version 1+

This issue can be considered close.

[shaayaansayed]

can confirm I'm still having this issue on keras 1.0.6.

[farizrahman4u]

3252

[m1sk]

I'm having a related problem #3310.

[keunwoochoi]

Same.

[SlimFrikha]

I encountered the same problem and I managed to pop the layers like this:

from convnetskeras.convnets import AlexNet
alexnet = AlexNet(weights_path="weights/alexnet_weights.h5")
alexnet.layers.pop() # Get rid of the classification layer softmax
alexnet.outputs = [alexnet.layers[-1].output]
alexnet.output_layers = [alexnet.layers[-1]] # added this line in addition to zo7 solution
alexnet.layers[-1].outbound_nodes = []

Hope this helps!
(run on keras version 1.2.0)

[alyato]

@joelthchao I wanna use this pretrained vgg19.

x = Flatten(name='flatten')(x)
x = Dense(4096, activation='relu', name='fc1')(x)
x = Dense(4096, activation='relu', name='fc2')(x)
x = Dense(1000, activation='softmax', name='predictions')(x)
model = Model(img_input, x)
return model

i have only 8 labels to classlify.
But it is used by the keras function API ,not the Sequential model.
how can i use the model.layers.pop()
like this

x = Dense(1000, activation='softmax', name='predictions')(x)
model = Model(img_input, x)
model.layers.pop()
x = Dense(8, activation='softmax', name='predictions')(x)
loading weight
return model

Do you give me some advices? Thanks

[joelthchao]

@alyato No need to pop, just find the layer you want to connect to, connect to another layer and make another new model with correct input and output.

[joelthchao]

Snippet for your question:

from keras.models import Model
from keras.layers import Dense, Input

# Assume we have a pretrained model
net_input = Input(shape=(10))
net = Dense(4) (net_input)
net_output = Dense(1) (net)
model1 = Model(net_input, net_output)

# Get input
new_input = model1.input
# Find the layer to connect
hidden_layer = model1.layers[-2].output
# Connect a new layer on it
new_output = Dense(2) (hidden_layer)
# Build a new model
model2 = Model(new_input, new_output)

And this is the result

model1:
____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
input_1 (InputLayer)             (None, 10)            0
____________________________________________________________________________________________________
dense_1 (Dense)                  (None, 4)             44          input_1[0][0]
____________________________________________________________________________________________________
dense_2 (Dense)                  (None, 1)             5           dense_1[0][0]
====================================================================================================

model2:
____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
input_1 (InputLayer)             (None, 10)            0
____________________________________________________________________________________________________
dense_1 (Dense)                  (None, 4)             44          input_1[0][0]
____________________________________________________________________________________________________
dense_3 (Dense)                  (None, 2)             10          dense_1[0][0]
====================================================================================================

[somewacko]

Also, there's an available method model.get_layer(name, idx) for easily getting intermediate layers out of a model.

https://keras.io/models/model/#methods

[alyato]

@joelthchao Thanks. I run it well. But I also don't know how to get the index of every layer.
Do i use the model.get_config() to count the index of every layer

[joelthchao]

@alyato Sort of, I use [x.name for x in model.layers] to find the index manually.

[srv902]

@joelthchao

Hi, I am trying to append vggnet-16 to an existing ResNet model, but after executing Pop() written by you, the output shape value is Multiple. How do I resolve this?

[joelthchao]

@srv902 Is it a sequential model or functional API model?

[srv902]

@joelthchao It is a functional API model.

[joelthchao]

@srv902 If the layer you want to connect to has multiple outputs, then you can only choose one output to connect. (I bet you are trying to connect to residual block.) BTW, you can provide a code snippet to help others to understand your problem.

[OnlyBelter]

Still has this problem!

[OnlyBelter]

Below method works:

from keras.models import Model
from keras.layers import Dense,Flatten
from keras.applications import vgg16
from keras import backend as K

model = vgg16.VGG16(weights='imagenet', include_top=True)

model.input

model.summary(line_length=150)

model.layers.pop()
model.layers.pop()

model.summary(line_length=150)

inp = model.input
out =model.layers[-1].output

model2 = Model(inp, out)  # create a new model which doesn't have the last two layers in VGG16
model2.summary(line_length=150)

reference here

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[hlfshell]

I can confirm that keras 1.1.0 model.pop() does not remove the last layer.

Found this thread due to running into the issue myself.

Code with the error:

vgg = Vgg16()

vgg.model.summary()

vgg.model.layers.pop()

vgg.models.summary()

Edit:

I got the pop to work by calling vgg.model.pop instead of vgg.model.layers.pop, as it is a Sequential model.

[winzee]

I think I have a similar problem. I am running _Keras 2.0.6_ with _Python 3.6.2_ and _Tensorflow-gpu 1.3.0_.

I run this code after having hand built a vgg16 architecture and loaded the weights:

        model = self.model
        model.pop()
        for layer in model.layers: layer.trainable=False
        model.add(Dense(num, activation='softmax'))
        self.compile()

And when I check the graph in Tensorboard I see (check top left of attached picture) dense_3 connected to dropout_2 but dangling by itself. And then next to it I see dense_4, also connected to dropout_2.

Curiously, I get a low accuracy when running this on the Kaggle cats vs dog competition where I hover around 90% whilst others running this code (it's adapted from fast.ai) on top of Theanos easily get 97%. Perhaps my accuracy problem comes from somewhere else, but I still don't think dense_3 should be dangling there and I am wondering if this could be the source of my precision problem.

Secondary question: how could I definitely disconnect and remove dense_3?

[AllardJM]

@winzee

Have you tried this:

model = self.model
model.pop()

add these

https://github.com/flyyufelix/cnn_finetune/blob/master/vgg16.py

model.layers[-1].outbound_nodes = []
model.outputs = [model.layers[-1].output]

for layer in model.layers: layer.trainable=False
model.add(Dense(num, activation='softmax'))
self.compile()

[mrgloom]

@zo7 Can you comment on outbound_nodes ?

i.e. why this code is not sufficient?

model.layers.pop() 
model.outputs = [model.layers[-1].output]

[dlhocker]

I am having a similar issue trying to pop the last layer of a ResNet50 model and add a new classifier. Using Keras version 2.2.0 with tensorflow backend. The previous fix in this thread for dealing with the outbound nodes and output is producing an error that I haven't seen mentioned yet. Is there a smarter way to pop the classification layer and add a new one, or a way to deal with this attribute error ?

from keras.models import Model
from resnet50 import ResNet50

model = ResNet50(weights='imagenet', include_top=True)`
model.layers.pop() # Get rid of the classification layer
model.outputs = [model.layers[-1].output]
model.output_layers = [model.layers[-1]] # added this line in addition to zo7 solution
model.layers[-1].outbound_nodes = []

This is the output

AttributeError                            Traceback (most recent call last)
<ipython-input-3-8d9365679847> in <module>()
     16 model.outputs = [model.layers[-1].output]
     17 model.output_layers = [model.layers[-1]] # added this line in addition to zo7 solution
---> 18 model.layers[-1].outbound_nodes = []

AttributeError: can't set attribute

As an alternative, I know that the include_top=False option will remove the classifier (and the Flatten layer), but I don’t know how to correctly attach the output of this form of ResNet to a new classifier (e.g., using model.layers[-1].output ?). Any advice on this?

[juanklopper]

pop() still does not work (Win 10, GPU Keras on CNTK backend, newest as of 2018-06-20). To be more specific, when a model is copied from a known model and is of type Sequential, it does not work. However, pop() will work on the original Model type model, so use it there to remove the last layer before creating a new Sequential type model from the original layers, i.e. vgg16.

[WilliamHoo]

pls fix this problem as soon as possible, please i need this!

[mrgloom]

https://stackoverflow.com/a/49403298/1179925

For some reason I need to build model with popped layer using Model before adding new layers to make things work.

def pop_layer(model):
    if not model.outputs:
        raise Exception('Sequential model cannot be popped: model is empty.')

    model.layers.pop()
    if not model.layers:
        model.outputs = []
        model.inbound_nodes = []
        model.outbound_nodes = []
    else:
        model.layers[-1].outbound_nodes = []
        model.outputs = [model.layers[-1].output]
    model.built = False

def get_model():
    #Fully convolutional part of VGG16
    model = VGG16(include_top=False, weights='imagenet')

    #Remove last max pooling layer
    pop_layer(model)

    #Freeze pretrained layers
    for layer in model.layers:
        layer.trainable = False

    model = Model(inputs=model.inputs, outputs=model.outputs)

    print('len(model.layers)', len(model.layers)) #
    print(model.summary()) #

    x = GlobalAveragePooling2D()(model.output)
    head = Dense(N_CLASS, activation='softmax')(x)

    model = Model(inputs=model.inputs, outputs=head)

    model.compile(optimizer=Adadelta(), loss='categorical_crossentropy', metrics=['accuracy'])

    print('len(model.layers)', len(model.layers)) #
    print(model.summary()) #

    return model

[Rajchowdhury420]

Sleep(1)

[amandeep25]

Hi,
I am using pre-trained VGG16 model and wanted to remove the last layer used in VGG16 for classification
For my code this worked:

model._layers.pop()
model.summary()

[llealgt]

I have a similar case(https://stackoverflow.com/questions/63202651/how-to-remove-layers-from-a-keras-model-in-order-to-use-as-baseline-for-creating) using model._layers.pop() as @amandeep25 posted seems to work (model.summary() returns desired result) however when checking tensorboard or exporting the model to .pb (final goal is to compile for deployment on raspberry) the original VGG16 model is still there. Any suggestions?

[hafiz031]

@somewacko is there any way to pop a range of layers using indices? I mean without using loop or calling the same pop() function multiple times?

[somewacko]

I believe model.layers is just a list, you can probably do:

model.layers = model.layers[:-3]  # everything but the last 3 layers