Keras: model.layers.pop() is broken - deleted layer reappears
I am using keras implementation of wrn network from here: https://github.com/asmith26/wide_resnets_keras but as far as i know, the problem appears for every transfer learning attempt.
model = wrn.create_wide_residual_network(init_shape, nb_classes=10, N=4, k=8, dropout=0.0)
model.compile(loss='categorical_crossentropy', optimizer="adadelta", metrics=['accuracy'])
model.summary()
model.layers.pop()
last = model.output
preds = Dense(nb_classes=47, activation='softmax', name="some_generic_name")(last)
model = Model(model.input, preds)
model.compile(loss='categorical_crossentropy', optimizer="adadelta", metrics=['accuracy'])
model.summary()
So, the end of first summary is:
flatten_1 (Flatten) (None, 512) 0
____________________________________________________________________________________________________
dense_1 (Dense) (None, 10) 5130
====================================================================================================
Than after removing classification head everything is OK:
flatten_1 (Flatten) (None, 512) 0
====================================================================================================
but after calling it once the new classification head is added, the old one reappears.
flatten_1 (Flatten) (None, 512) 0
____________________________________________________________________________________________________
dense_1 (Dense) (None, 10) 5130
____________________________________________________________________________________________________
some_generic_name (Dense) (None, 47) 517
====================================================================================================
The https://github.com/fchollet/keras/issues/2371 custom layer.pop doesn't solve this issue.
Huxwell
All 6 comments
last = model.layers[-1].output
If your model is Sequential, please use model.pop().
joelthchao
on 12 Apr 2017
It happens to work OK with model.layers[-2].output (without pop), thank you very much for the hint!
Although I still think that that layer 'being deleted, but not being deleted' is a bug and pop should either throw some exception if used with sequential or work with api in predictable manner.
Huxwell
on 12 Apr 2017
model.layers.pop is just a list pop, while model.output isn't taken care and replaced with the real output you want. That's why you will get the popped layer again
joelthchao
on 13 Apr 2017
I was confused by the pop and concatenate operation. I tried to pop the last layer of two VGG models and concatenate the last layers of the new model. When I try use model.summary I do not understand the structure. It seems I add new layer for concatenate operation. But the concatenate layer has 0 parameter. Could you help me to explain that?
Here is my model:
model1 = keras.applications.vgg16.VGG16(include_top=True, weights='imagenet',
input_tensor=None, input_shape=None,
pooling=None,
classes=1000)
model1.layers.pop()
model2 = keras.applications.vgg16.VGG16(include_top=True, weights='imagenet',
input_tensor=None, input_shape=None,
pooling=None,
classes=1000)
for layer in model2.layers:
layer.name = layer.name + str("two")
model2.layers.pop()
featureLayer1 = model1.layers[-1].output
featureLayer2 = model2.layers[-1].output
combineFeatureLayer = keras.layers.concatenate([featureLayer1, featureLayer2])
predictionLayer = Dense(1, activation='sigmoid', name='final_prediction')(combineFeatureLayer)
model = Model(inputs=[model1.input, model2.input], outputs= [predictionLayer])
Here is the summary:
`Layer (type) Output Shape Param # Connected to
====================================================================================================
input_1 (InputLayer) (None, 224, 224, 3) 0
____________________________________________________________________________________________________
input_2two (InputLayer) (None, 224, 224, 3) 0
____________________________________________________________________________________________________
block1_conv1 (Conv2D) (None, 224, 224, 64) 1792
____________________________________________________________________________________________________
block1_conv1two (Conv2D) (None, 224, 224, 64) 1792
____________________________________________________________________________________________________
block1_conv2 (Conv2D) (None, 224, 224, 64) 36928
____________________________________________________________________________________________________
block1_conv2two (Conv2D) (None, 224, 224, 64) 36928
____________________________________________________________________________________________________
block1_pool (MaxPooling2D) (None, 112, 112, 64) 0
____________________________________________________________________________________________________
block1_pooltwo (MaxPooling2D) (None, 112, 112, 64) 0
____________________________________________________________________________________________________
block2_conv1 (Conv2D) (None, 112, 112, 128) 73856
____________________________________________________________________________________________________
block2_conv1two (Conv2D) (None, 112, 112, 128) 73856
____________________________________________________________________________________________________
block2_conv2 (Conv2D) (None, 112, 112, 128) 147584
____________________________________________________________________________________________________
block2_conv2two (Conv2D) (None, 112, 112, 128) 147584
____________________________________________________________________________________________________
block2_pool (MaxPooling2D) (None, 56, 56, 128) 0
____________________________________________________________________________________________________
block2_pooltwo (MaxPooling2D) (None, 56, 56, 128) 0
____________________________________________________________________________________________________
block3_conv1 (Conv2D) (None, 56, 56, 256) 295168
____________________________________________________________________________________________________
block3_conv1two (Conv2D) (None, 56, 56, 256) 295168
____________________________________________________________________________________________________
block3_conv2 (Conv2D) (None, 56, 56, 256) 590080
____________________________________________________________________________________________________
block3_conv2two (Conv2D) (None, 56, 56, 256) 590080
____________________________________________________________________________________________________
block3_conv3 (Conv2D) (None, 56, 56, 256) 590080
____________________________________________________________________________________________________
block3_conv3two (Conv2D) (None, 56, 56, 256) 590080
____________________________________________________________________________________________________
block3_pool (MaxPooling2D) (None, 28, 28, 256) 0
____________________________________________________________________________________________________
block3_pooltwo (MaxPooling2D) (None, 28, 28, 256) 0
____________________________________________________________________________________________________
block4_conv1 (Conv2D) (None, 28, 28, 512) 1180160
____________________________________________________________________________________________________
block4_conv1two (Conv2D) (None, 28, 28, 512) 1180160
____________________________________________________________________________________________________
block4_conv2 (Conv2D) (None, 28, 28, 512) 2359808
____________________________________________________________________________________________________
block4_conv2two (Conv2D) (None, 28, 28, 512) 2359808
____________________________________________________________________________________________________
block4_conv3 (Conv2D) (None, 28, 28, 512) 2359808
____________________________________________________________________________________________________
block4_conv3two (Conv2D) (None, 28, 28, 512) 2359808
____________________________________________________________________________________________________
block4_pool (MaxPooling2D) (None, 14, 14, 512) 0
____________________________________________________________________________________________________
block4_pooltwo (MaxPooling2D) (None, 14, 14, 512) 0
____________________________________________________________________________________________________
block5_conv1 (Conv2D) (None, 14, 14, 512) 2359808
____________________________________________________________________________________________________
block5_conv1two (Conv2D) (None, 14, 14, 512) 2359808
____________________________________________________________________________________________________
block5_conv2 (Conv2D) (None, 14, 14, 512) 2359808
____________________________________________________________________________________________________
block5_conv2two (Conv2D) (None, 14, 14, 512) 2359808
____________________________________________________________________________________________________
block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808
____________________________________________________________________________________________________
block5_conv3two (Conv2D) (None, 14, 14, 512) 2359808
____________________________________________________________________________________________________
block5_pool (MaxPooling2D) (None, 7, 7, 512) 0
____________________________________________________________________________________________________
block5_pooltwo (MaxPooling2D) (None, 7, 7, 512) 0
____________________________________________________________________________________________________
flatten (Flatten) (None, 25088) 0
____________________________________________________________________________________________________
flattentwo (Flatten) (None, 25088) 0
____________________________________________________________________________________________________
fc1 (Dense) (None, 4096) 102764544
____________________________________________________________________________________________________
fc1two (Dense) (None, 4096) 102764544
____________________________________________________________________________________________________
fc2 (Dense) (None, 4096) 16781312
____________________________________________________________________________________________________
fc2two (Dense) (None, 4096) 16781312
____________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 8192) 0
____________________________________________________________________________________________________
final_prediction (Dense) (None, 1) 8193
====================================================================================================`
What I want to do is to concatenate fc2 and fc2two layers . But it gives me a new concatenate_1 layer. Why this happened? Also, did pop really work for tensor? I read your comment and it seems pop do not work for tensor
sl1316
on 19 Apr 2017
It seems I add new layer for concatenate operation. But the concatenate layer has 0 parameter.
You do things correctly. concatenate layer don't have any weights as expected.
What I want to do is to concatenate fc2 and fc2two layers . But it gives me a new concatenate_1 layer.
The concatenate_1 layer just concatenate two input to form a abstract layer.
Also, did pop really work for tensor?
Sequential.pop() pop the last layer in model.layers and take care all the model includes output.
List.pop() (what you do here) pop the last layer in model.layers and don't care about the original model. What we need to do is use the model.layers[-1] instead of model.output to form a new model, which is what you do here (correctly).
joelthchao
on 19 Apr 2017
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 19 Jul 2017
Related issues
braingineer
路
3Comments
kylemcdonald
路
3Comments
vinayakumarr
路
3Comments
rantsandruse
路
3Comments
anjishnu
路
3Comments
Most helpful comment
last = model.layers[-1].outputIf your model is
Sequential, please usemodel.pop().