Keras: How to use TimeDistributed if I have multiple outputs

[xn8812]

I found #3057 which can solve the case of multiple inputs with same feature dimensions.

But how about I have multiple outputs with different shapes? For example, output1: (None, 32, 32, 1), output2: (None, 16, 16, 2), output3: (None, 8, 8, 3)

Thanks!

[xn8812]

@farizrahman4u

[farizrahman4u]

Post your custom layer here.

[xn8812]

Say I first define a base model:

def BASE_MODEL(input_shape=None):
    image_input = Input(shape=input_shape) # default shape could be (None, 224, 224, 3)
    x_1 = Conv2D(64, (3, 3), activation='relu', padding='same')(img_input)
    x_2 = MaxPooling2D((2, 2), strides=(2, 2))(x_1)
    x_2 = Conv2D(64, (3, 3), activation='relu', padding='same')(x_2)
    x_3 = MaxPooling2D((2, 2), strides=(2, 2))(x_2)
    x_3 = Conv2D(64, (3, 3), activation='relu', padding='same')(x_3)
    model = Model(img_input, [x_1, x_2, x_3])
    return model

Then I define another model which is built on the base model.

def NEW_MODEL(input_shape = None):
    base_model = BASE_MODEL(input_shape=input_shape)
    # add one more dimension which represents the temporal axis
    batch_input = Input(shape=(None,)+input_shape)
    output1, output2, output3 = TimeDistributed(base_model)(batch_input)
    ...

Assume the batch_input tensor represents a batch of video frames, the basic idea is that I want to use the base_model to process each frame first and then do some recurrent thing. But I had some problems:

1. When I ran the above code, I have an error message: TypeError: can only concatenate tuple (not "list") to tuple

2. I only need the the the last-frame information of output3, i..e, output3[-1]. How to write it in code? Will some codes like this work?

    ...
    output1, output2, output3 = TimeDistributed(base_model)(batch_input)
    x = Conv2D(128, (3, 3), activation='relu', padding='same')(output3[-1])
   ...

@farizrahman4u Thank you!

[farizrahman4u]

1.

Try this:

def NEW_MODEL(input_shape = None):
    base_model = BASE_MODEL(input_shape=input_shape)
    # add one more dimension which represents the temporal axis
    batch_input = Input(shape=(None,)+input_shape)
    outputs = []
    for out in base_model.output:
        outputs.append(TimeDistributed(Model(base_model.input, out))(batch_input))
    output1, output2, output3 = outputs
    ...

2.

def get_last_timestep(x):
    x_shape = K.shape(x)
    input_length = x_shape[1]
    slices = [slice(None)] * K.ndim(x)
    slices[1] = input_length - 1
    return x[slices]

output3_last_timestep = Lambda(get_last_timestep, output_shape=lambda s: s[:1] + s[2:])(output3)
x = Conv2D(128, (3, 3), activation='relu', padding='same')(output3_last_timestep)

[xn8812]

Didn't have time to try it out until today. But it works! Thank you!

[xn8812]

I found that the solution above might have some problems. Regarding this line of code:

outputs.append(TimeDistributed(Model(base_model.input, out))(batch_input))

In my understanding, Model(base_model.input, out) will define a new model, which means the different outputs are created by different base_models. So when back propagation, the original base_model cannot be updated correctly. Am I right? @farizrahman4u

[StefanL19]

Did you find a solution to this problem? Because considering the keras.plot function output, it really seems that this approach defines two models.

[farizrahman4u]

No, it should be the same model. Note that the BASE_MODEL method is only called once. Keras's topology makes sure that layer's arent re-built and weights aren't reinitialized.

[wt-huang]

Closing as this is resolved