Keras: How could we use Leaky ReLU and Parametric ReLU as activation function ?

[gaoyuankidult]

Leaky ReLU and Parametric ReLU are implemented as classes, how could we use them as activation functions ?

Thanks

[patyork]

There's a PReLU example in the Kaggle Otto example; it can be used as a template for all of the Advanced Activation:

from keras.layers.advanced_activations import LeakyReLU, PReLU
..
..
model.add(Dense(512, 512, activation='linear')) # Add any layer, with the default of an identity/linear squashing function (no squashing)
model.add(LeakyReLU(alpha=.001))   # add an advanced activation
...
...
model.add(Dense(512, 123, activation='linear')) # Add any layer, with the default of an identity/linear squashing function (no squashing)
model.add(PReLU((123,)))   # add an advanced activation

[gaoyuankidult]

Thanks @patyork . But what if I would like to use them as inner_activation ?

[patyork]

Hmm, it doesn't appear that that is possible at the moment to use the advanced activation within recurrent layers as the inner activation. @fchollet may be able to say for sure. This is a good point.

[fchollet]

Not currently possible with Keras. It would be possible to modify a recurrent layer to add this capability, though.

[gaoyuankidult]

@patyork @fchollet Adding more flags might violate the principles of this library. Perhaps we need to come up with a good way of doing this.

[wxs]

@fchollet would a reasonable stopgap approach here be to add a "dummy" layer whose get_output() is just the identity, but also exposes the correct PReLU activation as a separate method, say activation()? Then by adding it to your model, nothing changes except that its parameters become part of the gradient update.

Finally: when adding the recurrent layer you can set its activation function to dummy.activation. Does that make sense? Something like this:

dummy = DummyPReLU(512)
model.add(dummy)
model.add(SimpleRNN(512,512, activation=dummy.activation))

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[iamtodor]

The problem is still appearing.
My code:

model.add(Conv2D(filters=512, kernel_size=(3, 3), kernel_initializer='he_normal',
                     padding='same'))
model.add(PReLU())
model.add(BatchNormalization())

I got the following warning: /data/Ilya/projects/whale/env/lib/python3.5/site-packages/keras/activations.py:115: UserWarning: Do not pass a layer instance (such as PReLU) as the activation argument of another layer. Instead, advanced activation layers should be used just like any other layer in a model. identifier=identifier.__class__.__name__))
And then such messages were rolled out:

2018-06-28 22:20:22.411542: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.55GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.

And eventually OOM.
How to handle PReLU?

[kirarpit]

@iamtodor, I think the problem here is that you didn't define any activation function in the first layer.

[naarkhoo]

five years later I have the same question. but seems it is rather a warning now. can you elaborate please.