Flux.jl: [feature request] L1, L2 regularization of weights

Created on 30 Jan 2018 · 7Comments · Source: FluxML/Flux.jl

The ability to L1- or L2- regularize the coefficients of the weights of the neural network would be very helpful for my research.

L2-regularization is easy, but L1- regularization I think requires the soft-thresholding operator. I initiated a discussion here that adding the absolute value of the magnitude of the coefficients to the loss is not appropriate, but I'm not 100% sure.

[Hopefully it's okay for users to request features; I am too eager to start using Flux.jl so I can do all of my research in Julia!]

Source

SimonEnsemble

Most helpful comment

Both of these are pretty easy to do:

using Flux

m = Chain(Dense(10, 5, σ), Dense(5, 2), softmax)

l1(x) = sum(x.^2)
l2(x) = sum(abs.(x))

sum(l1, params(m)) # Add this to your loss

We should probably add a note to the docs so it's more obvious that we have it, though :)

MikeInnes on 30 Jan 2018

👍2

All 7 comments

Both of these are pretty easy to do:

using Flux

m = Chain(Dense(10, 5, σ), Dense(5, 2), softmax)

l1(x) = sum(x.^2)
l2(x) = sum(abs.(x))

sum(l1, params(m)) # Add this to your loss

We should probably add a note to the docs so it's more obvious that we have it, though :)

MikeInnes on 30 Jan 2018

👍2

Oh, awesome! [l1, l2 are switched though, I think]. That would be great to add in the docs as an example. Also, if possible, to regularize the outputs of the neuron.

But as my Stack Overflow question suggests, I think the L1 regularization needs the soft threshold operator to bring the coefficients to exactly zero when they are close enough.

SimonEnsemble on 30 Jan 2018

Just skimming over this right now, but it looks like soft-threshold is a scalar function that you'd broadcast? In which case it should be trivial to define it, and broadcasting it will just work with our AD. Happy to help if you can't get that working.

MikeInnes on 30 Jan 2018

By the way, is it possible to integrate Distances.jl then gain tons of function like l2, l1, kl divergance?

iblis17 on 31 Jan 2018

👍1

Possibly, although it looks like an unnecessarily heavy API around call and broadcast from where I'm standing. I'd be happy to just add the definitions to Flux though.

MikeInnes on 9 Feb 2018

I've addressed this for now by documenting how to do it in the manual. I'm happy to help with the soft thresholding thing as well, if I can get some more detail on it.

MikeInnes on 9 Feb 2018

@SimonEnsemble
Yes, NNs do a bad job with L1 regularization by just adding it to the loss.
You could implement soft-thresholding via the call back. After each update step, you can soft-threshold the weights yourself.
Do you know any other way?