Caffe: Setting layer to not back-propagate? propagate_down and lr_mult

[Lehyu]

I have set propagate_down:false in some convolutional layer, but it seems like that doesn't work.
more details: http://stackoverflow.com/questions/40610723/how-do-i-to-prevent-backward-computation-in-specific-layers-in-caffe

[williford]

See:

• https://groups.google.com/forum/#!topic/caffe-users/54Z-B-CXmLE

From Evan:

propagate_down is intended to switch off backprop along certain paths from the loss while not entirely turning off layers earlier in the graph. If gradients propagate to a layer by another path, or regularization such as weight decay is not disabled, the parameters of these layers will still be updated. I suspect decay is still on for these layers, so you could set decay_mult: 0 for the weights and biases.

Setting lr_mult: 0 on the other hand fixes parameters and skips backprop where it is unnecessary.

If this doesn't address your issue, please ask your question on the caffe-users mailing list or stackoverflow and provide more information - like what you are trying to accomplish, your model prototxt and what makes you think it is not working.

[shelhamer]

Leaving this open as a reminder to better document the flags for controlling backprop.

[williford]

@shelhamer Where should this be documented? Is there currently documentation of the prototxt parameters? I'm guessing the closest thing is currently doxygen comments in the headers? BTW, I have started working on improving the Caffe tutorial layer catalogue documentation ... but I have a long ways to go still.

[shelhamer]

Most of the less obvious settings are documented in the proto itself.

I think this belongs somewhere in the tutorial documentation, although that needs to be re-arranged to better channel users to the various parts as I think most miss it entirely. Plus it needs to be brought up to date.

Re: the layer catalogue documentation, we should review what belongs there vs. in the API/doxygen documentation. Duplication is just more effort, and we don't need any more friction for contributing documentation.

[williford]

@shelhamer Could you check out #5148 and give me your general thoughts?

It links to the appropriate documentation elsewhere (doxygen, caffe.proto, header files, etc). Perhaps the more labor-intensive user-friendly documentation should be stripped?

For example: http://git.caffe.help/tutorial/layers/convolution.html

I am also hosting an alternative version here: http://caffe.help/manual/layers/convolution.html (repoy: https://github.com/caffe-help/caffe.help-website). I could move the samples and references to blogs, etc. to just this repository (I can make a special Jekyll page layout that includes additional documentation if the corresponding file exists).

It still doesn't really solve this issue yet.