Caffe: Add a Padding layer to Caffe

[twerdster]

We have developed a padding layer for Caffe similar in functionality to Keras, CoreML and others. The implementation is currently for GPU (Cuda) and CPU and has been working without issue for around a year. Is this likely to be accepted as a PR? If it is then we can make the effort to add the necessary layer tests.

This is what the proto message looks like.

message PaddingParameter {
  enum PadMethod {
    CONSTANT = 0; // constant border fill, can choose what value
    REPEAT = 1;   // repeat the values along the border (i.e. CLAMP)
    CLAMP = 2;    // same as REPEAT
    REFLECT = 3;  // mirror about the edge
    WRAP = 4;     // wrap the values around from the other side
  }
  optional PadMethod pad_method = 1 [default = REPEAT]; // The padding method
  // Explicit padding for every side is given
  // For reference: top left corner is the origin
  optional uint32 pad_l = 2 [default = 0]; // number of columns to add to the left
  optional uint32 pad_r = 3 [default = 0]; // number of columns to add to the right
  optional uint32 pad_t = 4 [default = 0]; // number of rows to add to the top 
  optional uint32 pad_b = 5 [default = 0]; // number of rows to add to the bottom
  optional float pad_value = 6 [default = 0]; // if pad_method is CONSTANT then use this value

  enum Engine {
    DEFAULT = 0;
    CAFFE = 1;
  }
  optional Engine engine = 7 [default = DEFAULT];
}

[twmht]

@twerdster

when do we need padding layer?

[twerdster]

Style transfer filters such as fast-neural-style can make use of it.

Also, the use of symmetric zero padding as the only option available seems to be an unnecessary limitation. Many image processing algorithms require non-zero boundary conditions.

Here are paddings provided in some other frameworks:
Torch
Tensorflow
Keras
CoreML
PyTorch

[twmht]

@twerdster

Can you submit a PR for that?

[twerdster]

@twmht
I would like to first know whether this is something that would be accepted. It requires non trivial effort on our part (which we are happy to do if it has a chance to be accepted) to create the test cases and gradient check tests. Also, it requires adding a new caffe layer ID which has not happened for a considerable period of time within Caffe so I presume this is simply uncommon.

Could you give me some feedback and what you see the chances of this happening?

[twmht]

@twerdster

I think this has a great chance to be accepted, since it has been used in many of frameworks.

So, you have not completed the code?

[twerdster]

@twmht Are you a Caffe maintainer? The padding code works in our networks without issue. The only thing left to build are the layer tests and gradient checks as required for a layer contribution.

[twmht]

@twerdster

I am not a caffe maintainer. I am just interested in your work.

[Noiredd]

@twerdster Sorry for making you wait for the answer.

In general, we want actual processing layers to handle padding themselves (as a part of the algorithm), since an explicit padding layer in most cases would only use up that precious RAM unnecessarily. However, I think an explicit padding layer could be useful - for example in networks that output images (autoencoders, segmentation etc.). I like the various padding modes in particular.

Anyway, we have means to cut from a blob (crop layer), so it would be good to be able to add to it too. If you decide to dedicate your time and effort to implementing the tests, I will gladly review and merge such PR. I suggest you submit it as is for now, even without the tests - so we can take a look at the layer itself while you code the tests.

[ujsyehao]

@Noiredd Hi, Dose caffe still support update?
I implement ResizeBilinear layer in caffe(equal to tf.image.resize_bilinear operation) and I have tested the added layer, Can it be accepted as a PR?

Additionally, caffe only supports symmetric padding, it is different from tensorflow, If I want to modify convolution and pooling padding code to make it support for asymmetric padding(as you say, let actual processing layer to handle padding themselves).
Which part should I modify?

[ujsyehao]

@Noiredd Hi, Can you give me some advice?

[taojiang82]

@ujsyehao Hi, can you share the ResizeBilinear layer code?

[Noiredd]

@ujsyehao If you've got the code working - post a PR and let the community see it. I think such a layer might be interesting to some.

Regarding the asymmetric padding, I don't think this will be easy, especially in the N-D case. @twerdster's solution would come in immensely handy in your situation.

[17cspence]

I just (perhaps stupidly) submitted a pull request for a pad layer, #6506. Now I see this issue. But I've had the code for a while. It only handles the 2D case. As @Noiredd commented, N-D is probably harder.

[ujsyehao]

@taojiang82 I have submitted a PR and you can see the code.

[ujsyehao]

@Noiredd Hi, I have posted a PR, please refer to #6508, Thank you for your reply!

[snownus]

@twerdster , have added the padding layer???

[twerdster]

Unfortunately I wont be adding the layer. Sorry.

On Thu, 23 Aug 2018 at 06:51, snownus [e-mail] wrote:

@twerdster https://github.com/twerdster , have added the padding
layer???

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BVLC/caffe/issues/6294#issuecomment-415278247, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZBQOrld7YMwh0sonPz2BcIolJtTHLvks5uTibXgaJpZM4SsyCf
.

[snownus]

Ok, thanks anyway.

Since padding, batch norm in caffe is different from tensorflow, if we want
to do a fully conversion, it is difficult.

On Thu, Aug 23, 2018 at 1:46 PM Aaron Wetzler [e-mail]
wrote:

Unfortunately I wont be adding the layer. Sorry.

On Thu, 23 Aug 2018 at 06:51, snownus [e-mail] wrote:

@twerdster https://github.com/twerdster , have added the padding
layer???

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/BVLC/caffe/issues/6294#issuecomment-415278247, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AAZBQOrld7YMwh0sonPz2BcIolJtTHLvks5uTibXgaJpZM4SsyCf

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/BVLC/caffe/issues/6294#issuecomment-415298685, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFXEca4QWX91NJ7k_droeTy4GTVMcBIuks5uTkHOgaJpZM4SsyCf
.

[17cspence]

@snownus, as mentioned above, I've submitted PR #6506 for a pad layer. Compared to @twerdster's, my code only allows padding that's the same on all sides. And for modes, twerdster's modes and mine are:
twerdster -- mine
CONSTANT -- ZERO, but that's the only value my code allows, twerdster allows any value
REPEAT or CLAMP -- REPLICATE
REFLECT -- REFLECT and REFLECT_101, don't know which version twerdster implemented
WRAP -- Mine doesn't have this.

For a pad width of 2, my REFLECT does abcdefg... -> baabcdefg ..., while REFLECT_101 does abcdefg... -> cbabcdefg ... The names REPLICATE, REFLECT, and REFLECT_101 are taken from the OpenCV library.

[snownus]

@17cspence ， you mean padding_l == padding_r == padding_t == padding_b?

[17cspence]

@snownus, yes, that's what I mean.

[snownus]

@17cspence ， ok, thanks.

[siyiding1216]

@snownus, as mentioned above, I've submitted PR #6506 for a pad layer. Compared to @twerdster's, my code only allows padding that's the same on all sides. And for modes, twerdster's modes and mine are:
twerdster -- mine
CONSTANT -- ZERO, but that's the only value my code allows, twerdster allows any value
REPEAT or CLAMP -- REPLICATE
REFLECT -- REFLECT and REFLECT_101, don't know which version twerdster implemented
WRAP -- Mine doesn't have this.

For a pad width of 2, my REFLECT does abcdefg... -> baabcdefg ..., while REFLECT_101 does abcdefg... -> cbabcdefg ... The names REPLICATE, REFLECT, and REFLECT_101 are taken from the OpenCV library.

Where did you see twerdster`s code?

[17cspence]

@siyiding1216, I did not see twerdster's code. Enough time has gone by that I'm not quite sure what I was thinking. I suspect I just worded that comment imprecisely. You probably saw twerdster's comment that they won't submit this code:

Unfortunately I wont be adding the layer. Sorry.

[17cspence]

@siyiding1216, to be complete I should mention that my list of twerdster's modes was from the first March 15 post, describing the proto message.

[vikramambrose]

@twerdster or anyone else who maintains this repo.

To be clear, was it the 6506 PR padding layer specifically that was denied or are you not interested adding a padding layer at all even if someone were to write a proper padding layer respecting the new blob vector shape api?

Or would you want to see this functionality added to the conv layer instead?