Mask_rcnn: Can we replace the "utils.batch_slice()"method with "KL.TimeDistributed()"
As described above, we just want to applies an operation to every batch slice of an input. So can the "utils.batch_slice()"method been replace with a wallpapered defined layer ?
I'm new to keras. Could you please give some suggestion?
Superlee506
All 7 comments
@Superlee506,
I am, by no means, an expert in using Keras but this is my take on the difference between the two methods.
The general purpose of utils.batch_slice() is to gather common elements from different tensors that correspond to specified indices. The purpose of KL.TimeDistributed is to apply the same layer across the temporal dimension of its input. Thus, I think the concept is inherently different between the uses of these two methods.
As examples:
The inputs to the ProposalLayer class is rpn_class (batch, num_anchors_total, 2) and rpn_bbox (batch, num_anchors_total, 4). The inputs to the first call to utils.batch_slice() inside ProposalLayer is then scores (batch, num_anchors_total) and ix (batch, pre_nms_limit). Inside utils.batch_slice(), we want to gather the top pre_nms_limit anchors across all batches as specified by ix, and it does this by using tf.gather. tf.gather typically operates on the first dimension (i.e., the batch dimension) so utils.batch_slice() has a for-loop that loops over one batch at a time and effectively conducts tf.gather between num_anchors_total and pre_nms_limit dimensions instead.
On the other hand, if you look at fpn_classifier_graph(), the output of the PyramidROIAlign method, x, has shape (batch, N, height, width, channel). This is a 5D tensor. We want to apply 2D convolution to x but Keras' Conv2D only accepts 4D tensors and the second dimension of x (i.e., N) is technically the batch dimension for the Conv2D operation. This is where we can use TimeDistributed layers. The output shape of x after the first ReLU operation is (batch, N, 1, 1, 1024) for reference.
FruVirus
on 27 Feb 2018
@FruVirus Thanks for your detailed explanation, this helps me a lot.
Superlee506
on 27 Feb 2018
@FruVirus Hi, Is a loop function along the first dimension (batch) the same with function batch_slice?
Superlee506
on 27 Feb 2018
batch_slice does indeed have a for-loop over the batch. You'll notice that the first line inside the for-loop:
inputs_slice = [x[i] for x in inputs]
effectively selects the non-batch dimensions for each batch
FruVirus
on 28 Feb 2018
@FruVirus Copy that, thanks you again~
Superlee506
on 28 Feb 2018
@Superlee506 , no problem! you ask some good questions
FruVirus
on 28 Feb 2018
@FruVirus Could you please help me with this problem #285 ? I thinks I was confused about batch_slice all the day.
Superlee506
on 28 Feb 2018
Related issues
Mhaiyang
路
4Comments
wjdhuster2018
路
3Comments
simone-codeluppi
路
3Comments
canerozer
路
3Comments
msson
路
4Comments
Most helpful comment
@Superlee506,
I am, by no means, an expert in using Keras but this is my take on the difference between the two methods.
The general purpose of utils.batch_slice() is to gather common elements from different tensors that correspond to specified indices. The purpose of KL.TimeDistributed is to apply the same layer across the temporal dimension of its input. Thus, I think the concept is inherently different between the uses of these two methods.
As examples:
The inputs to the ProposalLayer class is rpn_class (batch, num_anchors_total, 2) and rpn_bbox (batch, num_anchors_total, 4). The inputs to the first call to utils.batch_slice() inside ProposalLayer is then scores (batch, num_anchors_total) and ix (batch, pre_nms_limit). Inside utils.batch_slice(), we want to gather the top pre_nms_limit anchors across all batches as specified by ix, and it does this by using tf.gather. tf.gather typically operates on the first dimension (i.e., the batch dimension) so utils.batch_slice() has a for-loop that loops over one batch at a time and effectively conducts tf.gather between num_anchors_total and pre_nms_limit dimensions instead.
On the other hand, if you look at fpn_classifier_graph(), the output of the PyramidROIAlign method, x, has shape (batch, N, height, width, channel). This is a 5D tensor. We want to apply 2D convolution to x but Keras' Conv2D only accepts 4D tensors and the second dimension of x (i.e., N) is technically the batch dimension for the Conv2D operation. This is where we can use TimeDistributed layers. The output shape of x after the first ReLU operation is (batch, N, 1, 1, 1024) for reference.