Centernet: question about data augmentation

[LightToYang]

This is a good work!

When I read the data augmentation code, I met the question as follows

the affine_transform performed on img is based on input size

trans_input = get_affine_transform(
      c, s, 0, [input_w, input_h])

but it is performed on bbox anno based on output size, I don't understand the purpose, any explanation, please?

output_h = input_h // self.opt.down_ratio
output_w = input_w // self.opt.down_ratio
num_classes = self.num_classes
trans_output = get_affine_transform(c, s, 0, [output_w, output_h])

bbox[:2] = affine_transform(bbox[:2], trans_output)
bbox[2:] = affine_transform(bbox[2:], trans_output)
bbox[[0, 2]] = np.clip(bbox[[0, 2]], 0, output_w - 1)
bbox[[1, 3]] = np.clip(bbox[[1, 3]], 0, output_h - 1)

[LightToYang]

It seems that random scaling and cropping would not work together? The default setting in the paper is rand cropping rather than scaling ?

      if not self.opt.not_rand_crop:
        s = s * np.random.choice(np.arange(0.6, 1.4, 0.1))
        w_border = self._get_border(128, img.shape[1])
        h_border = self._get_border(128, img.shape[0])
        c[0] = np.random.randint(low=w_border, high=img.shape[1] - w_border)
        c[1] = np.random.randint(low=h_border, high=img.shape[0] - h_border)
      else:
        sf = self.opt.scale
        cf = self.opt.shift
        c[0] += s * np.clip(np.random.randn()*cf, -2*cf, 2*cf)
        c[1] += s * np.clip(np.random.randn()*cf, -2*cf, 2*cf)
        s = s * np.clip(np.random.randn()*sf + 1, 1 - sf, 1 + sf)

[xingyizhou]

Yes. rand_crop is from CornerNet. The purpose is resize the input to 512x512 and convert the points to the corresponding position at the 128x128 heatmap.

[LightToYang]

Would you please solve my first question? I still have no idea about the reason why the transform performed on bbox anno based on output size rather than input size.

[niaoyu]

i have the same issue too: Why the input transform is just used to transform the input image only?

@xingyizhou

[shuoshuoa]

i have the same issue too...
@xingyizhou

[LightToYang]

187

[niaoyu]

187

The problem is why "trans_output" is used , not "trans_input"

[HamsterHuey]

The reason it is done is as follows:

• The input image to the network is of dimension (input_w, input_h) which is typically (512, 512). So this is why the affine transform on the image is setup to output an image of dimension (input_w, input_h) which will be fed into the network (DLA/Hourglass/Resnet, etc)
• To reduce memory load, the input images in all models are first reduced by a factor of 4 by a couple of conv layers before being passed through the DLA/Hourglass/Resnet architecture. You can see this clearly in the network diagrams in the appendix in the paper. As a result, the output heatmap generated by the network is of dimension (output_w, output_h) where output_w = input_w // 4 and output_h = input_h // 4.
• Since the heatmaps predicted by the network for center-points are of dimension (output_w, output_h), the ground-truth heatmap for annotations are also generated via identical affine transform as on the image, but to an output dimension of (output_w, output_h). This allows 1-to-1 mapping between the ground-truth heatmap and the network predicted heatmaps and helps make the actual loss calculations simpler and more efficient
• Finally, the entire reason we have an offset regression for x,y is to account for the inaccuracy in center-point estimation that results due to the 4X reduction in resolution of the heatmap in relation to the input image. It helps recover better accuracy on the predicted center point.

Hope this helps