Vision: [Feature Request] Target Transforms for keypoints in Image

[Naman-ntc]

Torchvision's functional API allows user to explicitly specify angles to rotate the image or points to crop the image. Thus one can apply sane augmentation on target masks or bounding boxes.
But in cases where we have the output as keypoints in the image, for example say human body joint coordinates in given image, if I rotate the image I'll have to find corresponding joint locations in image (via elementary geometry). I felt that this effort off applying corresponding keypoints in augmented image could be handled in pytorch backend and would be very helpful.
Do you think it's worth it to add in torchvision package?
This could come handy in object detection, human joint annotations and maybe even more place

[fmassa]

Yes, this is useful will be included in torchvision in the next release.
If you have any code examples, it would be great if you could share.

[Naman-ntc]

Hi

def padTarget(input,padding):
        """
    input is a np array of size Nx2 (N is the number of keypoints you want to transform)
    padding is a int or tuple of size 2 or 4 
    (for this code I assumed it to be tuple of size 4 as per link below code    """

     return input - np.asarray([padding[0],padding[1]])

Function Padding Transforms Link
)
This is a rough code made for padding (pretty trivial I guess) need to work through all cases though. Similarly for crop as well as resize and rotate can be done
(PS : Did the simplest one :P others are also trivial but easy to get wrong so did this )

[fmassa]

Thanks!
I'll keep this issue open while we work on adding proper keypoint support in torchvision

[Naman-ntc]

Great! Would have loved to contribute but have some major commitments right now!
Thanks !

[Ridhwanluthra]

Hi, I would like to work on this if someone is not working.
from what I understand basically, we want to create a way to apply a particular transform to multiple points?

[fmassa]

I think it might be better to hold on a bit on this for the moment.
I believe the key point here will be to have a nice representation for the keypoints, which enables us to perform the transformations easily and efficiently.
I have such representations for bounding boxes and segmentation masks, and next I'll be looking at keypoints.

[Ridhwanluthra]

@fmassa I can help with building that representation as well. Please let me know if that is something I can contribute to

[fmassa]

@Ridhwanluthra thanks for the offer!
I've just started working on keypoints, I'll keep you posted!