Vision: TorchVision Roadmap

[datumbox]

This issue is be used to keep track of the roadmap of TorchVision and get feedback from our users.

2021H1

• Streamlined data pipelines:
  
  
  
  • [x] Improved io.image with new supported types and conversions (#2984, #2988, #3024, #3069)
  
  
  • [x] Add Auto-augment input transformation (#3123)
  
  
  • [ ] Add preset preprocessing transforms on pretrained models
  
  
  • [ ] Add GPU/CUDA ops for image decoding (jpeg)
  
  
  • [ ] Add and optimize native uint8 support for interpolate in PyTorch
  
  
• Mobile support:
  
  
  
  • [x] Refactor C++ codebase & make it mobile-friendly (#2885, #2891, #2892, #2893, #2944, #2945, #3011, #3097, #3135, #3135, #3146, #3156, #3154, #3163, #3218)
  
  
  • [x] Add mobile-ready models for detection (#3253, #3265)
  
  
  • [ ] Add mobile-ready models for segmentation (#3276)
  
  
  • [ ] Support D2Go OSS
  
  
  • [ ] Add quint8 implementations for nms and RoIAlign
  
  
  • [ ] Rotated Boxes
  
  
• Add newer models for classification:
  
  
  
  • [x] MobileNetV3 (#3252)
  
  

[masahi]

Can we revisit various workarounds for tracing Faster RCNN / MaskRCNN ? There are many such special code-path for tracing mode, to support ONNX export. These code were added one year ago when PyTorch ONNX had many limitation. Especially the one below does repeated concat in a loop so it is not efficient.

https://github.com/pytorch/vision/blob/3d60f498e71ba63b428edb184c9ac38fa3737fa6/torchvision/models/detection/roi_heads.py#L454-L461

Now that PyTorch ONNX export has support for lists and also inplace assignment, I think many of those workarounds can be removed. Even if removing is not possible, we should be able to make them more efficient, thanks to the improved ONNX export support in recent PyTorch. For example, The snippet above can be replaced with the one below, which uses a batched concat. I confirmed that ONNX export of maskrcnn works with this change applied. This is also what TVM prefers.

@torch.jit._script_if_tracing
def _onnx_paste_masks_in_image_loop(masks, boxes, im_h, im_w):
    res = []
    for i in range(masks.size(0)):
        mask_res = _onnx_paste_mask_in_image(masks[i][0], boxes[i], im_h, im_w)
        res += [mask_res]
    return torch.stack(res, dim=0)[:, None]

cc @fmassa @lara-hdr

[fmassa]

@masahi I would be happy to accept a PR to improve the ONNX export path, and removing as many workarounds as possible would be awesome. I've opened an issue in https://github.com/pytorch/vision/issues/3225 to centralize this discussion there.

[oke-aditya]

Just linking a few trackers here #2707 , #2980