Vision: To extend torchvision for video

[JuanFMontesinos]

Motivation

I've realized that the way torchvision is coded it's not possible to store a transformation to be applied several times. Video requires the same transformation to be applied to the whole sequence.

Proposed changes

I propose to restructure the code with minor changes such that:

A base transformation class (template) were created, providing get_params and reset_params method:

class BaseTransformation(object):  
    def get_params(self):  
        pass  
    def reset_params(self):  
        pass  

get_params would provide needed parameters if necessary meanwhile reset_params would act as param initilizer + reseter.

To modify compose class to deal with list/tuples of frames such that when the list were exhausted, paramters would be reset:

class Compose(object):
    """Composes several transforms together.

    Args:
        transforms (list of ``Transform`` objects): list of transforms to compose.

    Example:
        >>> transforms.Compose([
        >>>     transforms.CenterCrop(10),
        >>>     transforms.ToTensor(),
        >>> ])
    """

    def __init__(self, transforms):
        self.transforms = transforms

    def __call__(self, inpt):
        if isinstance(inpt,(list,tuple)):
            return self.apply_sequence(inpt)
        else:
            return self.apply_img(inpt)
    def apply_img(self,img):
        for t in self.transforms:
            img = t(img)
        return img
    def apply_sequence(self,seq):
        output = list(map(self.apply_img,seq))
        for t in self.transforms:
            t.reset_params()
        return output
    def __repr__(self):
        format_string = self.__class__.__name__ + '('
        for t in self.transforms:
            format_string += '\n'
            format_string += '    {0}'.format(t)
        format_string += '\n)'
        return format_string
 ```
To set random parameters and image parameters as object attributes. As some parameters requires image features to be computed, parameters would be initialized as None and computed/stored with the 1st frame:  
Example 1:

class RandomHorizontalFlip(object):
"""Horizontally flip the given PIL Image randomly with a given probability.

Args:
    p (float): probability of the image being flipped. Default value is 0.5
"""

def __init__(self, p=0.5):
    self.p = p

def __call__(self, img):
    """
    Args:
        img (PIL Image): Image to be flipped.

    Returns:
        PIL Image: Randomly flipped image.
    """
    if self.flag is None: #This was initially if random.random() < self.p: so it was not possible
                                   #to apply the same transformation to another frame
        self.get_paramters()
    if self.flag:
        return F.hflip(img)
    return img

def __repr__(self):
    return self.__class__.__name__ + '(p={})'.format(self.p)
def get_paramters(self):
    self.flag = random.random() < self.p
def reset_params(self):
    self.flag = None

Example 2:

class RandomResizedCrop(BaseTransformation):
"""Crop the given PIL Image to random size and aspect ratio.

A crop of random size (default: of 0.08 to 1.0) of the original size and a random
aspect ratio (default: of 3/4 to 4/3) of the original aspect ratio is made. This crop
is finally resized to given size.
This is popularly used to train the Inception networks.

Args:
    size: expected output size of each edge
    scale: range of size of the origin size cropped
    ratio: range of aspect ratio of the origin aspect ratio cropped
    interpolation: Default: PIL.Image.BILINEAR
"""

def __init__(self, size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation=Image.BILINEAR):
    if isinstance(size, tuple):
        self.size = size
    else:
        self.size = (size, size)
    if (scale[0] > scale[1]) or (ratio[0] > ratio[1]):
        warnings.warn("range should be of kind (min, max)")

    self.interpolation = interpolation
    self.scale = scale
    self.ratio = ratio
    self.reset_params()

def get_params(self,img, scale, ratio):
    """Get parameters for ``crop`` for a random sized crop.

    Args:
        img (PIL Image): Image to be cropped.
        scale (tuple): range of size of the origin size cropped
        ratio (tuple): range of aspect ratio of the origin aspect ratio cropped

    Returns:
        tuple: params (i, j, h, w) to be passed to ``crop`` for a random
            sized crop.
    """
    area = img.size[0] * img.size[1]

    for attempt in range(10):
        target_area = random.uniform(*scale) * area
        log_ratio = (math.log(ratio[0]), math.log(ratio[1]))
        aspect_ratio = math.exp(random.uniform(*log_ratio))

        w = int(round(math.sqrt(target_area * aspect_ratio)))
        h = int(round(math.sqrt(target_area / aspect_ratio)))

        if w <= img.size[0] and h <= img.size[1]:
            i = random.randint(0, img.size[1] - h)
            j = random.randint(0, img.size[0] - w)
            return i, j, h, w

    # Fallback to central crop
    in_ratio = img.size[0] / img.size[1]
    if (in_ratio < min(ratio)):
        w = img.size[0]
        h = w / min(ratio)
    elif (in_ratio > max(ratio)):
        h = img.size[1]
        w = h * max(ratio)
    else:  # whole image
        w = img.size[0]
        h = img.size[1]
    self.i = (img.size[1] - h) // 2
    self.j = (img.size[0] - w) // 2
    self.h = h
    self.w = w

def reset_params(self):
    self.i = None
    self.j = None
    self.h = None
    self.w = None    

def __call__(self, img):
    """
    Args:
        img (PIL Image): Image to be cropped and resized.

    Returns:
        PIL Image: Randomly cropped and resized image.
    """
    if self.i is None:
        assert self.i == self.h == self.j == self.w 
        self.get_params(img, self.size)

    return F.resized_crop(img, self.i, self.j, self.h,
                          self.w, self.size, self.interpolation)

def __repr__(self):
    interpolate_str = _pil_interpolation_to_str[self.interpolation]
    format_string = self.__class__.__name__ + '(size={0}'.format(self.size)
    format_string += ', scale={0}'.format(tuple(round(s, 4) for s in self.scale))
    format_string += ', ratio={0}'.format(tuple(round(r, 4) for r in self.ratio))
    format_string += ', interpolation={0})'.format(interpolate_str)
    return format_string

```

[fmassa]

Thanks for opening the issue!

Adding support for video data is in the plans, and will be integrated in the next major release of torchvision. This also involves the transforms.

[JuanFMontesinos]

Hi!
Could you tell me which branch contains those changes? pretty sure it will be better than what I'm suggesting.

[fmassa]

It's currently in a private branch, I'm working on some other things now and I'll get back to video once I've finished those next tasks, hopefully by the end of the week

[kateiyas]

@fmassa Are there news on this already? Using these transforms in video clips would be very useful for us right now.

[JuanFMontesinos]

@kateiyas You can download a pip package called flerken (under development), which contains a framework for pytorch but also torchvision adapted for video
You can import these submodules and they act as a drop-in wrt torchvision transforms
if you input a list of frames instead of a single frame it will apply the same transformation to all of them and then reset with new random parameters
from flerken.dataloaders import transforms as T
from flerken.dataloaders.transforms import Compose

You have all the torchvision transforms there (only main compose class has been rewritten)

[kateiyas]

@JuanFMontesinos Thanks, but a few small adaptations to this package fulfilled my needs: https://github.com/hassony2/torch_videovision
Good luck with your project though!

I hope all of this will be integrated into pytorch soon.

[fmassa]

We got a bit late with the work on video. It won't be present in the 0.3 release, but in the next one.

[ekagra-ranjan]

@fmassa What is the rough timeline that you have in mind for releasing the major changes in the torchvision.transform?

[fmassa]

@ekagra-ranjan

Next release with video is planned for end of July.

First PR adding video reading is already merged in https://github.com/pytorch/vision/pull/1039

[willprice]

@JuanFMontesinos you might also be interested in https://torchvideo.readthedocs.io/en/latest/

[JuanFMontesinos]

@willprice thanks! It looks really nice

[fmassa]

TorchVision 0.4 with support for video has been released, see https://github.com/pytorch/vision/releases/tag/v0.4.0

We still need to adapt the default transforms to support video data, but that might be a breaking change so we currently have them in the references/video_classification folder

[fmassa]

Initial set of transforms for video have been added in https://github.com/pytorch/vision/pull/1353
There will still be some refactoring happening, so it shouldn't be considered to be used in its current state, but it will soon be ready for consumption.

[bjuncek]

@fmassa I believe these have been closed by @vfdev-5 s tensor transform PRs.

Do you think it's safe to close this?

[Greeser]

What is a current state with video transformation?

What I've understood so far is that there was a plan to add transformation. Currently _transforms_video.py and _functional_video.py remain private. From this I can assume that these files won't live long and will be discarded soon.

[vfdev-5]

@Greeser with the upcoming release, almost all torchvision transformations will work on tensors of shape (B, C, H, W) which should cover the scope of _transforms_video.py. See https://github.com/pytorch/vision/tree/master/examples/python and please keep in mind that these examples are subject to a minor changes before the official release.