Dali: Unsupported layout: '' for input 0 'ExternalSource_id_0_output_0'

[ben0it8]

I switched to nightly (0.17) and the my external source pipeline that worked like a wonder in 0.14 now fails with:
Assert on "ImageLayoutInfo::IsImage(input_layout_)" failed: Unsupported layout: '' for input 0 'ExternalSource_id_0_output_0'

class ExtPipeline(Pipeline):

    device = "cpu"

    def __init__(self,
                 iterator: Iterator,
                 batch_size: int,
                 num_threads: int = 1,
                 device_id: int = 0,
                 mean: List[float] = [0.0, 0.0, 0.0],
                 std: List[float] = [1.0, 1.0, 1.0],
                 bytes_per_sample: int = 0,
                 prefetch_queue_depth={
                     "cpu_size": 1,
                     "gpu_size": 1
                 },
                 seed: int = 42):

        super(ExtPipeline,
              self).__init__(batch_size,
                             num_threads,
                             device_id,
                             prefetch_queue_depth=prefetch_queue_depth,
                             seed=seed,
                             bytes_per_sample=bytes_per_sample)
        self.iterator = iterator

        # Reads from external source, i.e. a generator
        self.input = ops.ExternalSource()

        self.norm = ops.CropMirrorNormalize(
            device=self.device,
            output_dtype=types.FLOAT,
            output_layout="CHW",
            mean=mean if mean else 0.0,
            std=std if std else 1.0)

    def define_graph(self, name="Reader"):
        self.batch = self.input(name=name)
        images = self.norm(self.batch)
        return images

    def iter_setup(self):
        batch = self.iterator.next()
        self.feed_input(self.batch, batch)

I only perform normalization with this, basically. Any idea why the below pipeline gives the runtime error?

Best,
Oliver

[jantonguirao]

Hi @ben0it8 ,
We recently started to be more assertive with the data layout requirements (before we just assumed "HWC" when the layout was not provided).
You can just specify the layout when feeding the input to ExternalSource:

self.feed_input(self.batch, batch, layout="HWC")

[ben0it8]

Thanks @jantonguirao! With this I get some type error:

[jantonguirao]

float64 is no longer supported in CropMirrorNormalize. Please set dtype=np.float32 when creating your numpy array

[BlackPepperAPI]

@jantonguirao I have similar problem when I use ops.VideoReader with labeled video dataset.

My NVIDIA Dali version is 0.17.0 from nightly build.

Issue/Problem

I encountered the same problem when reading .mp4 videos (constant frame rate) when I used ops.VideoReader for reading labeled video from a folder structure like:

├── HMDB51
│   ├── raw
│   │   ├── brush_hair  # class 0
│   │    |         | ─ brush_hair_vid_0.mp4  # video file mp4, 224x224 with 30fps
│   │    |         | ─ brush_hair_vid_1.mp4
│   │    |         | ─ brush_hair_vid_2.mp4
│   │    |         | ─ brush_hair_vid_3.mp4
│   │   ├── cartwheel   # class 1
│   │   ├── catch         # class 2
│   │   ├── chew         # and so on...

So I passed the argument file_root = data/HMDB51/raw to ops.VideoReader and I got the following error, coming from CropMirrorNormalise operation.

RuntimeError: Critical error in pipeline: [/opt/dali/dali/operators/fused/crop_mirror_normalize.h:147] Assert on "ImageLayoutInfo::IsImage(input_layout_)" failed: Unsupported layout: '' for input 0 'VideoReader_id_0_output_1'

Additional Observation

Aside of the above issue I tried different input to ops.VideoReader using file_root, filenames, and file_list:

1. Using file_root: same as above mentioned, error Unsupported layout
2. Using file_list: same as above mentioned error. My file_list contains the following format: filepath label, for example:
   data/HMDB51/processed/1/Estrela_mini_kelly_cartwheel_f_cm_np1_ri_med_0.mp4 1
   The filepath is relative to project directory and the DALI program can find the file. However it still throw Unsupported layout above
3. Using filenames: it worked, somehow and strange enough. It iterated through all batches and printing the size in torch.Size. This time I change the output_map = ["data"] for the pytorch iterator argument because it is just filenames, no label.

Code:

Sorry for the long comment, but this is my code if it helps.

import numpy as np

import nvidia
import nvidia.dali.ops as ops
import nvidia.dali.types as types
from nvidia.dali.pipeline import Pipeline
from nvidia.dali.plugin.pytorch import DALIGenericIterator


class VideoReaderPipeline(Pipeline):
    def __init__(self, file_root, batch_size, sequence_length, crop_size, 
        num_threads, device_id, random_shuffle=True, step=-1, seed=42):
        super().__init__(batch_size, num_threads, device_id, seed=seed)

        # Define video reader
        self.reader = ops.VideoReader(device = "gpu",
                file_root = file_root, sequence_length = sequence_length, normalized = False,
                random_shuffle = random_shuffle, image_type = types.RGB, 
                                dtype = types.UINT8, step = step, initial_fill = 16)

        self.cropnorm = ops.CropMirrorNormalize(device = "gpu",
                seed = seed, crop = crop_size, output_dtype = types.FLOAT,
                output_layout = types.NFCHW)

    def define_graph(self):
        """ Definition of graph-event that defines flow of video pipeline
        """
        input_vid  = self.reader(name = "Reader")
        output_vid = self.cropnorm(input_vid)

        return output_vid

# Running Video Reader Pipeline with pytorch DALIGenericIterator
def main():
    pipeline = VideoReaderPipeline(file_root, 
                batch_size = batch_size, sequence_length = sequence_length,
                crop_size = crop_size, num_threads = 2,
                device_id = 0, random_shuffle = random_shuffle,
                step = temp_stride)

    # Build pipeline
    pipeline.build()
        epoch_size = self.pipeline.epoch_size("Reader")

    # Define DALI iterator
    dali_iterator = DALIGenericIterator(pipeline, output_map = ["data", "label"], 
              size = epoch_size, auto_reset = True)

        for sequence in dali_iterator:
            print("Running pipeline")
            print(sequence[0]['data'].shape)

It would be great if I can use file_root or file_list for my Deep Learning research purpose. For now I'll do the hackaround using filenames and providing the label myself.

[BlackPepperAPI]

Should I continue this thread or make a new issue?

[JanuszL]

@BlackPepperAPI - the problem is that you try to crop labels returned by the reader as well, try something like (input_vid[0] are the frames, input_vid[1] are the labels):

        output_vid = self.cropnorm(input_vid[0])

        return output_vid, input_vid[1]