Dali: Labels for video files with ops.VideoReader

[alicranck]

Is there any way to return labels with the videos loaded by ops.VideoReader? Maybe composing it with another op in some way? It would be really helpful for video classification tasks (action recognition etc.).

I would be happy to know if something like this exists or is planned.
Thanks!

[JanuszL]

Hi,
It is an excellent question. Currently, it is not possible. Could you write more about how your data set looks like, how labels and videos are stored, are there any other applications you consider beyond classification?
Our initial goal for the Video loader was to support End-to-End Learning of Video Super-Resolution with Motion Compensation and see how people want to use it for the different cases like yours.

[alicranck]

Hi @JanuszL , Thanks for the reply!

I'm currently using the Kinetics-400 dataset that has the same structure that the ops.FilesReader uses (i.e. validation/train folders each containing 400 folders, 1 per class). But I think that since we pass a list of files to the reader anyway, the most flexible and general way to support labels would be to pass a matching list of labels.

Right now I don't have other applications in mind but I imagine that more people would want to move from image to video processing as the increase in compute capabilities allows this, and that this sort of functionality might be useful in this case.

[Kh4L]

Hi @alicranck ,

Indeed, currently VideoReader only supports a list of filenames as input.

Since video datasets as Kinetics-400 have a similar structure as FileReader, I think that we should extend VideoReader to return labels when a file_root would be provided as argument.

Thanks for the proposition! :+1:

Tracked DALI-614.

[willprice]

This would be super helpful. It'd be great to have some more flexible way of providing labels that just through filesystem hierarchy. For example, Something-something labels are provided as JSON and Jester they're a CSV, EPIC provides them in a CSV or pickle.
Ideally having some way of providing a labelling function given the filename/path of a video would support all these use cases I've described and the kinetics use case.

[willprice]

I'm happy to chip in and implement this, although I'm not very familiar with CPP. I'm guessing one would want to do something similar to
https://github.com/NVIDIA/DALI/blob/master/dali/pipeline/operators/reader/file_reader_op.h#L48
in
https://github.com/NVIDIA/DALI/blob/master/dali/pipeline/operators/reader/loader/video_loader.cc#L443

which would necessitate some wrapper like ImageLabelWrapper
but wrapping a Sequence and a file path. Would the implementation add to SequenceWrapper or wrap the sequence wrapper itself?

[Kh4L]

Hi @willprice ,

Sure, we would be happy to integrate your contribution! :-)

I think adding the labels to frame_starts_ would be the way to go - maybe by simply using std::vector<std::tuple<int, int, int>>, or even better, changing it to a struct SeqMeta for readability.

And then, as you suggested, the ReadSample would be

void VideoLoader::ReadSample(SequenceWrapper& tensor) {
    // TODO(spanev) remove the async between the 2 following methods?
    auto& seq_meta = frame_starts_[current_frame_idx_];
    push_sequence_to_read(filenames_[seq_meta.file_idx], seq_meta.frame_idx, count_);
    receive_frames(tensor);
    tensor.wait();
    ++current_frame_idx_;

    tensor.label = seq_meta.label;

    MoveToNextShard(current_frame_idx_);
}

and in https://github.com/NVIDIA/DALI/blob/master/dali/pipeline/operators/reader/video_reader_op.h#L68 , you would just have to set the Output #1 to tensor.label.

[shecker]

Hi everyone,
Thanks for this great library. Correct me if I'm wrong, but isn't the VideoReader of DALI based off of the NVVL project? https://github.com/NVIDIA/nvvl
In which case, wouldn't it be possible to handle obtaining labels just as in NVVL where you specify an optional callable for the VideoDataset class?
Also would it be possible to perform label specific image augmentation? For example, I'd like to rotate my image by a label specified angle?
Thanks a lot!
Best,
Simon

[JanuszL]

@shecker - that is true, the core of the Video decoder is based on nvvl but the logic around it is DALI specific.
In the case of callable, it may be difficult as it doesn't fit into current DALI architecture as labels are yet another data in the pipeline. In the case of nvvl labels cannot be processed and they are loaded and outputted at the very end of VideoDataset class. To make it really DALI way VideoLoader need to be able to call this Python callback (it is somehow possible https://github.com/NVIDIA/DALI/pull/732 but could be terribly slow).
Regarding label specific augmentations it seems to be doable by some custom operator translating image to parameters that drives other operators. Again https://github.com/NVIDIA/DALI/pull/73 could be some solution to that.
This is my brief ideas but we still need more discussion before we can propose anything definite.

[keunhong]

The current architecture seems a bit too rigid. I am currently trying to read video frames together with the associated audio waveforms, but DALI doesn't seem to have a way to return the video frame numbers without writing a custom C++ op.

It would be helpful to have some sort of arbitrary lambda operation support so that unsupported data can be loaded. Even with the performance degradation I think the accelerated video loading and augmentation would make it worth it. Right now my dataloader's bottleneck is JPEG decoding, and DALI would alleviate that but there doesn't seem to be any way for me to use it.

[Kh4L]

Hi @alicranck ,

I am looking into https://deepmind.com/research/open-source/open-source-datasets/kinetics/ and from what I see, the dataset structure is arbitrary since they only provide you CSV and JSON containing the scenes metadata and YT location.

Do you often see the 1 folder/class structure in the literature or is this just how you chose to organize the dataset in your usecase?

[willprice]

I'll chime in as well @Kh4L,
One folder/class is quite common, but so is a flat structure of all examples in a single folder.

[alicranck]

Hi @Kh4L ,
This is true, this structure comes from the script I used to download the videos. This type of structure is pretty common for classification tasks, and some frameworks have built in support for it (https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder), however it's definitely not the only use case.

I can think of many applications where the label would be an image/video as well (segmentation), a list of bounding boxes (detection), text (annotation) etc. These are not marginal cases in research today so I think providing maximum flexibility is important for people to be able to integrate DALI in their projects.

I don't really know what are the constraints you have to work with, but I imagine that having as optional input a list of labels (that could be any object) to be returned with the appropriate video, would answer most needs in that regard.

[Kh4L]

@willprice @alicranck right, thank you for you input!

Segmentation data support is on the roadmap, and supporting both image and video segmentation labels will be considered as soon as we get on it.

I don't really know what are the constraints you have to work with, but I imagine that having as optional input a list of labels (that could be any object) to be returned with the appropriate video, would answer most needs in that regard.

Sure but how would you parse the content of this "list of labels" ? Its quite hard to have a format and a generic parser supporting all the tasks.

[alicranck]

@Kh4L For me I know that even being able to provide a list of integers, that will be used as indices to a list of labels that I keep separately, would be useful.

This may result in sub-optimal performance but since you say that support for image and video labels is planned, this may cover most other use cases in a good-enough manner, and nullify the need to support many different formats.

[jbohnslav]

For action recognition, it would be great if we could get per-frame labels, not only video-level labels.

[JanuszL]

@jbohnslav - how do you think the user should pass those labels to the Pipeline itself? Some annotation file?

[jbohnslav]

@JanuszL

Yes, there can either be one annotation file per dataset or (potentially more simply) one annotation file per video. Here are two example formats: JSON with the starts and ends of each "action" in seconds, like the ActivityNet dataset:

"---9CpRcKoU": {
            "annotations": [
                {
                    "label": "Drinking beer", 
                    "segment": [
                        0.01000, 
                        12.64441
                    ]
                }
            ], 
            "duration": 14.07000, 
            "resolution": "320x240", 
            "subset": "training", 
            "url": "https://www.youtube.com/watch?v=---9CpRcKoU"
        }, 
        "--0edUL8zmA": {
            "annotations": [
                {
                    "label": "Dodgeball", 
                    "segment": [
                        5.46484, 
                        86.71838
                    ]
                }
            ], 
            "duration": 92.18000, 
            "resolution": "640x480", 
            "subset": "training", 
            "url": "https://www.youtube.com/watch?v=--0edUL8zmA"
        }
...

For the AVA dataset, it's a .csv file with columns: video, frame, person box (4 coordinates for bounding box), action_id (integer denoting class), and person_id denoting which person in the frame was doing the action.

A disk-inefficient, but easy-to-parse format I find useful is for each video, have a corresponding .csv file with the same number of rows as there are frames in the video. It has either 1 column with an integer for single-class classification, or N columns with 0s and 1s in a multilabel case.

[JanuszL]

@jbohnslav - understood. Tracked as DALI-890.

[ArunaUMedhekar]

@alicranck , @willprice ,
The initial requirement for VideoReader operator to generate and return labels based on file directory structure or using a file_list argument is implemented via https://github.com/NVIDIA/DALI/pull/1029 and https://github.com/NVIDIA/DALI/pull/998. It will be available in today's nightly build.

[cinjon]

Hi, I got really excited to see @jbohnslav and @JanuszL's last two comments re ActivityNet style of annotation loading in the VideoPlayer. However, I don't see how to do this in the examples or in the code re the two PRs that @ArunaUMedhekar mentioned (#1029 and #998). Is there an updated tutorial for this? I'm looking to be able to load ActivityNet style annotations along with the paired video (and ideally know which frames are used so I can further decipher which annotation segments are relevant).

Thanks for your help.

[JanuszL]

Hi,
Currently you can find the examples with multiple videos and labels.

[cinjon]

Hi, thanks for the quick reply. I've seen those two links and, as far as I can tell, they don't display a method for including annotations like is done in ActivityNet. I'm probably getting something wrong but the labels that it provides seems to be related to the directory structure rather than an external file with segment annotations. Is that right?

[JanuszL]

Hi,
In this case, you need to write some custom code that will correlate returned classes with the annotations you have and then pass it further to the model. Currently, there is no easy out of the box way.
@a-sansanwal any hint?

[cinjon]

Help on this would be really fantastic. @jbohnslav did you figure this out?

[suriyasingh]

Hi,

I am trying to read N frames (stride of ~10) at a time from a video and also it's label (one label per video).
VideoReader with file_list option throws
terminate called after throwing an instance of 'dali::DALIException' what(): [/opt/dali/dali/pipeline/operators/reader/loader/video_loader.cc:358] 0: failed to seek frame 0

Any help is appreciated. Thanks!

[JanuszL]

I am trying to read N frames at a time from a video and also it's label (one label per video).
VideoReader with file_list option throws
terminate called after throwing an instance of 'dali::DALIException' what(): [/opt/dali/dali/pipeline/operators/reader/loader/video_loader.cc:358] 0: failed to seek frame 0
Any help is appreciated. Thanks!

@a-sansanwal ?

[a-sansanwal]

@suriyasingh can you upload the video youre trying to read from ?

[suriyasingh]

@a-sansanwal they were videos from UCF101 dataset re-encoded with h264_nvenc. This seems to have been fixed in #1287.

[cinjon]

@a-sansanwal Just in case it was lost in the shuffle, did you see my problem as well? Thanks.

[a-sansanwal]

@suriyasingh Recently we also added support for reading directly from UCF-101 without re-encoding via the pull request https://github.com/NVIDIA/DALI/pull/1241.
It should be available in the nightly and/or weekly build.

[suriyasingh]

@a-sansanwal great! thanks!
is there a way to have a dataloader that iterate over one video at a time in a list of many videos? This is specially useful in case of inference/evaluation. Current loader seems to lack this feature.

[a-sansanwal]

@suriyasingh if you use file_list with labels and with random_shuffle=False(default), the sequence's will be in order and when you detect a label change, you can infer that the next video has begun.

[JanuszL]

Hi,
You can provide file_list to the VideoReader with the following format:

filename     label     start_frame    end_frame
file.mp4      0           5               10 
file.mp4      1           11              12

Based on those unique labels you can map the samples that video reader returns with any piece of information you have in your annotation file/files.

[JanuszL]

@suriyasingh - my bad. I have misread the code and this is not possible now. Sorry for the confusion. We have some PoC ready but there are many open questions about the flexibility, like:

• if the user provides such format it means that video can be returned within a given set of pairs start/end or samples can cross the boundary of one label, if so what then?

@a-sansanwal - maybe the VideoReader could return, with labels, two tensors with start and end of a given sequence?

[a-sansanwal]

@cinjon we have PoC that helps specify valid start+end timestamps and a label associated with it. But as @JanuszL mentioned there are questions about it.

@JanuszL We could return the timestamp of first and last frame in each sequence. But I imagine its more friendly to have DALI read valid timestamps and only return sequences from between the valid timestamps.

[JanuszL]

@a-sansanwal - we can do that.
@suriyasingh - do you think it meets your use case?

[a-sansanwal]

@JanuszL its related to @cinjon's request not @suriyasingh
And Yes, I can send a PR when I find some time.

[cinjon]

@a-sansanwal What do you mean by _valid_? Even just returning the first and last timestamp of frame in the sequence, along with the label of what video it was from, would be sufficient. Then I could cross reference that with a side annotation dict to get what the labels should be. This would be super helpful.

[a-sansanwal]

@cinjon By valid I mean that sequences will only be generated from between the specified start+end timestamp. Frames in a video will not be returned if they do not fall between any of the start+end timestamps provided as input to VideoReader, these frames become invalid. In this case there would be no need to return the timestamps since a unique label will be assigned to each clip specified. You can then associate the label with an annotation.
As an example, the input is of the form.

filename   label  start   end
file.mp4      1    5.0    10.0
file.mp4      2    15.0   20.0

Returning the first and last timestamp is trivial too, we already know the frame number of first frame of each sequence, number of frames in the sequence and we also know the frame rate of the videos. We just need to multiply frame number and 1/fps and return that from VideoReader.

[cinjon]

I see. That's not necessary for me as my model needs to get inputs from all parts of the video.

The ideal for me is that I can select a number of frames N, an fps F, and some way of specifying the allowed starting frames. By the latter, I don't really need to provide a list; it can be as simple as every Kth frame starting from zero (that's how I am doing it when using images).

This would then yield a batch of <filename, starting_timestamp, frames> where the assumption is that len(frames) = N, the first frame was from time starting_timestamp and the last frame was from starting_timestamp + N/F. From this batch, I can figure out the annotations because I have the filename key as well as the window from where in that video these frames came.

Is something like this on the roadmap / is there a PR somewhere that is near completion? That would be super.

[a-sansanwal]

It can be as simple as every Kth frame starting from zero (that's how I am doing it when using images).

VideoReader supports step and stride parameters
check https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/supported_ops.html#videoreader

[cinjon]

Splendid, those are super useful. So then I should be set as long as I can get the filename back. Is the way to do that through the use of the label like in this demo (https://github.com/NVIDIA/DALI/blob/b3e406bd7b454c8afaf5aa2d0156e1f8774df48c/docs/examples/video/video_label_example.py)?

[cinjon]

Ok, I have this almost working now. The remaining difficulty is yielding what frame number is the data. All of the videos are the same FPS and I can figure out the per-video information by using the label obtained from passing in a file_list of . However, I can't figure out the correct label for the returned frames unless I know at least the start (and preferably the end as verification) frame. @a-sansanwal, is this on the agenda? Or is there already a way to do it?

For reference, my workflow is that I have a pytorch data loader DL that defines a series of VideoReader pipes (one per gpu) and uses those to process the videos. I can execute pipes.run() in DL's __getitem__, but I expect that to match up with the index DL receives and there's no guarantee that it does. If I could pass in the index to the pipes, that would work because then I could get back the right frames for each __getitem__ call.

Another approach would be to sidestep the PyTorch dataloader completely and just use the Dali video reader. In that case, the index wouldn't matter, but I still need to get the start and end frame number to align the returned frames with the right data.

Thanks!

(appears that #753 is also about this)

[a-sansanwal]

@cinjon I have change that allows you to get starting frame numbers. You can add the sequence length to get the end frame number. Link
I dont plan to send it as a pull request in its current form.
I also havent tested this recently, so if it doesnt work as is, it might need minimal changes.

[cinjon]

awesome! ill get on building locally with that adjustment. thanks @a-sansanwal, will report back.

[cinjon]

I tried building by pulling and then filling in your commit, but ran into an error. Have you seen this before?

[ 43%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/resampling_test/resampling_compare_test.cc.o
nvcc error   : 'cicc' died due to signal 9 (Kill signal)
CMake Error at dali_operators_generated_expression_impl_factory_gpu.cu.o.Release.cmake:279 (message):
  Error generating file
  /opt/dali/build-docker-Release-36-10_x86_64/dali/operators/CMakeFiles/dali_operators.dir/expressions/./dali_operators_generated_expression_impl_factory_gpu.cu.o


make[2]: *** [dali/operators/CMakeFiles/dali_operators.dir/expressions/dali_operators_generated_expression_impl_factory_gpu.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 43%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/resampling_test/resampling_impl_cpu_test.cc.o
[ 43%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/resampling_test/separable_cpu_test.cc.o
[ 44%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/resampling_test/separable_impl_test.cc.o
[ 44%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/warp_test/warp_cpu_test.cc.o
[ 44%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/warp_test/warp_transform_test.cc.o
[ 44%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/alloc_test.cc.o
[ 45%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/any_test.cc.o
[ 45%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/block_setup_test.cc.o
[ 45%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/kernel_poc_test.cc.o
[ 45%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/kernel_test.cc.o
[ 46%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/manager_test.cc.o
[ 46%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/scatter_gather_test.cc.o
[ 46%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/scratch_copy_test.cc.o
[ 46%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/scratch_test.cc.o
[ 47%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/static_switch_test.cc.o
[ 47%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/test_data_test.cc.o
[ 47%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/test_utils_test.cc.o
[ 47%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/tuple_test.cc.o
[ 48%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/test/util_test.cc.o
[ 48%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/dali_kernel_test.cc.o
[ 48%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/__/test/dali_test_config.cc.o
[ 48%] Linking CXX executable ../python/nvidia/dali/test/dali_kernel_test.bin
[ 48%] Built target dali_kernel_test.bin
make[1]: *** [dali/operators/CMakeFiles/dali_operators.dir/all] Error 2
make: *** [all] Error 2

[cinjon]

This was via the docker approach described here --> https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/compilation.html. I ran it with DALI_BUILD_FLAVOR=nightly-frames PYVER=3.6 CUDA_VERSION=10 ./build.sh

[JanuszL]

This is strange. Can you try to build clean source code from master?

[cinjon]

I stashed the changes that @a-sansanwal suggested and reissued the build command, but ran into another error.

[ 45%] Building NVCC (Device) object dali/operators/CMakeFiles/dali_operators.dir/optical_flow/turing_of/dali_operators_generated_optical_flow_turing.cu.o
nvcc error   : 'cicc' died due to signal 9 (Kill signal)
CMake Error at dali_operators_generated_expression_impl_factory_gpu.cu.o.Release.cmake:279 (message):
  Error generating file
  /opt/dali/build-docker-Release-36-10_x86_64/dali/operators/CMakeFiles/dali_operators.dir/expressions/./dali_operators_generated_expression_impl_factory_gpu.cu.o


make[2]: *** [dali/operators/CMakeFiles/dali_operators.dir/expressions/dali_operators_generated_expression_impl_factory_gpu.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [dali/operators/CMakeFiles/dali_operators.dir/all] Error 2
make: *** [all] Error 2

[cinjon]

The command i used was DALI_BUILD_FLAVOR=nightly-check PYVER=3.6 CUDA_VERSION=10 ./build.sh.

[JanuszL]

That is very strange. Could you remove a directory with the build output and try again?
Also you can try to download latest nightly for CUDA 10 and checkout on the same SHA and check if that work for you. Also we just recently update our docker build scrips so you may want to run `REBUILD_BUILDERS=YES DALI_BUILD_FLAVOR=nightly-check PYVER=3.6 CUDA_VERSION=10 ./build.sh

[cinjon]

First report: a fresh directory run w that command failed with error:

[ 48%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.bin.dir/__/test/dali_test_config.cc.o
[ 48%] Building NVCC (Device) object dali/operators/CMakeFiles/dali_operators.dir/expressions/dali_operators_generated_expression_impl_factory_gpu.cu.o
[ 48%] Building NVCC (Device) object dali/operators/CMakeFiles/dali_operators.dir/fused/dali_operators_generated_crop_mirror_normalize.cu.o
[ 48%] Linking CXX executable ../python/nvidia/dali/test/dali_kernel_test.bin
[ 48%] Built target dali_kernel_test.bin
[ 48%] Building NVCC (Device) object dali/operators/CMakeFiles/dali_operators.dir/geometric/dali_operators_generated_bb_flip.cu.o
nvcc error   : 'cicc' died due to signal 9 (Kill signal)
CMake Error at dali_operators_generated_expression_impl_factory_gpu.cu.o.Release.cmake:279 (message):
  Error generating file
  /opt/dali/build-docker-Release-36-10_x86_64/dali/operators/CMakeFiles/dali_operators.dir/expressions/./dali_operators_generated_expression_impl_factory_gpu.cu.o


make[2]: *** [dali/operators/CMakeFiles/dali_operators.dir/expressions/dali_operators_generated_expression_impl_factory_gpu.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [dali/operators/CMakeFiles/dali_operators.dir/all] Error 2
make: *** [all] Error 2

I'll try the nightly SHA after dinner.

[cinjon]

Excuse my ignorance, but how do I find the SHA of the nightly? Having some trouble with that.

[JanuszL]

If you install the whl, you can issue pip show nvidia-dali-nightly and in the description you would have something like Summary: NVIDIA DALI for CUDA 10.0. Git SHA: 78f78b0a26b385f0bc4450104fb5e9e30c787a85

[cinjon]

Ok, so I did that and found that the SHA was fa5919985f3bab0aca13a4a9596e9368851a505c. I then detached at that commit and ran REBUILD_BUILDERS=YES DALI_BUILD_FLAVOR=nightly-check PYVER=3.7 CUDA_VERSION=10 ./build.sh. The result was a new error around getting tensorflow-gpu:

Step 16/17 : COPY qa/setup_packages.py qa/setup_packages.py
 ---> f9122e90b7be
Step 17/17 : RUN export USE_CUDA_VERSION=$(cat /usr/local/cuda/version.txt | head -1 | sed 's/.*Version \([0-9]\+\)\.\([0-9]\+\).*/\1\2/') &&     export last_config_index=$(python qa/setup_packages.py -n -u tensorflow-gpu --cuda ${USE_CUDA_VERSION}) &&     for i in `seq 0 $last_config_index`; do         pip download $(python qa/setup_packages.py -i $i -u tensorflow-gpu --cuda ${USE_CUDA_VERSION}) -d /pip-packages;     done
 ---> Running in 00782917caec
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting tensorflow-gpu==1.13.1
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Could not find a version that satisfies the requirement tensorflow-gpu==1.13.1 (from versions: )
  Could not fetch URL https://pypi.org/simple/tensorflow-gpu/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/tensorflow-gpu/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
No matching distribution found for tensorflow-gpu==1.13.1
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting tensorflow-gpu==1.14.0
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/tensorflow-gpu/
  Could not fetch URL https://pypi.org/simple/tensorflow-gpu/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/tensorflow-gpu/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
  Could not find a version that satisfies the requirement tensorflow-gpu==1.14.0 (from versions: )
No matching distribution found for tensorflow-gpu==1.14.0
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
The command '/bin/sh -c export USE_CUDA_VERSION=$(cat /usr/local/cuda/version.txt | head -1 | sed 's/.*Version \([0-9]\+\)\.\([0-9]\+\).*/\1\2/') &&     export last_config_index=$(python qa/setup_packages.py -n -u tensorflow-gpu --cuda ${USE_CUDA_VERSION}) &&     for i in `seq 0 $last_config_index`; do         pip download $(python qa/setup_packages.py -i $i -u tensorflow-gpu --cuda ${USE_CUDA_VERSION}) -d /pip-packages;     done' returned a non-zero code: 1

[cinjon]

I just tried building with PREBUILD_TF_PLUGINS=NO and same error.

[cinjon]

Is it an issue that I'm building this locally on my MBP and not on the cluster it's going to live? I suspect not because it's being built with Docker, but wondering if that is an issue here.

[JanuszL]

In the code you are referring to we are building docker image to build TF plugins. In the recent master it should be skipped if you are not building TF plugins. You can try to comment out/remove https://github.com/NVIDIA/DALI/blob/fa5919985f3bab0aca13a4a9596e9368851a505c/docker/build.sh#L99LL112 and try again.

[cinjon]

Just tried commenting that out and I'm getting the same error as I was before :(.

I am running this command: REBUILD_BUILDERS=YES DALI_BUILD_FLAVOR=nightly-check PYVER=3.7 CUDA_VERSION=10 PREBUILD_TF_PLUGINS=NO ./build.sh.

I'd love to help out and keep testing this, but any way one of you guys can build a wheel for me to use with @a-sansanwal 's frame change in the meantime? This would help a lot so that I can start running jobs on this dataset.

[ 17%] Building NVCC (Device) object dali/kernels/CMakeFiles/dali_kernel_test.dir/test/resampling_test/dali_kernel_test_generated_resampling_internal_test.cu.o
[ 17%] Building NVCC (Device) object dali/CMakeFiles/dali_operators.dir/pipeline/operators/displacement/dali_operators_generated_water.cu.o
[ 17%] Building NVCC (Device) object dali/CMakeFiles/dali_operators.dir/pipeline/operators/expressions/dali_operators_generated_arithmetic.cu.o
[ 17%] Building NVCC (Device) object dali/CMakeFiles/dali_operators.dir/pipeline/operators/expressions/dali_operators_generated_expression_impl_factory_gpu.cu.o
/opt/dali/dali/pipeline/operators/expressions/expression_impl_factory_gpu.cu(83): warning: statement is unreachable

[ 17%] Building NVCC (Device) object dali/kernels/CMakeFiles/dali_kernel_test.dir/test/warp_test/dali_kernel_test_generated_warp_gpu_test.cu.o
nvcc error   : 'cicc' died due to signal 9 (Kill signal)
CMake Error at dali_operators_generated_expression_impl_factory_gpu.cu.o.Release.cmake:279 (message):
  Error generating file
  /opt/dali/build-docker--37-10/dali/CMakeFiles/dali_operators.dir/pipeline/operators/expressions/./dali_operators_generated_expression_impl_factory_gpu.cu.o


make[2]: *** [dali/CMakeFiles/dali_operators.dir/pipeline/operators/expressions/dali_operators_generated_expression_impl_factory_gpu.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 17%] Building NVCC (Device) object dali/kernels/CMakeFiles/dali_kernel_test.dir/test/dali_kernel_test_generated_dev_array_test.cu.o
[ 18%] Building NVCC (Device) object dali/kernels/CMakeFiles/dali_kernel_test.dir/test/dali_kernel_test_generated_kernel_poc_test.cu.o
make[1]: *** [dali/CMakeFiles/dali_operators.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 18%] Building NVCC (Device) object dali/kernels/CMakeFiles/dali_kernel_test.dir/test/dali_kernel_test_generated_span_gpu_test.cu.o
[ 18%] Building NVCC (Device) object dali/kernels/CMakeFiles/dali_kernel_test.dir/test/dali_kernel_test_generated_static_switch_test.cu.o
Scanning dependencies of target dali_kernel_test
[ 18%] Building CXX object dali/kernels/CMakeFiles/dali_kernel_test.dir/imgproc/color_manipulation/hsv_cpu_test.cc.o

[mzient]

What commit are you at? There's no unreachable statement (or no statement whatsoever) in the offending line in current master - but I'm quite sure there _was_ one at some point. Also, you may try removing the semicolon after DALI_FAIL, although I'd find it _very_ surprising if it was the problem.

[cinjon]

I checked out the repo at commit fa5919985f3bab0a as that was the SHA in the nightly according to pip show ....

[JanuszL]

I have updated the code from link that @a-sansanwal provided and managed to build it without any problem. No crash as you have experienced.
Regarding the builds we cannot not provide intermediate build other than nightly/weekly/monthly release.
The last thing I can suggest is to limit the compilation concurrency and change https://github.com/NVIDIA/DALI/blob/master/docker/build_helper.sh#L68 from 'make -jtomake -j2`.

[cinjon]

Sadly, I still can't build this. I have now tried:

1. Building via Docker with another computer. That failed with the same error (see below*).
2. Building from source in a conda env. That keeps failing with the below error that was addressed in two other github issues, however neither solution worked for me as the cmake is picking up the turbo library np.

[ 22%] Building CXX object dali/CMakeFiles/dali.dir/pipeline/util/copy_with_stride.cc.o
[ 22%] Building CXX object dali/CMakeFiles/dali.dir/pipeline/workspace/mixed_workspace.cc.o
/private/home/cinjon/Code/dali/dali/image/jpeg_mem.cc: In function 'dali::uint8* dali::jpeg::{anonymous}::UncompressLow(const void*, dali::jpeg::{anonymous}::FewerArgsForCompiler*)':
/private/home/cinjon/Code/dali/dali/image/jpeg_mem.cc:135:9: error: 'JCS_EXT_BGR' was not declared in this scope
         JCS_EXT_BGR : JCS_RGB;
         ^
/private/home/cinjon/Code/dali/dali/image/jpeg_mem.cc:490:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (argball->height_read_ > target_output_height) {
                               ^
[ 22%] Building CXX object dali/CMakeFiles/dali.dir/pipeline/workspace/sample_workspace.cc.o
dali/CMakeFiles/dali.dir/build.make:182: recipe for target 'dali/CMakeFiles/dali.dir/image/jpeg_mem.cc.o' failed

This is extremely sad for me because the only thing missing from my usage of this library is the frame commit that @a-sansanwal very helpfully provided.

I am also not sure how others are building this without an issue.

*

[ 48%] Building NVCC (Device) object dali/operators/CMakeFiles/dali_operators.dir/expressions/dali_operators_generated_expression_impl_factory_gpu.cu.o
[ 48%] Building NVCC (Device) object dali/operators/CMakeFiles/dali_operators.dir/fused/dali_operators_generated_crop_mirror_normalize.cu.o
nvcc error   : 'cicc' died due to signal 9 (Kill signal)
CMake Error at dali_operators_generated_expression_impl_factory_gpu.cu.o.Release.cmake:279 (message):
  Error generating file
  /opt/dali/build-docker-Release-37-10_x86_64/dali/operators/CMakeFiles/dali_operators.dir/expressions/./dali_operators_generated_expression_impl_factory_gpu.cu.o

[cinjon]

Fwiw, this is my current build command: cmake -DJPEG_INCLUDE_DIRS=~/anaconda3/envs/dalibuild/lib/ -DJPEG_LIBRARIES=~/anaconda3/envs/dalibuild/lib/ -DTIFF_INCLUDE_DIR=~/anaconda3/envs/dalibuild/lib/ -DTIFF_LIBRARY=~/anaconda3/envs/dalibuild/lib/ -DJPEG_LIBRARY_RELEASE=~/anaconda3/envs/dalibuild/lib/ -DFFMPEG_ROOT_DIR=/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib ..

And this is the output:

-- The CXX compiler identification is GNU 5.5.0
-- Check for working CXX compiler: /home/linuxbrew/.linuxbrew/bin/c++
-- Check for working CXX compiler: /home/linuxbrew/.linuxbrew/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- DALI version: 0.16.0dev
-- DALI_extra version: 846e3786d3934e3f149f568a0ed286109dde68dd
-- Build configuration: Release
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
q-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /public/apps/cuda/10.0 (found suitable version "10.0", minimum required is "8.0") 
-- Found NVJPEG: /public/apps/cuda/10.0/include (found suitable version "10.0", minimum required is "9.0") 
/public/apps/cuda/10.0
-- Looking for nvjpegCreateEx
-- Looking for nvjpegCreateEx - not found
-- Looking for nvjpegBufferPinnedCreate
-- Looking for nvjpegBufferPinnedCreate - not found
nvJPEG found in /public/apps/cuda/10.0/include
-- Found OpenCV: /usr (found suitable version "3.2.0", minimum required is "3.0") found components:  core imgproc imgcodecs 
-- Found OpenCV: /usr/include;/usr/include/opencv (found suitable version "3.2.0", minimum required is "3.0")
OpenCV libraries: opencv_core;opencv_imgproc;opencv_imgcodecs
-- The C compiler identification is GNU 5.5.0
-- Check for working C compiler: /home/linuxbrew/.linuxbrew/bin/cc
-- Check for working C compiler: /home/linuxbrew/.linuxbrew/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PythonInterp: /private/home/cinjon/anaconda3/envs/dalibuild/bin/python (found version "3.7.4") 
-- Failed to find LLVM FileCheck
-- Found Git: /usr/bin/git (found version "2.21.0") 
-- git Version: v1.4.0-505be96a
-- Version: 1.4.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Found JPEG: /private/home/cinjon/anaconda3/envs/dalibuild/lib (found suitable version "80", minimum required is "62") 
Using libjpeg-turbo at /private/home/cinjon/anaconda3/envs/dalibuild/lib
-- Found TIFF: ~/anaconda3/envs/dalibuild/lib/  
Using libtiff at ~/anaconda3/envs/dalibuild/lib/
-- Found PythonLibs: /private/home/cinjon/anaconda3/envs/dalibuild/lib/libpython3.7m.so
-- pybind11 v2.2.4
-- Building WITHOUT LMDB support
-- Found PkgConfig: /home/linuxbrew/.linuxbrew/bin/pkg-config (found version "0.29.2") 
-- avformat
-- avcodec
-- avfilter
-- avutil
-- Performing Test HAVE_AVSTREAM_CODECPAR
-- Performing Test HAVE_AVSTREAM_CODECPAR - Success
-- Looking for C++ include sys/types.h
-- Looking for C++ include sys/types.h - found
-- Looking for C++ include stdint.h
-- Looking for C++ include stdint.h - found
-- Looking for C++ include stddef.h
-- Looking for C++ include stddef.h - found
-- Check size of AVBSFContext
-- Check size of AVBSFContext - done
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.a;-lpthread (found suitable version "3.0.0", minimum required is "2.0") 
-- Enabling TensorFlow TFRecord file format support
-- CUDA supported archs: 35;50;52;60;61;70;75
-- CUDA targeted archs: 35;50;52;60;61;70;75
-- Generated gencode flags:  -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppicom_static.a:libnppicc_static.a:libnppig_static.a:libnppc_static.a:libculibos.a:libopencv_core.a:libopencv_imgproc.a:libopencv_highgui.a:libopencv_imgcodecs.a:liblibwebp.a:libittnotify.a:libpng.a:liblibtiff.a:liblibjasper.a:libIlmImf.a:liblibjpeg-turbo.a:libprotobuf.a:libsupc++.a:libstdc++.a:libstdc++_nonshared.a'
-- Adding dependencies to dali: '/public/apps/cuda/10.0/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/public/apps/cuda/10.0/lib64/libnvjpeg_static.a;/public/apps/cuda/10.0/lib64/libnppicom_static.a;/public/apps/cuda/10.0/lib64/libnppicc_static.a;/public/apps/cuda/10.0/lib64/libnppig_static.a;/public/apps/cuda/10.0/lib64/libnppc_static.a;/public/apps/cuda/10.0/lib64/libculibos.a;opencv_core;opencv_imgproc;opencv_imgcodecs;/private/home/cinjon/anaconda3/envs/dalibuild/lib;~/anaconda3/envs/dalibuild/lib/;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavformat.so;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavcodec.so;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavfilter.so;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavutil.so;/usr/lib/x86_64-linux-gnu/libprotobuf.a'
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Configuring done
WARNING: Target "dali_test.bin" requests linking to directory "/private/home/cinjon/anaconda3/envs/dalibuild/lib".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "dali" requests linking to directory "/private/home/cinjon/anaconda3/envs/dalibuild/lib".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "dali_core_test.bin" requests linking to directory "/private/home/cinjon/anaconda3/envs/dalibuild/lib".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "dali_kernel_test.bin" requests linking to directory "/private/home/cinjon/anaconda3/envs/dalibuild/lib".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "dali_operators" requests linking to directory "/private/home/cinjon/anaconda3/envs/dalibuild/lib".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "dali_operator_test.bin" requests linking to directory "/private/home/cinjon/anaconda3/envs/dalibuild/lib".  Targets may link only to libraries.  CMake is dropping the item.
WARNING: Target "python_function_plugin" requests linking to directory "/private/home/cinjon/anaconda3/envs/dalibuild/lib".  Targets may link only to libraries.  CMake is dropping the item.
-- Generating done
-- Build files have been written to: /private/home/cinjon/Code/dali/build

[JanuszL]

Sadly, I still can't build this. I have now tried:

1. Building via Docker with another computer. That failed with the same error (see below*).

The only reason is some compiler error that causes it to crash. It is strange as it doesn't reproduce in our env despite we are using the same dockerized environment. You can try to build with CUDA 10.1 (this is just an idea), editing https://github.com/NVIDIA/DALI/blob/master/docker/Dockerfile.cuda10.deps#L6L17 and adding http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run instead.

2. Building from source in a conda env. That keeps failing with the below error that was addressed in two other github issues, however neither solution worked for me as the cmake is picking up the turbo library np.

[ 22%] Building CXX object dali/CMakeFiles/dali.dir/pipeline/util/copy_with_stride.cc.o
[ 22%] Building CXX object dali/CMakeFiles/dali.dir/pipeline/workspace/mixed_workspace.cc.o
/private/home/cinjon/Code/dali/dali/image/jpeg_mem.cc: In function 'dali::uint8* dali::jpeg::{anonymous}::UncompressLow(const void*, dali::jpeg::{anonymous}::FewerArgsForCompiler*)':
/private/home/cinjon/Code/dali/dali/image/jpeg_mem.cc:135:9: error: 'JCS_EXT_BGR' was not declared in this scope
         JCS_EXT_BGR : JCS_RGB;
         ^
/private/home/cinjon/Code/dali/dali/image/jpeg_mem.cc:490:31: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (argball->height_read_ > target_output_height) {
                               ^
[ 22%] Building CXX object dali/CMakeFiles/dali.dir/pipeline/workspace/sample_workspace.cc.o
dali/CMakeFiles/dali.dir/build.make:182: recipe for target 'dali/CMakeFiles/dali.dir/image/jpeg_mem.cc.o' failed

For the second error please build with make VERBOSE=1, see the compilation command and check if the proper include directories are used (seems like wrong library or wrong header). Maybe this will help https://github.com/NVIDIA/DALI/issues/884.

[cinjon]

Right, so I saw #884 as well as #710 and that's why I tried changing the library paths.

Wrt checking if the proper include directories are there, can you describe that a little bit more please? There's a lot of output.

Here's the last command before it fails:

cd /private/home/cinjon/Code/dali/build/dali && /home/linuxbrew/.linuxbrew/bin/c++  -DDALI_BUILD_PROTO3=1 -DDALI_USE_JPEG_TURBO -DDALI_USE_NVJPEG -DFFMPEG_ENABLED=1 -DJPEG_TURBO_ENABLED=1 -DLIBTIFF_ENABLED=1 -DLMDB_ENABLED=0 -DNVDEC_ENABLED=1 -DNVJPEG_ENABLED=1 -DNVML_ENABLED=1 -DNVOF_ENABLED=1 -DPYTHON_ENABLED=1 -Ddali_EXPORTS -I/private/home/cinjon/Code/dali/build -I/private/home/cinjon/Code/dali/include -I/private/home/cinjon/Code/dali -I/private/home/cinjon/anaconda3/envs/dalibuild/lib -I/private/home/cinjon/Code/dali/third_party/boost/preprocessor/include -I/private/home/cinjon/Code/dali/third_party/rapidjson/include -I/private/home/cinjon/Code/dali/third_party/turing_of -isystem /public/apps/cuda/10.0/include -isystem /usr/include/opencv -isystem /private/home/cinjon/Code/dali/third_party/googletest/googletest/include -isystem /private/home/cinjon/Code/dali/third_party/benchmark/include/benchmark  -Wall -Wno-unused-variable -Wno-unused-function -fno-strict-aliasing -fPIC -fvisibility=hidden -O3 -DNDEBUG -O2 -DDALI_DEBUG=0 -fPIC   -std=c++14 -o CMakeFiles/dali.dir/c_api/c_api.cc.o -c /private/home/cinjon/Code/dali/dali/c_api/c_api.cc

And here's the beginning parts before all of the make processes get to full steam:

/private/home/cinjon/anaconda3/envs/dalibuild/bin/cmake -S/private/home/cinjon/Code/dali -B/private/home/cinjon/Code/dali/build --check-build-system CMakeFiles/Makefile.cmake 0
Re-run cmake file: Makefile older than: dali/core/CMakeFiles/dali_core_test.bin.dir/dali_core_test.bin_generated_geom_mat_test.cu.o.depend
-- DALI version: 0.16.0dev
-- DALI_extra version: 846e3786d3934e3f149f568a0ed286109dde68dd
-- Build configuration: Release
/public/apps/cuda/10.0
nvJPEG found in /public/apps/cuda/10.0/include
-- Found OpenCV: /usr/include;/usr/include/opencv (found suitable version "3.2.0", minimum required is "3.0")
OpenCV libraries: opencv_core;opencv_imgproc;opencv_imgcodecs
-- Failed to find LLVM FileCheck
-- git Version: v1.4.0-505be96a
-- Version: 1.4.0
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK -- success
Using libjpeg-turbo at ~/anaconda3/envs/dalibuild/lib
Using libtiff at ~/anaconda3/envs/dalibuild/lib/
-- pybind11 v2.2.4
-- Building WITHOUT LMDB support
-- avformat
-- avcodec
-- avfilter
-- avutil
-- Enabling TensorFlow TFRecord file format support
-- CUDA supported archs: 35;50;52;60;61;70;75
-- CUDA targeted archs: 35;50;52;60;61;70;75
-- Generated gencode flags:  -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75
-- Exclude libs 'libcudart_static.a:libnvjpeg_static.a:libnppicom_static.a:libnppicc_static.a:libnppig_static.a:libnppc_static.a:libculibos.a:libopencv_core.a:libopencv_imgproc.a:libopencv_highgui.a:libopencv_imgcodecs.a:liblibwebp.a:libittnotify.a:libpng.a:liblibtiff.a:liblibjasper.a:libIlmImf.a:liblibjpeg-turbo.a:libprotobuf.a:libsupc++.a:libstdc++.a:libstdc++_nonshared.a'
-- Adding dependencies to dali: '/public/apps/cuda/10.0/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so;/public/apps/cuda/10.0/lib64/libnvjpeg_static.a;/public/apps/cuda/10.0/lib64/libnppicom_static.a;/public/apps/cuda/10.0/lib64/libnppicc_static.a;/public/apps/cuda/10.0/lib64/libnppig_static.a;/public/apps/cuda/10.0/lib64/libnppc_static.a;/public/apps/cuda/10.0/lib64/libculibos.a;opencv_core;opencv_imgproc;opencv_imgcodecs;~/anaconda3/envs/dalibuild/lib;~/anaconda3/envs/dalibuild/lib/;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavformat.so;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavcodec.so;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavfilter.so;/private/home/cinjon/anaconda3/pkgs/ffmpeg-4.2-h167e202_0/lib/libavutil.so;/usr/lib/x86_64-linux-gnu/libprotobuf.a'
-- Configuring done
-- Generating done
-- Build files have been written to: /private/home/cinjon/Code/dali/build
/private/home/cinjon/anaconda3/envs/dalibuild/bin/cmake -E cmake_progress_start /private/home/cinjon/Code/dali/build/CMakeFiles /private/home/cinjon/Code/dali/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all

[JanuszL]

I wanted to see if libjpeg-turbo is properly included. It is found as I see:

Using libjpeg-turbo at ~/anaconda3/envs/dalibuild/lib

And in the compilation command it seems that it is included

-I/private/home/cinjon/anaconda3/envs/dalibuild/lib

Can you double check jpeglib.h is present in that dir and it correctly defines JCS_EXT_BGR like here.

[cinjon]

It's not there. It is however in ~/anaconda3/envs/dalibuild/include/jpeglib.h but unfortunately does not define JCS_EXT_BGR.

[JanuszL]

So I guess you need to verify the version and correctness of your libjpeg-turbo installation.

[jbohnslav]

Hi all, away from build errors and back to the main thread: has anyone figured out a way to get frame numbers returned for each sequence? Or better yet, per-frame labels from prespecified label files?

[JanuszL]

This should address your request https://github.com/NVIDIA/DALI/pull/1500.

[JanuszL]

Can you check how https://github.com/NVIDIA/DALI/pull/1500 works for you with the latest nightly build?