Detr: 'assert (boxes1[:, 2:] >= boxes1[:, :2]).all()' happened when training

[LovPe]

Thanks for amazing work
I have questions when training with your code, assert (boxes1[:, 2:] >= boxes1[:, :2]).all() happened in function generalized_box_iou
After reading the code i find that boxes1 is the predictd bbox from a MLP layer, which i think the above assertion may happen during early training time, and then break the training.
I wonder if there are Mechanism that can make sure to avoid this happen

My Environment:

Provide your environment information using the following command:
Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 4.9.3-13ubuntu2) 4.9.3
CMake version: version 3.16.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: TITAN Xp
GPU 1: TITAN Xp
GPU 2: TITAN Xp
GPU 3: TITAN Xp

Nvidia driver version: 410.48
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.17.1
[pip3] torch==1.4.0
[pip3] torchfile==0.1.0
[pip3] torchvision==0.5.0
[conda] mkl 2019.4 243 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] pytorch 1.4.0 py3.6_cuda10.0.130_cudnn7.6.3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
[conda] torch 1.0.0
[conda] torchfile 0.1.0
[conda] torchvision 0.5.0 py36_cu100 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

[alcinos]

@LovPe Thank you for your interest in DETR.

This assertion is not a fluke, if you're getting it, it means that something is going wrong in your training. Here are some potential things you can look into:

1. We normally enforce that the boxes are non-degenerate thanks to this sigmoid: https://github.com/facebookresearch/detr/blob/10a2c759454930813aeac7af5e779f835dcb75f5/models/detr.py#L68 Did you, by any chance, remove that?
2. This is possibly a red-herring, and the error is somewhere else. You can try running the code with CUDA_LAUNCH_BLOCKING=1 python main.py to see if anything comes up. See #28 for more details
3. I note that your libraries are a bit old, we recommend PyTorch 1.5+ and torchvision 0.6+. If everything else fails, you can try upgrading.

Hope this helps, good luck with the debugging.

[raviv]

@LovPe I was getting this error when the learning rate was too high.

[kuixu]

@LovPe

I was getting this error and found all the boxes are NaN, which is the problem of a full mask generated in the interpolation step (Backbone forward), mainly because of very large zero-padding in loading batch image and mask.

If so, the variables below (outputs_coord, outputs_class and hs) are all NaN.

https://github.com/facebookresearch/detr/blob/10a2c759454930813aeac7af5e779f835dcb75f5/models/detr.py#L65-L68

[zlyin]

I think @raviv is correct. In my dataset, LR = 2e-4 works well but I came across such error when I set LR = 2e-3.

[zlyin]

Sorry for my wrong answer. Setting small LR actually only delays the error popping up. I've already set the LR to e-6 level, but still got the error....

[fmassa]

Hi @zlyin

I believe we went over most of the debugging tips to identify where the root cause of this issue might be in https://github.com/facebookresearch/detr/issues/28.

In particular, I would look to see if there are other error messages that appear in your code before the assert from the beginning, such as

RuntimeError: CUDA error: device-side assert triggered

Which could be a different issue and caused by wrong number of classes.

[zlyin]

Hi @fmassa, thanks for your reply. I solved this issue by changing the bbox format into the normalized coco format.

[liminn]

I am training DETR on COCO panoptic dataset, and I also meet the error assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
I use:
python3 main.py --lr_drop 100 --epochs 150 --coco_path /mnt/hdd1/zlm/data/open_source/coco/ --coco_panoptic_path /mnt/hdd1/zlm/data/open_source/panoptic_annotations_trainval2017/ --dataset_file coco_panoptic --output_dir /mnt/hdd1/zlm/data/train_results/detr_4/
I just use single GPU, and change the lr to lr/8.0, and nothing other changed.

[LovPe]

@fmassa
sorry for late replaying. You are right the code can make sure the assert will not happen(sorry for my misunderstanding ).
The core problem is caused somewhere else that the box value is all NaN. From some reply, this may due to incorrect learning rate. so I trained with 4 gpu and I tuned the batch size follow the linear scaling rule, but it still not work.
there are 2 main differences in training settings:
1- 4GPU vs 8GPU (8GPU is not avalible for me. I wonder if you ever tried training successfully with 4 GPU device.)
2- torch1.4-cuda10-torchvision0.5 vs torch1.5-cuda10.1-torchvision0.6 (for some reason, i can not use update to cuda10.1)

[liminn]

the box value is all NaN

@LovPe
Agree with you, the same phenomenon：the box value is all NaN.
My envs: torch==1.5.0, cuda==10.1, torchvision==0.6.0, single GPU

@fmassa Hi, could you give us some advice? I have experimented with both COCO datasets and custom dataset, and I didn't change any hyperparameters except gpu number. I have this problem all the time.
And my experiments is panoptic segmentation. 'assert (boxes1[:, 2:] >= boxes1[:, :2]).all()' happened on the first stage training: python main.py --coco_path /path/to/coco --coco_panoptic_path /path/to/coco_panoptic --dataset_file coco_panoptic --output_dir /output/path/box_model

[Chris-hughes10]

I also seem to be running into this error when training on custom data. It seems to happen with certain combinations of batch and image sizes according to no pattern that I can determine. I am feeding the bbox values into the network in xywh (normalised) format, and I don't seem to be running into any other errors.

[d-li14]

I met the same issue when training with the default setting on COCO detection.

python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --coco_path /path/to/coco 

[sarmientoj24]

any solutions on this?

[zhengye1995]

I meet the same issue, any solutions on this?