Detr: Run panoptic segmentation

[SharifElfouly]

Great paper and repo btw, congrats!

I wanted to do a simple single image inference with the panoptic segmentation model. I archived that in a very 'hacky' way by editing the main.sh file (Can be seen here).

Are you planing to release a demo notebook like the one for object detection or uploading the .pth files to torch.Hub?

[fmassa]

Hi,

Thanks for the message!

We are planning on releasing a colab notebook illustrating how to perform predictions with detr_panoptic, and we will also include torchhub models for it. It should be available sometime next week.

[SharifElfouly]

Awesome, thanks for the info!

[alcinos]

Hi all, the panoptic notebook is now available here https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/DETR_panoptic.ipynb
cc @shidilrzf
Let me know what you think.

[SharifElfouly]

Thank you this is super helpful!

I had to upgrade pyyaml for whatever reason.
Just run:
pip3 install pyyaml -U
if you run into the same error.

[shidilrzf]

Thanks a lot! That really helped.
I had some problems with metadata to be able to use detecton2 Visualizer. The category-id reported in the "segment_info" was different than the one used in detectron2.

[alcinos]

Thanks a lot! That really helped.
I had some problems with metadata to be able to use detecton2 Visualizer. The category-id reported in the "segment_info" was different than the one used in detectron2.

Yes indeed, Detectron2 does a remapping to ensure contiguous IDs, which we don't. Did the Notebook clarify that part for you?

[rkinas]

I made simple demo using DETR and panoptic segmentation here:

https://youtu.be/CJj6V1eafVs

Each class has predefined color (to prevend color blinking on video).

[shidilrzf]

Thanks a lot! That really helped.
I had some problems with metadata to be able to use detecton2 Visualizer. The category-id reported in the "segment_info" was different than the one used in detectron2.

Yes indeed, Detectron2 does a remapping to ensure contiguous IDs, which we don't. Did the Notebook clarify that part for you?

Yes, thanks a lot ;)

[alcinos]

I believe the issue at hand was addressed, as such I'm closing this. Feel free to ask if you have further questions.

[NIkimih]

Is it possible to train panoptic segmentation on my dataset, which is not in COCO format? @alcinos

[alcinos]

@NIkimih We currently only support COCO format. You'll need to convert your dataset, see https://cocodataset.org/#format-data for information on what is expected. You essentially need three things: the images, some jsons (for train and val) describing the segments (class, associated bounding box, id), and a specially formatted png encoding per-pixel segment ids (the color correspond to the id, as described in the link)

[NIkimih]

@NIkimih We currently only support COCO format. You'll need to convert your dataset, see https://cocodataset.org/#format-data for information on what is expected. You essentially need three things: the images, some jsons (for train and val) describing the segments (class, associated bounding box, id), and a specially formatted png encoding per-pixel segment ids (the color correspond to the id, as described in the link)

Thanks for the quick response