Detectron2: How to extract feature from any layer in a pre-trained detector?

[yeliudev]

Thanks for your great work! The performances of the detectors are amazing!

I have some questions on detectron2.

1. Now I have a list of bbox coordinates with respect to some images, how can I run a pre-trained detector (i.e. Faster-RCNN-R50-FPN) on these images, and crop the feature map (i.e. extract the 7x7 rois in the detector) according to the bbox coordinates? Is there an API in detectron2 to do the work?
2. How can I generate the proposals for Fast-RCNN, should I run the RPN directly to get the .pkl file?

Thank you again for your attention!

[ppwwyyxx]

UPDATE: see https://detectron2.readthedocs.io/tutorials/models.html#partially-execute-a-model

1.

from detectron2.modeling import build_backbone
from detectron2.modeling.poolers import ROIPooler
backbone = build_backbone(cfg)
pooler = ROIPooler(...)
rois = pooler(backbone(NCHW_images), [kx4 tensors])

1. Running inference on a dataset with a RPN-only model will produce the proposals for the dataset in output/inference.
   An example command is

python tools/train_net.py --config-file configs/COCO-Detection/rpn_R_50_FPN_1x.yaml --eval-only

[yeliudev]

Thank you for your kind reply!

As for Q1, regarding that the coordinates of the bboxes (either abs or rel ones) might change due to the pre-processing of images (i.e. padding or scaling), should I modify the [kx4 tensors] in your code manually to meet these transforms?

[ppwwyyxx]

The boxes should be absolute coordinates in XYXY, on the scale of [0, H or W], where H or W matches the size of tensor you give to the backbone.

[yeliudev]

Thanks

[kachiO]

Hello,
I have a related question. After running inference with Detectron2 and I get the class predictions and bounding boxes, how do I get the features associated with each detected object/region? It seems redundant to have to run re-run the images via the backbone and ROIPool.

[ppwwyyxx]

@kachiO the densepose example does what you described:
https://github.com/facebookresearch/detectron2/blob/817050985edb0a6a5ee6b795a6653dceed36d933/projects/DensePose/densepose/roi_head.py#L84-L85

[aligholami]

@ppwwyyxx I am confused. I think this line

rois = pooler(backbone(NCHW_images), [kx4 tensors])

has already generated a features for the regions. Is that right?

[ppwwyyxx]

That's right.

[Programmnix]

Hi,

for everybody how is interested in feature extraction for bounding boxes. There is a fork of this repo with example code. https://github.com/airsplay/py-bottom-up-attention

[albertchristianto]

Hi @ppwwyyxx,
given this line of code

rois = pooler(images_features, bboxes_coordinates)

rois are contain features of the bounding boxes and are corresponding each other, right?
for example
rois[0] is for bbox[0], and so on.

this image below is my implementation about it.

is it good ? or detectron2 already provides some function that can be use to extract features easily.
I want to extract the features and use it as inputs to my object tracker.

best regards,
Albert Christianto

[mahsaep]

from detectron2.modeling import build_backbone
from detectron2.modeling.poolers import ROIPooler
backbone = build_backbone(cfg)
pooler = ROIPooler(...)
rois = pooler(backbone(NCHW_images), [kx4 tensors])

1. Running inference on a dataset with a RPN-only model will produce the proposals for the dataset in output/inference.
   An example command is

python tools/train_net.py --config-file configs/COCO-Detection/rpn_R_50_FPN_1x.yaml --eval-only

Thanks for the code! I want to extract features for the predicted bounding boxes obtained by detectron at inference. Thus, 1) I used a defaultpredictor() to produce detections. then, 2) I used this code to extract feature for the obtained detections. I have two questions.
1) Is the built backbone initialized with pretrained weights for part 2?
2) Do I need to normalize input image for part 2?
Cheers,
Mahsa

[etiennedupont]

Hi @mahsaep,

1. The backbone when built is initialised randomly.
   Basically you can use the model and DetectionCheckpointer to load the weights referenced in the config file:

from detectron2.checkpoint import DetectionCheckpointer
from detectron2.modeling import build_model

model = build_model(cfg)
checkpointer = DetectionCheckpointer(model)
checkpointer.load(cfg.MODEL.WEIGHTS)
# Now the model (including its backbone) contains the weights in cfg.MODEL.WEIGHTS
backbone = model.backbone

1. Yes, the standard workflow used inside the models is below

from detectron2.structures import ImageList

# batched_inputs: a list, batched outputs of :class:`DatasetMapper`
images = [x["image"].to(model.device) for x in batched_inputs]
images = [(x - cfg.MODEL.PIXEL_MEAN) / cfg.MODEL.PIXEL_STD for x in images]
images = ImageList.from_tensors(images, model.backbone.size_divisibility)
features = model.backbone(images.tensor)

[mahsaep]

Hi @mahsaep,

1. The backbone when built is initialised randomly.
   Basically you can use the model and DetectionCheckpointer to load the weights referenced in the config file:

from detectron2.checkpoint import DetectionCheckpointer
from detectron2.modeling import build_model

model = build_model(cfg)
checkpointer = DetectionCheckpointer(model)
checkpointer.load(cfg.MODEL.WEIGHTS)
# Now the model (including its backbone) contains the weights in cfg.MODEL.WEIGHTS
backbone = model.backbone

1. Yes, the standard workflow used inside the models is below

from detectron2.structures import ImageList

# batched_inputs: a list, batched outputs of :class:`DatasetMapper`
images = [x["image"].to(model.device) for x in batched_inputs]
images = [(x - cfg.MODEL.PIXEL_MEAN) / cfg.MODEL.PIXEL_STD for x in images]
images = ImageList.from_tensors(images, model.backbone.size_divisibility)
features = model.backbone(images.tensor)

Thank you very much!

[ssd-ls-a]

@ppwwyyxx
I want to use the RoI Pooled features from mask head (using Mask R-CNN). How is that possible?

[ppwwyyxx]

The tutorial has examples for such questions: https://detectron2.readthedocs.io/tutorials/models.html#partially-execute-a-model

[ssd-ls-a]

Thank you very much!

Am 02.11.2020 um 20:58 schrieb Yuxin Wu [e-mail]:

The tutorial has examples for such questions: https://detectron2.readthedocs.io/tutorials/models.html#partially-execute-a-model https://detectron2.readthedocs.io/tutorials/models.html#partially-execute-a-model
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/facebookresearch/detectron2/issues/5#issuecomment-720691863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQCWGN4M25M2GENY7TEVYQ3SN4FPXANCNFSM4I7UUPVA.