Yolact: Features From Intermediate Layers

[bayraktare]

Hi @dbolya, congrats for the Yolact++. It seems great improvement.
Maybe this is trivial for you but I am a little bit stuck due to a confusion.
I wonder how can I get features from intermediate layers? For example from C5 or P3 or P7 or right before prediction head or right before Protonet?
I will just try all of these if they are meaningful enough to refine the results.
Thanks for your attention. I’m looking forward to your reply.

[manyaafonso]

Hi @dbolya , can you please respond to this issue? How can we generate figures similar to fig 5 in your ICCV 2019 paper? Thanks.

[dbolya]

@bayraktare To get the features from before the prediction head, you could either use x from here:
https://github.com/dbolya/yolact/blob/1d3f7bb94f48dff6324b22483161374c35567306/yolact.py#L133
and use self.index to identify which prediction head it is.

Alternatively, you can use pred_x from here:
https://github.com/dbolya/yolact/blob/1d3f7bb94f48dff6324b22483161374c35567306/yolact.py#L620

The protonet one should be index 0 in that stack (or the last index, but probably just index 0).

@manyaafonso In order to get the figure 5 stuff, just run eval.py with --mask_proto_debug, though you may need to modify the code to get it to work because I'm not sure what state I left it in (that's not supposed to be user-facing, after all). The function it calls is here:
https://github.com/dbolya/yolact/blob/1d3f7bb94f48dff6324b22483161374c35567306/layers/output_utils.py#L147
and I'm pretty sure the version in fig.5 didn't multiply in the coefficients.

[bayraktare]

Can't I use https://github.com/dbolya/yolact/blob/1d3f7bb94f48dff6324b22483161374c35567306/yolact.py#L133 as follows directly?

import torchvision.models as models
model = models.FPN.forward(pretrained=True) # Initialize the model with default weights
feature_embedding=model.predict(image) # Utilize model for prediction

[dbolya]

@bayraktare I'm not sure what you mean by that?

I guess you're trying to use a torchvision pretrained FPN and pass those features into the YOLACT prediction head? In that case, the way the FPN was trained in YOLACT and the way some pretrained version of it in torch's model zoo are completely different, and the features aren't likely to be compatible. So I don't think that would work.

[bayraktare]

Hi @dbolya.
No, I am not trying to use another thing. I am just trying to obtain features from intermediate layers of YOLACT. In particular, the features i try to get are at just before the mask prediction. I will use these features for concatenating them with semantics and forming an embedding to feed another shallow model. And i also continue using the outputs of the YOLACT together. So i am trying to get something similar to my above comment that will seem like this;

model = the_model_just_before_mask_prediction()
output_features=model.predict(input_image/video)

Then, i will use this output_features for the operations i wrote above.

[dbolya]

@bayraktare Oh I see. Do you want the protonet activation then? Or the activations before protonet?

If you want to stop right after the FPN activations (before protonet / prediction heads), you can just put return outs here: https://github.com/dbolya/yolact/blob/eee5ce36242fa1de808a287a3cfe6055486e42b5/yolact.py#L578

If you want just protonet activations, you can put return proto_out here: https://github.com/dbolya/yolact/blob/eee5ce36242fa1de808a287a3cfe6055486e42b5/yolact.py#L605

In both cases to use this you would go

model = Yolact()
# ...
feats = model(input_image/frame)

In the FPN case feats will be a list of tensors, while in the protonet case it will be a tensor.

[bayraktare]

@dbolya actually, i wanted to see the performance for both.
So, thanks a lot for this satisfying reply. i will let you know about the results.

[bayraktare]

Hi @dbolya i could have been tried a few times but unfortunately couldn't overcome the issues.

    classes, scores, boxes, masks = postprocess(preds , w, h, score_threshold=cst)

  File ".../yolact/layers/output_utils.py", line 41, in postprocess
    keep = dets['score'] > score_threshold

IndexError: too many indices for tensor of dimension 3

When i try to get outputs from postprocess which calls output_utils, the tensor shape is failing to to continue as above. I placed return proto_out here as you wrote at the same indent with the above if:
https://github.com/dbolya/yolact/blob/eee5ce36242fa1de808a287a3cfe6055486e42b5/yolact.py#L605
If I put it at the same indent with the above line, it gives (but this is not our case):

  File ".../yolact/yolact.py", line 595, in forward
    _, _, img_h, img_w = x.size()

TypeError: 'int' object is not callable

If I trace the keep = dets['score'] > score_threshold, it gives the following:
A tensor tensor([[[.....]]..., [[.......]], ....[[........]], [[....]]]]) with the size of

dets.size()--->torch.Size([138, 138, 32])
w = 1920
h = 1080
batch_idx = 0
interpolation_mode = 'bilinear'
visualize_lincomb = False
crop_masks = True
score_threshold = 0.2

[dbolya]

@bayraktare
After the changes you should instantiate a yolact object and call it directly. Either that, or you can just save the detections as a numpy array instead of returning it from YOLACT. In your first snippet, you could also just save preds. If you want to use these features for some downstream task, then I'm not sure why you're calling postprocess in the first place?

I guess if you want to use both the masks and the intermediate features, instead of returning outs or returning proto_out, you can just add it to the output dictionary. Here:
https://github.com/dbolya/yolact/blob/eee5ce36242fa1de808a287a3cfe6055486e42b5/yolact.py#L638
you could add:

pred_outs['feats'] = outs

then the prototype activations is already in pred_outs['proto'] so you can just use that. Then to get this to not error out in postprocess, just change:
https://github.com/dbolya/yolact/blob/eee5ce36242fa1de808a287a3cfe6055486e42b5/layers/output_utils.py#L46
to

if k != 'proto' and k != 'feats':

Then when you use this downstream, when you have

classes, scores, boxes, masks = postprocess(preds , w, h, score_threshold=cst)

# You can access feats here
preds['feats'] # Do stuff with prediction head inputs
preds['proto'] # Do stuff with protonet activations

[bayraktare]

Hi @dbolya, thank you so much.
It is working now. But a simple modification for example preds[0]['proto'] since preds is a list and preds[0] is a dict.
On the other hand, there is nothing related to preds['feats'] or preds[0]['feats'] due to nothing in preds about ['feats']. It completely gives error.

[dbolya]

@bayraktare

On the other hand, there is nothing related to preds['feats'] or preds[0]['feats'] due to nothing in preds about ['feats']. It completely gives error.

Oops, you'll also have to add

result['feats'] = predictions['feats']

here:
https://github.com/dbolya/yolact/blob/092554ad707c2749631dfe545c8a953b2b3f4a68/layers/functions/detection.py#L74

[bayraktare]

Thanks, it runs properly with these modifications.
Really cool! I think this issue can be closed...

[bayraktare]

Sorry, @dbolya but I have a further question related to this issue.
Can I get object specific protonet features?
For example; there are 5 objects detected in the image. So, I want to get 5 different protonet features (can be the dimension 138x138xk or something else) for each object independently, is this possible?
If this is not possible, can I get this kind of object specific features from the prediction head part?
Thanks for your help again.

[dbolya]

@bayraktare Each object uses the same k features. The object specific information comes from the mask coefficient branch (I think there should be a 'mask' element in that same dict above? Those are the mask coefficients).