Vision: It seems the IntermediateLayerGetter will not use the forward function in the original model

[songyuc]

Hi guys,
I am working on reproducing DeepLabV3+ model these days,
and I need to get some intermediate layers from the backbone.
And I found a class of IntermediateLayerGetter with similar effect.
When I read the code, I found that in the forward function of IntermediateLayerGetter, that the IntermediateLayerGetter would do the inference like,

for name, module in self.named_children():
            x = module(x)
            # rest of the code

But this may mean that, the IntermediateLayerGetter class would not use the forward function in the original wrapped model, which seems unsuual.
I don't think this would let the original model behave in a right way.

Any explanation or idea would be appreciated!

[cpuhrsch]

Hello @songyuc - could you link the code snippets you're mentioning?

[songyuc]

@cpuhrsch ，hi,
here is the link of IntermediateLayerGetter.

[fmassa]

Hi @songyuc

Indeed, IntermediateLayerGetter is a simplistic module that doesn't take the forward of the original module into account. It's because of this limitation that IntermediateLayerGetter is present in the private file _utils.py, because this means it should be used with care and its API can change at any time. We make some warnings in the documentation https://github.com/pytorch/vision/blob/5306635396d1ebd3047d56b16745350cf8c1e2c4/torchvision/models/_utils.py#L12-L19 , but maybe we should add some more noting that the forward of the model is not taken into account.

For now, the most correct way of extracting intermediate features from a model forward (although it's not 100% bullet-proof, like when a module is repeatedly used) is to use forward hooks in your model.
Here is an example:

outs = []
hook0 = model.layer1.register_forward_hook(lambda self, input, output: outs.append(output))
output = model(inputs)
hook0.remove()
print(outs[0])

I've opened an issue in PyTorch in https://github.com/pytorch/pytorch/issues/21064 to discuss about this, as I think that a solution based on torchscript could be a good way to address this.

Let us know if you have any further questions.

[songyuc]

Hi, @fmassa Fancisco, thank you for your explanation and the code.
I am so excited to find that I have the same idea with you that the hook is the best way in current PyTorch to obtain the intermediate features from the backbone model.

[fmassa]

Cool, let us know if you find any other issues!