Transformers: LXMERT visual feature extraction during training/fine-tuning phase

[mmiakashs]

🚀 Feature request

Thanks a lot for releasing LXMERT model. In the LXMERT model code samples, the visual feature extraction code (using generalized faster-rcnn: modeling_frcnn) only in the inference step is given. However, the visual feature extraction during the training phase is not given. For this reason if we use the same code for fine-tuning, it raises NotImplementedError as the visual feature extraction during training is not implemented. Is it possible to share the visual feature extraction during training?

[TashinAhmed]

Yes, I also came up with this error. It would be great if the feature gets published. TIA.

[LysandreJik]

Tagging LXMERT's implementation author @eltoto1219

[eltoto1219]

Haha, yes we only added the FRCNN for evaluation to accommodate lxmert in the demo. I'll add the training code sometime this week, and then post back here once it is done, in the future it may be useable as a publicly available model following the HF api, but for the time being ill just push the changes to where it is now.

[TashinAhmed]

Thanks for the prompt feedback.
Looking forward to it.
@eltoto1219

[mmiakashs]

@eltoto1219 thanks, that will be quite a help.

[xhyandwyy]

@eltoto1219 Looking forward to it.