Fairseq: Export a fairseq trained transformer (base) model

[gvskalyan]

I trained a translation model based on scaling neural machine translation and wanted to convert into into caffe2 but when used docker image it is throwing an error ::

According to https://github.com/pytorch/translate/issues/60 only RNN trained models are supported ?
How about the transformer trained models .
Should I retrain again to export to caffe2 ? @myleott

Is this under works or is their a problem with the command, as in some other command to export transformer?

[myleott]

Hmm, I'm not sure the latest status of ONNX export for Transformer models, but generally ONNX export is handled by pytorch/translate. Thoughts @jmp84?

[jmp84]

@gvskalyan, export for transformer is working for us. Have you tried training a transformer with pytorch/translate directly?

[gvskalyan]

Yeah, will train in translate and let you know. Closing For now.

It would be good to bring all the translate domain under a single repository/code base (translate + fairseq) and this helps in future to incorporate changes and representations like these https://github.com/facebookresearch/LASER

[Esaada]

hi @jmp84 , I think that you're the only one online that succeeded to get transformer onnx, is there any chance of sharing the onnx file on some drive? or any other way?
And maybe show you command, and where you wrote you command, I'm not kidding when saying that I saw no one, actually no one succeeding this export!

[Charleshzhang]

hi @myleott, I noticed that in /pytorch_translate/ensemble_export.py, there's a list of model supported:

https://github.com/pytorch/translate/blob/master/pytorch_translate/ensemble_export.py

and that's where i stuck when trying model export
will there be more in the future or do your team have such plan?

[jmp84]

@Charleshzhang, see https://github.com/pytorch/fairseq/pull/1620 for scripting transformer.
Please use fairseq instead of translate (some functionality in translate will be moved to fairseq and translate will be deprecated).

[Charleshzhang]

@Charleshzhang, see #1620 for scripting transformer.
Please use fairseq instead of translate (some functionality in translate will be moved to fairseq and translate will be deprecated).

Thanks for reply, I tried it today, using the method in 'export unittest' and find myself ran into such problem:

"Compiled functions can't take variable number of arguments "
"or use keyword-only arguments with defaults"

in https://github.com/pytorch/pytorch/blob/master/torch/jit/frontend.py line 223

py_args.kwarg not support

So is this going to be supported in the future, or if I did something wrong with my code? Many thanks.

P.S. I am trying exporting a 'transformer_vaswani_wmt_en_de_big' Arch. network to onnx, from a pre-built check-point file, and it's base class have forward Func like forward( ... , **kwargs)

[jmp84]

@Charleshzhang ONNX will not be supported, we are only working with TorchScript (https://pytorch.org/docs/stable/jit.html).
You probably want to adapt the test_export_transformer function from https://github.com/pytorch/fairseq/pull/1620

[Charleshzhang]

@Charleshzhang ONNX will not be supported, we are only working with TorchScript (https://pytorch.org/docs/stable/jit.html).
You probably want to adapt the test_export_transformer function from #1620

Got it, thanks a lot.
感谢大佬

[gvskalyan]

is there a way to export completely from end-end including sequence_generator, like in https://github.com/pytorch/translate/blob/master/pytorch_translate/ensemble_export.py#L1137

[cndn]

@gvskalyan, we are working on a export solution for a simplified sequence generator, that works with scripted models like transformer. It's still under testing.

[jahutwb]

Hi!
@cndn how are you doing with these tests? Is there a way to to try it myself? What would I have to do to export MT model to TorchScript?

[Mr-wang2016]

@jahutwb, hi, do you know how to export a MT model to TorchScript now? I still can't find a solution to this.

[jahutwb]

look here #2439

[jmp84]

You can export an MT model with the following code: https://gist.github.com/jmp84/b98dfc5788b530d594220a552e4fdb27

[gvskalyan]

@jmp84 In the gist, can you please tell how to use the scripted_generator, it is getting stuck when:

    generator = SequenceGenerator([model], model_dict, beam_size=args.beam_size)
    src_tokens = torch.randint(3, 50, (2, 10)).long()
    src_tokens = torch.cat((src_tokens, torch.LongTensor([[eos], [eos]])), -1)
    src_lengths = torch.LongTensor([2, 10])
    sample = {
            "net_input": {"src_tokens": src_tokens, "src_lengths": src_lengths}
        }
    generator(sample)                                        # Working
    scripted_generator = torch.jit.script(generator)
    scripted_generator(sample)                               # Not Working

Sources - 1,2

[jmp84]

@gvskalyan could you open a separate issue with a repro (including model training and error messages, etc.)?