Fairseq: ONNX export (for transformer)

[starsplash]

Hi,

I've seen that you have started to work on ONNX export.
I tried it on my own for the transformer model but have issues here:

File "fairseq/fairseq/models/fairseq_model.py", line 160, in forward
    decoder_out = self.decoder(prev_output_tokens, encoder_out)
[...]
ValueError: Auto nesting doesn't know how to process an input object of type dict. Accepted types: Tensors, or lists/tuples of them

The issue is that "encoder_out" is a dictionary.
Any idea how to solve that in an elegant way (other than converting to list)? :)
Or will be more support from pytorch soon?

Update: Conversion is not possible right now since "expand_as" and "ne" are not implemented/merged yet (see https://github.com/pytorch/pytorch/issues/10882 and https://github.com/pytorch/pytorch/pull/10914).

Cheers!

[myleott]

I believe ONNX export is still a work in progress. What’s the latest status with this @jhcross?

[myleott]

I spoke with @jhcross and apparently ONNX export works when using the latest fairseq and pytorch! Please reopen if this is still an issue.

[vince62s]

@jhcross are you sure the transformer is actually exportable ?
it uses some expand_as and masked_fill functions that are not currently supported by onnx export.
Just curious if it actually works.

[jhcross]

@jamesr66a can you suggest the best way to confirm that transformer export works with open-source dependencies (and fix if there is a problem)?

[Esaada]

I would like to know also. because I have tons of errors.

[jhcross]

I'm hoping this will be resolved after official PyTorch 1.0 launch. @jamesr66a, do we know what version is necessary for ONNX export of expand_as and masked_fill?

[jamesr66a]

Hi everyone,

It is in fact the case that the Transformer model is not supported by pure-ONNX export, but rather it is exportable via the ONNX-ATen fallback path. For an example of how that is enabled, see this PR: https://github.com/pytorch/pytorch/pull/14492. With ONNX ATen fallback, an ONNX backend that supports running ops from the ATen library (i.e. PyTorch's library) can interpret these ops and run the original implementations directly.

The reason this is necessary is that the ONNX specification does not support many operators that are used in modern PyTorch code, such as the Transformer network. Adding operators to ONNX involves a lengthy approval process, and requires writing code for both ONNX front-ends and back-ends. So, to provide a solution that works earlier, this fallback path is available.

cc @zrphercule do you want to take a look at the ops missing for this use case and make sure they're on the roadmap for addition?

[zrphercule]

Thanks James!

Yeah currently we are working on increasing the exporting coverage of pytorch=>onnx. We just finished a few operators like LogSigmoid and RReLU, and I am happy to put expand_as and masked_fill on my list. But as I remember, expand_as is going to be a complicate one, since we have similar but also different definitions of "expand" op in onnx, which is a balance between a few different framework and make the export of torch.expand_as much difficult.

We will try to find a solution regards your requirements of expand_as and masked_fill, and right now, like @jamesr66a suggested, we encourage you to use ATen fallback when exporting pytorch models.

To enable ATen fallback, you can add a param when calling the onnx.export function. For example, you have a calling function like this:

torch.onnx._export(a, b, c)

Now, if you want to use ONNX_ATEN_FALLBACK, just change it to:

torch.onnx._export(a, b, c, operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK)

btw, if you want us to have any new operators in onnx, please feel free to add some comments in
https://github.com/onnx/onnx/issues/1646
yes we are collecting such requests :)

[fumihwh]

@starsplash Did you export onnx successfully after adding torch.onnx.symbolic.ne and torch.onnx.symbolic.expand_as?
BTW, I stuck in following error.

  File "/home/hwh193906/.pyenv/versions/3.7.2/lib/python3.7/site-packages/torch/onnx/utils.py", line 132, in _optimize_graph
    torch._C._jit_pass_constant_propagation(graph)
RuntimeError: 
Schema not found for node. File a bug report.
Node: %279 : Long(31, 128) = aten::masked_scatter(%270, %276, %278), scope: TransformerModel/TransformerEncoder[encoder]/SinusoidalPositionalEmbedding[embed_positions]

I insert following code to L91 train.py to export.

torch.onnx.export(model, (
        dummy_batch["net_input"]["src_tokens"].to("cuda"), dummy_batch["net_input"]["src_lengths"].to("cuda"),
        dummy_batch["net_input"]["prev_output_tokens"].to("cuda")),
                       "fairseq_transformer.onnx", verbose=True,
                       operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK)

[stephanpeitz]

@fumihwh No, I haven't but I wanted to try tonight :) I will let you know if I had success!

[stephanpeitz]

@fumihwh Nope, doesn't work for me as well (out of the box). It seems to need some additional effort to make it work, e.g. we need to keep track of self_attn_state (transformer.py, L746FF).

[vipulbjj]

Unable to cast from non-held to held instance (T& to Holder) (compile in debug mode for type information)

I'm getting this error after adding ONNX_ATEN_FALLBACK as mentioned by @zrphercule .
Did anyone solve this?