Coremltools: Conversion of a pytorch with slicing creates op 'to[]' with 8 arguments and such is not defined in ops.py

[j-paulus]

🐞Describe the bug

Conversion of a pytorch model that uses indexing with a torch.tensor produces an intermediate graph with an operator to[] having 8 arguments. The pytorch operator definitions support only up to 6 arguments.

In the example code below, the indexing tensor is defined in forward(), but in the real model it is a member variable of the model class defined in the constructor depending on the arguments.

Trace

File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 303, in convert
proto_spec = _convert(
File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 134, in _convert
prog = frontend_converter(model, *kwargs)
File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 84, in __call__
return load(args, **kwargs)
File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 86, in load
raise e
File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 76, in load
prog = converter.convert()
File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 224, in convert
convert_nodes(self.context, self.graph)
File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 55, in convert_nodes
_add_op(context, node)
File "/Users/name/opt/anaconda3/envs/coreml_test/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 1782, in to
raise ValueError(
ValueError: Received invalid arguments for PyTorch conversion of op %tmp_indices = to

To Reproduce

import torch
import coremltools as ct

class TriggerBug(torch.nn.Module):
    def __init__(self):
        super(TriggerBug, self).__init__()

    def forward(self, x):
        tmp_indices = torch.tensor([0,])
        return x[:, tmp_indices]

model = TriggerBug()
x = torch.randn((3, 10))
traced_model = torch.jit.trace(model, x)
model_ct = ct.convert(traced_model, inputs=[ct.TensorType(shape=ct.Shape(x.shape), name='x')])

System environment (please complete the following information):

• coremltools version: 4.0b3
• OS: MacOS
• macOS version: 10.15.6
• How you install python (anaconda, virtualenv, system): anaconda
• python version: 3.8.5
• pytorch version: 1.6.0

Additional context

Add any other context about the problem here.

[leovinus2001]

FYI I have seen a similar issue with to() having an None argument as final arg and then bailing out as it was one more arg then what was expected. I built a composite operator to() which did the trick. A bit more robustness for to() would be great indeed.

[j-paulus]

The unsupported argument list is created by TorchConverter._expand_and_optimize_ir() from the provided torchscript representation. More specifically, by the call to _torch._C._jit_pass_lower_graph() on line 250 of converter.py.

There, the arguments are given as tmp_indices0 = torch.to(tmp_indices, dtype=4, layout=0, device=torch.device("cpu"), pin_memory=False, non_blocking=False, copy=False, memory_format=None)

So my guess for the needed branch would be

if len(inputs) == 8:
    _input = inputs[0]
    dtype = inputs[1].val
    # layout = input[2]
    device = inputs[3]
    # pin_memory = inputs[4]
    # non_blocking = inputs[5]
    # copy = inputs[6]
    # memory_format = inputs[7]

This solves the acute problem, but triggers another missing operator, but this is then a different problem:

RuntimeError: PyTorch convert function for op 'index' not implemented.

which is caused by the last statement in the torchscript code:

def forward(self,
    x: Tensor) -> Tensor:
  _0 = torch.to(CONSTANTS.c0, torch.device("cpu"), 4, False, False, None)
  tmp_indices = torch.detach(_0)
  _1 = torch.slice(x, 0, 0, 9223372036854775807, 1)
  tmp_indices0 = torch.to(tmp_indices, dtype=4, layout=0, device=torch.device("cpu"), pin_memory=False, non_blocking=False, copy=False, memory_format=None)
  _2 = annotate(List[Optional[Tensor]], [None, tmp_indices0])
  return torch.index(_1, _2)

[j-paulus]

Fix pull request: #903

[j-paulus]

It seems that the 4.0 release contains the fix to the extra arguments.
The second problem with the missing operator 'index' still exists.

[shiron8bit]

@j-paulus Could you somehow solve the problem with 'index'?

[j-paulus]

@shiron8bit No, not directly. I ended up using a work-around with torch.cat() in my real network. I.e., instead of defining indices in an array and using that for indexing, slicing with each index separately and concatenating these. Doesn't look nice, but works.

[logan40hands]

@j-paulus can you be more specific on how you changed the network to get past this issue?