With regular pytorch tensors, I can just insert a breakpoint (I'm using PyCharm and VS Code, though debugging with VS Code is painfully slow) somewhere inside the model and just take a look at the values. For XLA tensors, I can never see the tensor values and doing something like logits.cpu()[0, 0] just results in a timeout...
Does anyone have nice debugging tips for torch-xla? (Other than prints. I know how to print)
harpone
All 4 comments
Timeout? It is likely compiling.
dlibenzi
on 31 Jan 2020
@harpone I guess the timeout is from PyCharm/VS Code? Could you try increasing that value a bit as the first compile is expected to take much longer than the latter runs?
I'm working in a terminal where the logits.cpu() also returns without timeout, one trick I found out is querying less elements would take much less time like logits[0].cpu() and I usually try to avoid querying the whole tensor in debugging.
ailzhang
on 31 Jan 2020
fwiw, I look at xla tensors pretty often in an ssh session inside a terminal , using pdb and it works reliably for me.
taylanbil
on 31 Jan 2020
OK I think it could actually be due to VS Code's and PyCharm's variable inspector, which tries to load all kinds of info about the tensors (pdb + terminal clearly doesn't have that problem). I think I could try to just disable all variable inspectors etc. I'll report my findings. Thanks all!
harpone
on 1 Feb 2020
Related issues
mruberry
路
4Comments
ibeltagy
路
8Comments
Clive2312
路
6Comments
myleott
路
6Comments
magicknight
路
3Comments
Most helpful comment
@harpone I guess the timeout is from PyCharm/VS Code? Could you try increasing that value a bit as the first compile is expected to take much longer than the latter runs?
I'm working in a terminal where the
logits.cpu()also returns without timeout, one trick I found out is querying less elements would take much less time likelogits[0].cpu()and I usually try to avoid querying the whole tensor in debugging.