馃悰 Bug
Using the output of xla's nonzero caused a failure on GPU backend and inconsistensy result on TPU backend
To Reproduce
Steps to reproduce the behavior:
import torch
import torch_xla
import torch_xla.core.xla_model as xm
import torch_xla.debug.metrics as met
import torch_xla.core.functions as xf
def remove_small_boxes(boxes, min_size):
# type: (Tensor, float) -> Tensor
"""
Remove boxes which contains at least one side smaller than min_size.
Arguments:
boxes (Tensor[N, 4]): boxes in (x1, y1, x2, y2) format
min_size (float): minimum size
Returns:
keep (Tensor[K]): indices of the boxes that have both sides
larger than min_size
"""
ws, hs = boxes[:, 2] - boxes[:, 0], boxes[:, 3] - boxes[:, 1]
keep = (ws >= min_size) & (hs >= min_size)
keep = keep.nonzero().squeeze(1)
return keep
def test():
BOXES = (
(0, 0, 3, 2),
(3, 3, 11, 7),
(2, 2, 5, 7),
(7, 4, 15, 12),
(1, 1, 2, 2)
)
SCORES = (0.9, 0.5, 0.95, 0.4, 0.3)
SCORE_THRESHOLD = 0.1
IOU_THRESHOLD = 0.08
xla_device = xm.xla_device()
boxes = torch.tensor(BOXES, dtype=torch.float).to(xla_device)
scores = torch.tensor(SCORES, dtype=torch.float).to(xla_device)
keep = remove_small_boxes(boxes, 1.2)
boxes = boxes[keep]
scores = scores[keep]
print(boxes)
print(scores)
if __name__ == "__main__":
test()
Then
$ XLA_EXPERIMENTAL="nonzero" python test.py
In the colab TPU environment:
The output is
tensor([[ 0., 0., 3., 2.],
[ 3., 3., 11., 7.],
[ 2., 2., 5., 7.],
[ 7., 4., 15., 12.],
[ 1., 1., 2., 2.]], device='xla:1')
tensor([0.9000, 0.5000, 0.9500, 0.4000], device='xla:1')
In the remote GPU environment
The second print will caused a remote GPU failure:
RuntimeError: Internal: From /job:worker/replica:0/task:0:
2 root error(s) found.
(0) Internal: Failed to complete all kernels launched on stream 0x5590e11f5410: Could not synchronize CUDA stream: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address
[[{{node XRTExecute}}]]
[[XRTExecute_G12]]
(1) Internal: Failed to complete all kernels launched on stream 0x5590e11f5410: Could not synchronize CUDA stream: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address
[[{{node XRTExecute}}]]
When I try to print keep and keep.shape, the result of keep shows 4 elements but the keep.shape outputs 5.
# print(keep)
tensor([0, 1, 2, 3], device='xla:1')
# print(keep.shape)
torch.Size([5])
Expected behavior
keep is [0,1,2,3]. The final result of scores and boxes should have the same value in 1st dimension, which is 4.
Environment
- Reproducible on XLA backend [CPU/TPU]: yes
- torch_xla version: 1.6
Additional context
Clive2312
All 6 comments
I havne't look at GPU runtime error but I can explain your TPU observation.
In short, shape is not an accurate way of telling the actual size of a dynamic xla tensor, you should be torch_xla._XLAC._get_xla_tensor_dimension_size to do that, example in
https://github.com/pytorch/xla/blob/02e270008cfabb6417e07d28d469764e3c795c8c/test/test_operations.py#L712
The reason behind this is related to PT/XLA's lazy tensor approach. Ideally, we should not execute the graph until hitting a xm.mark_step() at the end of the training loop. However we need to know the shape of the intermediate result to construct the graph. We want the graph to be the same so we can hit the cache and don't need to recompile everytime. Ops like nonzero will introduce the dynamic shape which is challenging because
- we need to compile/execute early to know the actual size of the result (runtime shape)
- size might be different each time so we will keep recompiling
PT/XLA's way of handling dynamic shape (which is experimental right now) is always allocate maximum amount of memory for a dynamic tensor. For example, if you use torch_xla._XLAC._get_xla_tensors_hlo([keep]) to print the hlo text of the keep you will see at the end
%reshape.69 = s32[<=5]{0} reshape(s32[<=5,1]{1,0} %set-dimension-size.68), inferred_dimension=0, metadata={op_type="aten::view" source_file="_vecto
r_str@_tensor_str.py" source_line=189}
ROOT %tuple.70 = (s32[<=5]{0}) tuple(s32[<=5]{0} %reshape.69)
This means we know that keep will be a 1d tensor with at most 5 elements. In the compile time we don't know the exact size of the keep but we can still build the graph. In the runtime xla compiler will smartly only perform the execution with the actual value of the tensor. We keep the shape to be static to pervent early execution and use _get_xla_tensor_dimension_size to access the real size of a dynamic tensor after the execution.
JackCaoG
on 6 Nov 2020
Thanks for your explaination.
But it still confused me that the number of elements in scores[keep] is 4 while the number of boxes in boxes[keep] is 5.
BTW, I've encountered another problem when I tried to apply nms after remove small boxes:
def test_nms():
BOXES = (
(0, 0, 3, 2),
(3, 3, 11, 7),
(2, 2, 5, 7),
(7, 4, 15, 12),
(1, 1, 2, 2)
)
SCORES = (0.9, 0.5, 0.95, 0.4, 0.3)
SCORE_THRESHOLD = 0.1
IOU_THRESHOLD = 0.08
xla_device = xm.xla_device()
boxes = torch.tensor(BOXES, dtype=torch.float).to(xla_device)
scores = torch.tensor(SCORES, dtype=torch.float).to(xla_device)
keep = remove_small_boxes(boxes, 1.2)
boxes = boxes[keep]
scores = scores[keep]
keep = xf.nms(boxes, scores, torch.tensor(0, device=xla_device, dtype=torch.float), torch.tensor(IOU_THRESHOLD, device=xla_device, dtype=torch.float), len(boxes))[0]
print(keep)
I've run this code on the colab tpu environment and got a rpc failure on print(keep)
2020-11-06 08:01:15.784969: W tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:160] RPC failed with status = "Unavailable: Socket closed" and grpc_error_string = "{"created":"@1604649675.784874538","description":"Error received from peer ipv4:10.12.84.106:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}", maybe retrying the RPC
2020-11-06 08:01:31.967016: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] >>> Dumping Computation 0
2020-11-06 08:01:31.967092: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] HloModule SyncTensorsGraph.282
2020-11-06 08:01:31.967105: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.967120: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add_S32.52 (lhs.53: s32[], rhs.54: s32[]) -> s32[] {
2020-11-06 08:01:31.967128: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %lhs.53 = s32[] parameter(0)
2020-11-06 08:01:31.967136: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %rhs.54 = s32[] parameter(1)
2020-11-06 08:01:31.967148: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %add.55 = s32[] add(s32[] %lhs.53, s32[] %rhs.54)
2020-11-06 08:01:31.967161: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.967170: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.967179: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare-greater-than.59 (p.0.lhs.60: s32[], p.0.rhs.61: s32[], p.1.lhs.62: s32[], p.1.rhs.63: s32[]) -> pred[] {
2020-11-06 08:01:31.967189: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.lhs.62 = s32[] parameter(2)
2020-11-06 08:01:31.967199: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.rhs.63 = s32[] parameter(3)
2020-11-06 08:01:31.967207: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.64 = pred[] constant(true)
2020-11-06 08:01:31.967226: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.65 = pred[] broadcast(pred[] %constant.64), dimensions={}
2020-11-06 08:01:31.967239: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.lhs.60 = s32[] parameter(0)
2020-11-06 08:01:31.967249: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.rhs.61 = s32[] parameter(1)
2020-11-06 08:01:31.967259: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.66 = pred[] compare(s32[] %p.0.lhs.60, s32[] %p.0.rhs.61), direction=GT, type=TOTALORDER
2020-11-06 08:01:31.967274: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %select.67 = pred[] select(pred[] %broadcast.65, pred[] %compare.66, pred[] %broadcast.65)
2020-11-06 08:01:31.967294: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.967303: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.967312: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare-greater-than.109 (p.0.lhs.110: f32[], p.0.rhs.111: f32[], p.1.lhs.112: f32[], p.1.rhs.113: f32[]) -> pred[] {
2020-11-06 08:01:31.967322: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.lhs.112 = f32[] parameter(2)
2020-11-06 08:01:31.967334: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.rhs.113 = f32[] parameter(3)
2020-11-06 08:01:31.967344: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.114 = pred[] constant(true)
2020-11-06 08:01:31.967353: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.115 = pred[] broadcast(pred[] %constant.114), dimensions={}
2020-11-06 08:01:31.967362: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.lhs.110 = f32[] parameter(0)
2020-11-06 08:01:31.967371: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.rhs.111 = f32[] parameter(1)
2020-11-06 08:01:31.967380: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.116 = pred[] compare(f32[] %p.0.lhs.110, f32[] %p.0.rhs.111), direction=GT, type=TOTALORDER
2020-11-06 08:01:31.967392: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %select.117 = pred[] select(pred[] %broadcast.115, pred[] %compare.116, pred[] %broadcast.115)
2020-11-06 08:01:31.967402: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.967411: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.967419: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare-greater-than.121 (p.0.lhs.122: f32[], p.0.rhs.123: f32[], p.1.lhs.124: s32[], p.1.rhs.125: s32[]) -> pred[] {
2020-11-06 08:01:31.967429: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.lhs.124 = s32[] parameter(2)
2020-11-06 08:01:31.967437: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.rhs.125 = s32[] parameter(3)
2020-11-06 08:01:31.967449: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.126 = pred[] constant(true)
2020-11-06 08:01:31.967459: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.127 = pred[] broadcast(pred[] %constant.126), dimensions={}
2020-11-06 08:01:31.967468: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.lhs.122 = f32[] parameter(0)
2020-11-06 08:01:31.967477: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.rhs.123 = f32[] parameter(1)
2020-11-06 08:01:31.967493: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.128 = pred[] compare(f32[] %p.0.lhs.122, f32[] %p.0.rhs.123), direction=GT, type=TOTALORDER
2020-11-06 08:01:31.967503: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %select.129 = pred[] select(pred[] %broadcast.127, pred[] %compare.128, pred[] %broadcast.127)
2020-11-06 08:01:31.967513: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.967537: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.967550: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %BoxSuppressLoop_body.204 (parameter.205: (s32[], s32[], pred[<=5,<=5], pred[5])) -> (s32[], s32[], pred[<=5,<=5], pred[5]) {
2020-11-06 08:01:31.967565: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %parameter.205 = (s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) parameter(0)
2020-11-06 08:01:31.967575: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.206 = s32[] get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.205), index=0
2020-11-06 08:01:31.967585: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.211 = s32[] constant(1)
2020-11-06 08:01:31.967594: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add.224 = s32[] add(s32[] %get-tuple-element.206, s32[] %constant.211)
2020-11-06 08:01:31.967603: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.209 = pred[5]{0} get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.205), index=3
2020-11-06 08:01:31.967613: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %dynamic-slice.212 = pred[1]{0} dynamic-slice(pred[5]{0} %get-tuple-element.209, s32[] %get-tuple-element.206), dynamic_slice_sizes={1}
2020-11-06 08:01:31.967623: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.213 = pred[] reshape(pred[1]{0} %dynamic-slice.212)
2020-11-06 08:01:31.967632: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.207 = s32[] get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.205), index=1
2020-11-06 08:01:31.967641: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add.214 = s32[] add(s32[] %get-tuple-element.207, s32[] %constant.211)
2020-11-06 08:01:31.967651: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.215 = s32[] select(pred[] %reshape.213, s32[] %add.214, s32[] %get-tuple-element.207)
2020-11-06 08:01:31.967660: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.208 = pred[<=5,<=5]{1,0} get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.205), index=2
2020-11-06 08:01:31.967669: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.221 = pred[5]{0} broadcast(pred[] %reshape.213), dimensions={}
2020-11-06 08:01:31.967679: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.210 = s32[] constant(0)
2020-11-06 08:01:31.967689: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %dynamic-slice.216 = pred[1,5]{1,0} dynamic-slice(pred[<=5,<=5]{1,0} %get-tuple-element.208, s32[] %get-tuple-element.206, s32[] %constant.210), dynamic_slice_sizes={1,5}
2020-11-06 08:01:31.967699: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.217 = pred[1,1]{1,0} constant({ {0} })
2020-11-06 08:01:31.967708: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %dynamic-update-slice.218 = pred[1,5]{1,0} dynamic-update-slice(pred[1,5]{1,0} %dynamic-slice.216, pred[1,1]{1,0} %constant.217, s32[] %constant.210, s32[] %get-tuple-element.206)
2020-11-06 08:01:31.967718: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.219 = pred[5]{0} reshape(pred[1,5]{1,0} %dynamic-update-slice.218)
2020-11-06 08:01:31.967735: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %not.220 = pred[5]{0} not(pred[5]{0} %reshape.219)
2020-11-06 08:01:31.967745: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %and.222 = pred[5]{0} and(pred[5]{0} %get-tuple-element.209, pred[5]{0} %not.220)
2020-11-06 08:01:31.967754: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.223 = pred[5]{0} select(pred[5]{0} %broadcast.221, pred[5]{0} %and.222, pred[5]{0} %get-tuple-element.209)
2020-11-06 08:01:31.967763: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %tuple.225 = (s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) tuple(s32[] %add.224, s32[] %select.215, pred[<=5,<=5]{1,0} %get-tuple-element.208, pred[5]{0} %select.223)
2020-11-06 08:01:31.967773: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.967781: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.967789: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %BoxSuppressLoop_condition.226 (parameter.227: (s32[], s32[], pred[<=5,<=5], pred[5])) -> pred[] {
2020-11-06 08:01:31.967798: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %parameter.227 = (s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) parameter(0)
2020-11-06 08:01:31.967806: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.230 = pred[<=5,<=5]{1,0} get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.227), index=2
2020-11-06 08:01:31.967815: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.231 = pred[5]{0} get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.227), index=3
2020-11-06 08:01:31.967823: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.228 = s32[] get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.227), index=0
2020-11-06 08:01:31.967835: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.232 = s32[] constant(5)
2020-11-06 08:01:31.967846: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.233 = pred[] compare(s32[] %get-tuple-element.228, s32[] %constant.232), direction=LT, type=UNSIGNED
2020-11-06 08:01:31.967858: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.229 = s32[] get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %parameter.227), index=1
2020-11-06 08:01:31.967868: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.234 = s32[] constant(5)
2020-11-06 08:01:31.967878: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.235 = pred[] compare(s32[] %get-tuple-element.229, s32[] %constant.234), direction=LT, type=UNSIGNED
2020-11-06 08:01:31.967887: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %and.236 = pred[] and(pred[] %compare.233, pred[] %compare.235)
2020-11-06 08:01:31.967897: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.967909: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.967918: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare-greater-than.254 (p.0.lhs.255: f32[], p.0.rhs.256: f32[], p.1.lhs.257: s32[], p.1.rhs.258: s32[]) -> pred[] {
2020-11-06 08:01:31.967928: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.lhs.257 = s32[] parameter(2)
2020-11-06 08:01:31.967937: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.1.rhs.258 = s32[] parameter(3)
2020-11-06 08:01:31.967947: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.259 = pred[] constant(true)
2020-11-06 08:01:31.967956: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.260 = pred[] broadcast(pred[] %constant.259), dimensions={}
2020-11-06 08:01:31.967965: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.lhs.255 = f32[] parameter(0)
2020-11-06 08:01:31.967974: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p.0.rhs.256 = f32[] parameter(1)
2020-11-06 08:01:31.967983: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.261 = pred[] compare(f32[] %p.0.lhs.255, f32[] %p.0.rhs.256), direction=GT, type=TOTALORDER
2020-11-06 08:01:31.967993: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %select.262 = pred[] select(pred[] %broadcast.260, pred[] %compare.261, pred[] %broadcast.260)
2020-11-06 08:01:31.968002: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.968011: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.968020: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add_S32.273 (lhs.274: s32[], rhs.275: s32[]) -> s32[] {
2020-11-06 08:01:31.968029: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %lhs.274 = s32[] parameter(0)
2020-11-06 08:01:31.968038: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %rhs.275 = s32[] parameter(1)
2020-11-06 08:01:31.968047: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %add.276 = s32[] add(s32[] %lhs.274, s32[] %rhs.275)
2020-11-06 08:01:31.968056: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.968065: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.968073: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ENTRY %SyncTensorsGraph.282 (p0.1: f32[], p1.5: f32[], p2.11: f32[5,4], p3.74: s32[], p4.87: f32[5]) -> (s32[<=5]) {
2020-11-06 08:01:31.968085: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.2 = f32[] constant(0)
2020-11-06 08:01:31.968095: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.4 = f32[] constant(0)
2020-11-06 08:01:31.968104: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.12 = f32[] constant(0)
2020-11-06 08:01:31.968113: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.13 = f32[1,1]{1,0} reshape(f32[] %constant.12)
2020-11-06 08:01:31.968122: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.14 = f32[1,1]{1,0} broadcast(f32[1,1]{1,0} %reshape.13), dimensions={0,1}
2020-11-06 08:01:31.968131: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.15 = f32[] reshape(f32[1,1]{1,0} %broadcast.14)
2020-11-06 08:01:31.968141: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.16 = f32[5,4]{1,0} broadcast(f32[] %reshape.15), dimensions={}
2020-11-06 08:01:31.968150: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.88 = f32[] constant(0)
2020-11-06 08:01:31.968159: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.89 = f32[1]{0} reshape(f32[] %constant.88)
2020-11-06 08:01:31.968174: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.90 = f32[1]{0} broadcast(f32[1]{0} %reshape.89), dimensions={0}
2020-11-06 08:01:31.968186: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.91 = f32[] reshape(f32[1]{0} %broadcast.90)
2020-11-06 08:01:31.968196: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.92 = f32[5]{0} broadcast(f32[] %reshape.91), dimensions={}
2020-11-06 08:01:31.968206: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.201 = s32[] constant(0)
2020-11-06 08:01:31.968215: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p4.87 = f32[5]{0} parameter(4)
2020-11-06 08:01:31.968224: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p2.11 = f32[5,4]{0,1} parameter(2)
2020-11-06 08:01:31.968233: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.36 = f32[5,4]{1,0} slice(f32[5,4]{0,1} %p2.11), slice={[0:5], [0:4]}
2020-11-06 08:01:31.968243: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.37 = f32[5,1]{1,0} slice(f32[5,4]{1,0} %slice.36), slice={[0:5], [2:3]}
2020-11-06 08:01:31.968252: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.38 = f32[5]{0} reshape(f32[5,1]{1,0} %slice.37)
2020-11-06 08:01:31.968262: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.32 = f32[5,4]{1,0} slice(f32[5,4]{0,1} %p2.11), slice={[0:5], [0:4]}
2020-11-06 08:01:31.968271: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.33 = f32[5,1]{1,0} slice(f32[5,4]{1,0} %slice.32), slice={[0:5], [0:1]}
2020-11-06 08:01:31.968289: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.34 = f32[5]{0} reshape(f32[5,1]{1,0} %slice.33)
2020-11-06 08:01:31.968299: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.27 = f32[] constant(1)
2020-11-06 08:01:31.968308: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.28 = f32[1]{0} reshape(f32[] %constant.27)
2020-11-06 08:01:31.971418: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.29 = f32[1]{0} broadcast(f32[1]{0} %reshape.28), dimensions={0}
2020-11-06 08:01:31.971477: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.30 = f32[] reshape(f32[1]{0} %broadcast.29)
2020-11-06 08:01:31.971488: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.31 = f32[5]{0} broadcast(f32[] %reshape.30), dimensions={}
2020-11-06 08:01:31.971496: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %multiply.35 = f32[5]{0} multiply(f32[5]{0} %reshape.34, f32[5]{0} %broadcast.31)
2020-11-06 08:01:31.971504: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %subtract.39 = f32[5]{0} subtract(f32[5]{0} %reshape.38, f32[5]{0} %multiply.35)
2020-11-06 08:01:31.971513: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p1.5 = f32[] parameter(1)
2020-11-06 08:01:31.971542: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.40 = f32[5]{0} broadcast(f32[] %p1.5), dimensions={}
2020-11-06 08:01:31.971552: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.41 = pred[5]{0} compare(f32[5]{0} %subtract.39, f32[5]{0} %broadcast.40), direction=GE, type=UNSIGNED
2020-11-06 08:01:31.971561: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.21 = f32[5,4]{1,0} slice(f32[5,4]{0,1} %p2.11), slice={[0:5], [0:4]}
2020-11-06 08:01:31.971569: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.22 = f32[5,1]{1,0} slice(f32[5,4]{1,0} %slice.21), slice={[0:5], [3:4]}
2020-11-06 08:01:31.971577: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.23 = f32[5]{0} reshape(f32[5,1]{1,0} %slice.22)
2020-11-06 08:01:31.971584: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.17 = f32[5,4]{1,0} slice(f32[5,4]{0,1} %p2.11), slice={[0:5], [0:4]}
2020-11-06 08:01:31.971592: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.18 = f32[5,1]{1,0} slice(f32[5,4]{1,0} %slice.17), slice={[0:5], [1:2]}
2020-11-06 08:01:31.971609: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.19 = f32[5]{0} reshape(f32[5,1]{1,0} %slice.18)
2020-11-06 08:01:31.971617: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.6 = f32[] constant(1)
2020-11-06 08:01:31.971625: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.7 = f32[1]{0} reshape(f32[] %constant.6)
2020-11-06 08:01:31.971633: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.8 = f32[1]{0} broadcast(f32[1]{0} %reshape.7), dimensions={0}
2020-11-06 08:01:31.971641: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.9 = f32[] reshape(f32[1]{0} %broadcast.8)
2020-11-06 08:01:31.971664: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.10 = f32[5]{0} broadcast(f32[] %reshape.9), dimensions={}
2020-11-06 08:01:31.971674: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %multiply.20 = f32[5]{0} multiply(f32[5]{0} %reshape.19, f32[5]{0} %broadcast.10)
2020-11-06 08:01:31.971682: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %subtract.24 = f32[5]{0} subtract(f32[5]{0} %reshape.23, f32[5]{0} %multiply.20)
2020-11-06 08:01:31.971690: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.25 = f32[5]{0} broadcast(f32[] %p1.5), dimensions={}
2020-11-06 08:01:31.971699: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.26 = pred[5]{0} compare(f32[5]{0} %subtract.24, f32[5]{0} %broadcast.25), direction=GE, type=UNSIGNED
2020-11-06 08:01:31.971708: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %and.42 = pred[5]{0} and(pred[5]{0} %compare.41, pred[5]{0} %compare.26)
2020-11-06 08:01:31.971726: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.43 = pred[] constant(false)
2020-11-06 08:01:31.971734: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.44 = pred[5]{0} broadcast(pred[] %constant.43), dimensions={}
2020-11-06 08:01:31.971743: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.45 = pred[5]{0} compare(pred[5]{0} %and.42, pred[5]{0} %broadcast.44), direction=NE
2020-11-06 08:01:31.971751: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %convert.46 = s32[5]{0} convert(pred[5]{0} %compare.45)
2020-11-06 08:01:31.971758: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %iota.57 = s32[5]{0} iota(), iota_dimension=0
2020-11-06 08:01:31.972175: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.58 = s32[5]{0} reshape(s32[5]{0} %iota.57)
2020-11-06 08:01:31.972216: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %sort.68 = (s32[5]{0}, s32[5]{0}) sort(s32[5]{0} %convert.46, s32[5]{0} %reshape.58), dimensions={0}, is_stable=true, to_apply=%compare-greater-than.59
2020-11-06 08:01:31.972231: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.69 = s32[5]{0} get-tuple-element((s32[5]{0}, s32[5]{0}) %sort.68), index=1
2020-11-06 08:01:31.972243: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.70 = s32[5,1]{1,0} reshape(s32[5]{0} %get-tuple-element.69)
2020-11-06 08:01:31.972255: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %concatenate.71 = s32[5,1]{1,0} concatenate(s32[5,1]{1,0} %reshape.70), dimensions={1}
2020-11-06 08:01:31.972265: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.47 = s32[] constant(0)
2020-11-06 08:01:31.972284: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.48 = s32[5]{0} broadcast(s32[] %constant.47), dimensions={}
2020-11-06 08:01:31.972295: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.49 = pred[5]{0} compare(s32[5]{0} %convert.46, s32[5]{0} %broadcast.48), direction=GT, type=UNSIGNED
2020-11-06 08:01:31.972307: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %convert.50 = s32[5]{0} convert(pred[5]{0} %compare.49)
2020-11-06 08:01:31.972317: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.51 = s32[] constant(0)
2020-11-06 08:01:31.972328: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reduce.56 = s32[] reduce(s32[5]{0} %convert.50, s32[] %constant.51), dimensions={0}, to_apply=%add_S32.52
2020-11-06 08:01:31.972340: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %set-dimension-size.72 = s32[<=5,1]{1,0} set-dimension-size(s32[5,1]{1,0} %concatenate.71, s32[] %reduce.56), dimensions={0}
2020-11-06 08:01:31.972351: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.73 = s32[<=5]{0} reshape(s32[<=5,1]{1,0} %set-dimension-size.72), inferred_dimension=0
2020-11-06 08:01:31.972362: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %convert.81 = s64[<=5]{0} convert(s32[<=5]{0} %reshape.73)
2020-11-06 08:01:31.972373: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.80 = s64[] constant(0)
2020-11-06 08:01:31.972384: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.82 = s64[5]{0} broadcast(s64[] %constant.80), dimensions={}
2020-11-06 08:01:31.972395: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.83 = pred[<=5]{0} compare(s64[<=5]{0} %convert.81, s64[5]{0} %broadcast.82), direction=LT, type=UNSIGNED
2020-11-06 08:01:31.972407: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p3.74 = s32[] parameter(3)
2020-11-06 08:01:31.972418: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.75 = s32[1]{0} reshape(s32[] %p3.74)
2020-11-06 08:01:31.972427: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.76 = s32[1]{0} broadcast(s32[1]{0} %reshape.75), dimensions={0}
2020-11-06 08:01:31.972436: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.77 = s32[] reshape(s32[1]{0} %broadcast.76)
2020-11-06 08:01:31.972444: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.78 = s32[5]{0} broadcast(s32[] %reshape.77), dimensions={}
2020-11-06 08:01:31.972452: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add.79 = s32[<=5]{0} add(s32[<=5]{0} %reshape.73, s32[5]{0} %broadcast.78)
2020-11-06 08:01:31.972460: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.84 = s32[<=5]{0} select(pred[<=5]{0} %compare.83, s32[<=5]{0} %add.79, s32[<=5]{0} %reshape.73)
2020-11-06 08:01:31.972468: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.85 = s32[<=5,1]{1,0} reshape(s32[<=5]{0} %select.84), inferred_dimension=0
2020-11-06 08:01:31.972477: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %concatenate.86 = s32[<=5,1]{1,0} concatenate(s32[<=5,1]{1,0} %reshape.85), dimensions={1}
2020-11-06 08:01:31.972486: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %gather.93 = f32[<=5]{0} gather(f32[5]{0} %p4.87, s32[<=5,1]{1,0} %concatenate.86), offset_dims={}, collapsed_slice_dims={0}, start_index_map={0}, index_vector_dim=1, slice_sizes={1}
2020-11-06 08:01:31.972500: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.108 = f32[4,<=5]{1,0} broadcast(f32[<=5]{0} %gather.93), dimensions={1}
2020-11-06 08:01:31.972509: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %convert.100 = s64[<=5]{0} convert(s32[<=5]{0} %reshape.73)
2020-11-06 08:01:31.972517: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.99 = s64[] constant(0)
2020-11-06 08:01:31.972554: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.101 = s64[5]{0} broadcast(s64[] %constant.99), dimensions={}
2020-11-06 08:01:31.972563: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.102 = pred[<=5]{0} compare(s64[<=5]{0} %convert.100, s64[5]{0} %broadcast.101), direction=LT, type=UNSIGNED
2020-11-06 08:01:31.972571: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.94 = s32[1]{0} reshape(s32[] %p3.74)
2020-11-06 08:01:31.972579: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.95 = s32[1]{0} broadcast(s32[1]{0} %reshape.94), dimensions={0}
2020-11-06 08:01:31.972587: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.96 = s32[] reshape(s32[1]{0} %broadcast.95)
2020-11-06 08:01:31.972595: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.97 = s32[5]{0} broadcast(s32[] %reshape.96), dimensions={}
2020-11-06 08:01:31.972604: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add.98 = s32[<=5]{0} add(s32[<=5]{0} %reshape.73, s32[5]{0} %broadcast.97)
2020-11-06 08:01:31.972613: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.103 = s32[<=5]{0} select(pred[<=5]{0} %compare.102, s32[<=5]{0} %add.98, s32[<=5]{0} %reshape.73)
2020-11-06 08:01:31.972622: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.104 = s32[<=5,1]{1,0} reshape(s32[<=5]{0} %select.103), inferred_dimension=0
2020-11-06 08:01:31.972630: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %concatenate.105 = s32[<=5,1]{1,0} concatenate(s32[<=5,1]{1,0} %reshape.104), dimensions={1}
2020-11-06 08:01:31.972639: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %gather.106 = f32[<=5,4]{1,0} gather(f32[5,4]{0,1} %p2.11, s32[<=5,1]{1,0} %concatenate.105), offset_dims={1}, collapsed_slice_dims={0}, start_index_map={0}, index_vector_dim=1, slice_sizes={1,4}
2020-11-06 08:01:31.972647: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %transpose.107 = f32[4,<=5]{0,1} transpose(f32[<=5,4]{1,0} %gather.106), dimensions={1,0}
2020-11-06 08:01:31.972656: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %sort.118 = (f32[4,<=5]{1,0}, f32[4,<=5]{0,1}) sort(f32[4,<=5]{1,0} %broadcast.108, f32[4,<=5]{0,1} %transpose.107), dimensions={1}, to_apply=%compare-greater-than.109
2020-11-06 08:01:31.972666: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.119 = f32[4,<=5]{0,1} get-tuple-element((f32[4,<=5]{1,0}, f32[4,<=5]{0,1}) %sort.118), index=1
2020-11-06 08:01:31.972674: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.135 = f32[1,<=5]{1,0} slice(f32[4,<=5]{0,1} %get-tuple-element.119), slice={[1:2], [0:5]}
2020-11-06 08:01:31.972683: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.136 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %slice.135)
2020-11-06 08:01:31.972692: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.139 = f32[1,<=5]{1,0} slice(f32[4,<=5]{0,1} %get-tuple-element.119), slice={[3:4], [0:5]}
2020-11-06 08:01:31.972805: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.140 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %slice.139)
2020-11-06 08:01:31.972830: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.147 = pred[<=5]{0} compare(f32[<=5]{0} %reshape.136, f32[<=5]{0} %reshape.140), direction=LE, type=UNSIGNED
2020-11-06 08:01:31.972842: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.148 = f32[<=5]{0} select(pred[<=5]{0} %compare.147, f32[<=5]{0} %reshape.140, f32[<=5]{0} %reshape.136)
2020-11-06 08:01:31.972853: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.155 = f32[1,<=5]{1,0} broadcast(f32[<=5]{0} %select.148), dimensions={1}
2020-11-06 08:01:31.972865: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.170 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %broadcast.155)
2020-11-06 08:01:31.972876: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.171 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.170), dimensions={1}
2020-11-06 08:01:31.972887: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %transpose.169 = f32[<=5,1]{0,1} transpose(f32[1,<=5]{1,0} %broadcast.155), dimensions={1,0}
2020-11-06 08:01:31.972899: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.172 = f32[<=5]{0} reshape(f32[<=5,1]{0,1} %transpose.169)
2020-11-06 08:01:31.972910: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.173 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.172), dimensions={0}
2020-11-06 08:01:31.972922: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %minimum.174 = f32[<=5,<=5]{1,0} minimum(f32[<=5,<=5]{1,0} %broadcast.171, f32[<=5,<=5]{1,0} %broadcast.173)
2020-11-06 08:01:31.972938: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.145 = pred[<=5]{0} compare(f32[<=5]{0} %reshape.136, f32[<=5]{0} %reshape.140), direction=LE, type=UNSIGNED
2020-11-06 08:01:31.972953: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.146 = f32[<=5]{0} select(pred[<=5]{0} %compare.145, f32[<=5]{0} %reshape.136, f32[<=5]{0} %reshape.140)
2020-11-06 08:01:31.972966: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.154 = f32[1,<=5]{1,0} broadcast(f32[<=5]{0} %select.146), dimensions={1}
2020-11-06 08:01:31.972978: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.158 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %broadcast.154)
2020-11-06 08:01:31.972990: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.159 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.158), dimensions={1}
2020-11-06 08:01:31.973003: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %transpose.157 = f32[<=5,1]{0,1} transpose(f32[1,<=5]{1,0} %broadcast.154), dimensions={1,0}
2020-11-06 08:01:31.973015: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.160 = f32[<=5]{0} reshape(f32[<=5,1]{0,1} %transpose.157)
2020-11-06 08:01:31.973028: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.161 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.160), dimensions={0}
2020-11-06 08:01:31.973041: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %maximum.162 = f32[<=5,<=5]{1,0} maximum(f32[<=5,<=5]{1,0} %broadcast.159, f32[<=5,<=5]{1,0} %broadcast.161)
2020-11-06 08:01:31.973053: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %subtract.183 = f32[<=5,<=5]{1,0} subtract(f32[<=5,<=5]{1,0} %minimum.174, f32[<=5,<=5]{1,0} %maximum.162)
2020-11-06 08:01:31.973066: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.181 = f32[] constant(0)
2020-11-06 08:01:31.973078: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.182 = f32[5,5]{1,0} broadcast(f32[] %constant.181), dimensions={}
2020-11-06 08:01:31.973090: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %maximum.184 = f32[<=5,<=5]{1,0} maximum(f32[<=5,<=5]{1,0} %subtract.183, f32[5,5]{1,0} %broadcast.182)
2020-11-06 08:01:31.973101: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.133 = f32[1,<=5]{1,0} slice(f32[4,<=5]{0,1} %get-tuple-element.119), slice={[0:1], [0:5]}
2020-11-06 08:01:31.973121: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.134 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %slice.133)
2020-11-06 08:01:31.973133: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.137 = f32[1,<=5]{1,0} slice(f32[4,<=5]{0,1} %get-tuple-element.119), slice={[2:3], [0:5]}
2020-11-06 08:01:31.973146: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.138 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %slice.137)
2020-11-06 08:01:31.973159: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.143 = pred[<=5]{0} compare(f32[<=5]{0} %reshape.134, f32[<=5]{0} %reshape.138), direction=LE, type=UNSIGNED
2020-11-06 08:01:31.973172: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.144 = f32[<=5]{0} select(pred[<=5]{0} %compare.143, f32[<=5]{0} %reshape.138, f32[<=5]{0} %reshape.134)
2020-11-06 08:01:31.973185: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.153 = f32[1,<=5]{1,0} broadcast(f32[<=5]{0} %select.144), dimensions={1}
2020-11-06 08:01:31.973197: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.176 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %broadcast.153)
2020-11-06 08:01:31.973210: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.177 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.176), dimensions={1}
2020-11-06 08:01:31.973223: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %transpose.175 = f32[<=5,1]{0,1} transpose(f32[1,<=5]{1,0} %broadcast.153), dimensions={1,0}
2020-11-06 08:01:31.973236: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.178 = f32[<=5]{0} reshape(f32[<=5,1]{0,1} %transpose.175)
2020-11-06 08:01:31.973249: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.179 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.178), dimensions={0}
2020-11-06 08:01:31.973261: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %minimum.180 = f32[<=5,<=5]{1,0} minimum(f32[<=5,<=5]{1,0} %broadcast.177, f32[<=5,<=5]{1,0} %broadcast.179)
2020-11-06 08:01:31.973273: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.141 = pred[<=5]{0} compare(f32[<=5]{0} %reshape.134, f32[<=5]{0} %reshape.138), direction=LE, type=UNSIGNED
2020-11-06 08:01:31.973296: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.142 = f32[<=5]{0} select(pred[<=5]{0} %compare.141, f32[<=5]{0} %reshape.134, f32[<=5]{0} %reshape.138)
2020-11-06 08:01:31.973309: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.152 = f32[1,<=5]{1,0} broadcast(f32[<=5]{0} %select.142), dimensions={1}
2020-11-06 08:01:31.973321: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.164 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %broadcast.152)
2020-11-06 08:01:31.973331: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.165 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.164), dimensions={1}
2020-11-06 08:01:31.973343: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %transpose.163 = f32[<=5,1]{0,1} transpose(f32[1,<=5]{1,0} %broadcast.152), dimensions={1,0}
2020-11-06 08:01:31.973355: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.166 = f32[<=5]{0} reshape(f32[<=5,1]{0,1} %transpose.163)
2020-11-06 08:01:31.973368: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.167 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.166), dimensions={0}
2020-11-06 08:01:31.973379: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %maximum.168 = f32[<=5,<=5]{1,0} maximum(f32[<=5,<=5]{1,0} %broadcast.165, f32[<=5,<=5]{1,0} %broadcast.167)
2020-11-06 08:01:31.973391: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %subtract.185 = f32[<=5,<=5]{1,0} subtract(f32[<=5,<=5]{1,0} %minimum.180, f32[<=5,<=5]{1,0} %maximum.168)
2020-11-06 08:01:31.973403: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %maximum.186 = f32[<=5,<=5]{1,0} maximum(f32[<=5,<=5]{1,0} %subtract.185, f32[5,5]{1,0} %broadcast.182)
2020-11-06 08:01:31.973415: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %multiply.187 = f32[<=5,<=5]{1,0} multiply(f32[<=5,<=5]{1,0} %maximum.184, f32[<=5,<=5]{1,0} %maximum.186)
2020-11-06 08:01:31.973427: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %subtract.149 = f32[<=5]{0} subtract(f32[<=5]{0} %select.144, f32[<=5]{0} %select.142)
2020-11-06 08:01:31.973440: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %subtract.150 = f32[<=5]{0} subtract(f32[<=5]{0} %select.148, f32[<=5]{0} %select.146)
2020-11-06 08:01:31.973452: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %multiply.151 = f32[<=5]{0} multiply(f32[<=5]{0} %subtract.149, f32[<=5]{0} %subtract.150)
2020-11-06 08:01:31.973464: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.156 = f32[1,<=5]{1,0} broadcast(f32[<=5]{0} %multiply.151), dimensions={1}
2020-11-06 08:01:31.973476: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.189 = f32[<=5]{0} reshape(f32[1,<=5]{1,0} %broadcast.156)
2020-11-06 08:01:31.973489: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.190 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.189), dimensions={1}
2020-11-06 08:01:31.973502: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %transpose.188 = f32[<=5,1]{0,1} transpose(f32[1,<=5]{1,0} %broadcast.156), dimensions={1,0}
2020-11-06 08:01:31.973515: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reshape.191 = f32[<=5]{0} reshape(f32[<=5,1]{0,1} %transpose.188)
2020-11-06 08:01:31.973544: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.192 = f32[<=5,<=5]{1,0} broadcast(f32[<=5]{0} %reshape.191), dimensions={0}
2020-11-06 08:01:31.973557: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add.193 = f32[<=5,<=5]{1,0} add(f32[<=5,<=5]{1,0} %broadcast.190, f32[<=5,<=5]{1,0} %broadcast.192)
2020-11-06 08:01:31.973570: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %subtract.194 = f32[<=5,<=5]{1,0} subtract(f32[<=5,<=5]{1,0} %add.193, f32[<=5,<=5]{1,0} %multiply.187)
2020-11-06 08:01:31.973582: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %divide.195 = f32[<=5,<=5]{1,0} divide(f32[<=5,<=5]{1,0} %multiply.187, f32[<=5,<=5]{1,0} %subtract.194)
2020-11-06 08:01:31.973595: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %p0.1 = f32[] parameter(0)
2020-11-06 08:01:31.973608: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.196 = f32[5,5]{1,0} broadcast(f32[] %p0.1), dimensions={}
2020-11-06 08:01:31.973619: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %add.197 = f32[5,5]{1,0} add(f32[5,5]{1,0} %broadcast.196, f32[5,5]{1,0} %broadcast.182)
2020-11-06 08:01:31.973632: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.198 = pred[<=5,<=5]{1,0} compare(f32[<=5,<=5]{1,0} %divide.195, f32[5,5]{1,0} %add.197), direction=GT, type=UNSIGNED
2020-11-06 08:01:31.973645: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.199 = pred[] constant(true)
2020-11-06 08:01:31.973658: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.200 = pred[5]{0} broadcast(pred[] %constant.199), dimensions={}
2020-11-06 08:01:31.973671: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %tuple.203 = (s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) tuple(s32[] %constant.201, s32[] %constant.201, pred[<=5,<=5]{1,0} %compare.198, pred[5]{0} %broadcast.200)
2020-11-06 08:01:31.973685: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %while.237 = (s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) while((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %tuple.203), condition=%BoxSuppressLoop_condition.226, body=%BoxSuppressLoop_body.204
2020-11-06 08:01:31.973696: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.239 = s32[] get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %while.237), index=1
2020-11-06 08:01:31.973709: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.240 = pred[<=5,<=5]{1,0} get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %while.237), index=2
2020-11-06 08:01:31.973721: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %iota.120 = s32[5]{0} iota(), iota_dimension=0
2020-11-06 08:01:31.973732: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %sort.130 = (f32[<=5]{0}, s32[5]{0}) sort(f32[<=5]{0} %gather.93, s32[5]{0} %iota.120), dimensions={0}, to_apply=%compare-greater-than.121
2020-11-06 08:01:31.973745: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.132 = f32[<=5]{0} get-tuple-element((f32[<=5]{0}, s32[5]{0}) %sort.130), index=0
2020-11-06 08:01:31.973757: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.3 = f32[] constant(0)
2020-11-06 08:01:31.973769: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.242 = f32[5]{0} broadcast(f32[] %constant.3), dimensions={}
2020-11-06 08:01:31.973781: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.243 = pred[<=5]{0} compare(f32[<=5]{0} %get-tuple-element.132, f32[5]{0} %broadcast.242), direction=GT, type=UNSIGNED
2020-11-06 08:01:31.973794: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.241 = pred[5]{0} get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %while.237), index=3
2020-11-06 08:01:31.973807: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %and.244 = pred[<=5]{0} and(pred[<=5]{0} %compare.243, pred[5]{0} %get-tuple-element.241)
2020-11-06 08:01:31.973819: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.238 = s32[] get-tuple-element((s32[], s32[], pred[<=5,<=5]{1,0}, pred[5]{0}) %while.237), index=0
2020-11-06 08:01:31.973832: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.245 = s32[5]{0} broadcast(s32[] %get-tuple-element.238), dimensions={}
2020-11-06 08:01:31.973845: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %compare.246 = pred[5]{0} compare(s32[5]{0} %iota.120, s32[5]{0} %broadcast.245), direction=LT, type=UNSIGNED
2020-11-06 08:01:31.973858: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %and.247 = pred[<=5]{0} and(pred[<=5]{0} %and.244, pred[5]{0} %compare.246)
2020-11-06 08:01:31.973871: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.202 = s32[] constant(1)
2020-11-06 08:01:31.973883: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.270 = s32[5]{0} broadcast(s32[] %constant.202), dimensions={}
2020-11-06 08:01:31.973895: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.271 = s32[5]{0} broadcast(s32[] %constant.201), dimensions={}
2020-11-06 08:01:31.973907: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.272 = s32[<=5]{0} select(pred[<=5]{0} %and.247, s32[5]{0} %broadcast.270, s32[5]{0} %broadcast.271)
2020-11-06 08:01:31.973920: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %reduce.277 = s32[] reduce(s32[<=5]{0} %select.272, s32[] %constant.201), dimensions={0}, to_apply=%add_S32.273
2020-11-06 08:01:31.973933: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.278 = s32[] constant(5)
2020-11-06 08:01:31.973945: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %minimum.279 = s32[] minimum(s32[] %reduce.277, s32[] %constant.278)
2020-11-06 08:01:31.973956: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.131 = s32[5]{0} get-tuple-element((f32[<=5]{0}, s32[5]{0}) %sort.130), index=1
2020-11-06 08:01:31.973969: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %constant.248 = f32[] constant(-inf)
2020-11-06 08:01:31.973981: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %broadcast.249 = f32[5]{0} broadcast(f32[] %constant.248), dimensions={}
2020-11-06 08:01:31.973993: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %select.250 = f32[<=5]{0} select(pred[<=5]{0} %and.247, f32[<=5]{0} %get-tuple-element.132, f32[5]{0} %broadcast.249)
2020-11-06 08:01:31.974006: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %iota.251 = s32[5]{0} iota(), iota_dimension=0
2020-11-06 08:01:31.974019: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-dimension-size.252 = s32[] get-dimension-size(f32[<=5]{0} %select.250), dimensions={0}
2020-11-06 08:01:31.974030: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %set-dimension-size.253 = s32[<=5]{0} set-dimension-size(s32[5]{0} %iota.251, s32[] %get-dimension-size.252), dimensions={0}
2020-11-06 08:01:31.974043: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %sort.263 = (f32[<=5]{0}, s32[<=5]{0}) sort(f32[<=5]{0} %select.250, s32[<=5]{0} %set-dimension-size.253), dimensions={0}, is_stable=true, to_apply=%compare-greater-than.254
2020-11-06 08:01:31.974055: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.264 = f32[<=5]{0} get-tuple-element((f32[<=5]{0}, s32[<=5]{0}) %sort.263), index=0
2020-11-06 08:01:31.974067: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.265 = f32[<=5]{0} slice(f32[<=5]{0} %get-tuple-element.264), slice={[0:5]}
2020-11-06 08:01:31.974080: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.266 = s32[<=5]{0} get-tuple-element((f32[<=5]{0}, s32[<=5]{0}) %sort.263), index=1
2020-11-06 08:01:31.974093: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %slice.267 = s32[<=5]{0} slice(s32[<=5]{0} %get-tuple-element.266), slice={[0:5]}
2020-11-06 08:01:31.974105: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %tuple.268 = (f32[<=5]{0}, s32[<=5]{0}) tuple(f32[<=5]{0} %slice.265, s32[<=5]{0} %slice.267)
2020-11-06 08:01:31.974118: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %get-tuple-element.269 = s32[<=5]{0} get-tuple-element((f32[<=5]{0}, s32[<=5]{0}) %tuple.268), index=1
2020-11-06 08:01:31.974131: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] %gather.280 = s32[<=5]{0} gather(s32[5]{0} %get-tuple-element.131, s32[<=5]{0} %get-tuple-element.269), offset_dims={}, collapsed_slice_dims={0}, start_index_map={0}, index_vector_dim=1, slice_sizes={1}
2020-11-06 08:01:31.974146: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] ROOT %tuple.281 = (s32[<=5]{0}) tuple(s32[<=5]{0} %gather.280)
2020-11-06 08:01:31.974159: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] }
2020-11-06 08:01:31.974171: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974183: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974195: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] OutputShape: (s32[<=5]{0})
2020-11-06 08:01:31.974207: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974220: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] StackTrace:
2020-11-06 08:01:31.974232: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] *** Begin stack trace ***
2020-11-06 08:01:31.974244: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] tensorflow::CurrentStackTrace()
2020-11-06 08:01:31.974257: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] xla::util::ReportComputationError(tensorflow::Status const&, absl::lts_2020_02_25::Span<xla::XlaComputation const* const>, absl::lts_2020_02_25::Span<xla::Shape const* const>)
2020-11-06 08:01:31.974271: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] xla::XrtComputationClient::CheckCompileStatus(tensorflow::Status const&, std::vector<xla::ComputationClient::CompileInstance, std::allocator<xla::ComputationClient::CompileInstance> > const&, xla::XrtComputationClient::SessionWork const&)
2020-11-06 08:01:31.974298: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974310: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] xla::util::MultiWait::Complete(std::function<void ()> const&)
2020-11-06 08:01:31.974323: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974335: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974346: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974358: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] clone
2020-11-06 08:01:31.974370: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] *** End stack trace ***
2020-11-06 08:01:31.974381: E tensorflow/compiler/xla/xla_client/xla_util.cc:76]
2020-11-06 08:01:31.974392: E tensorflow/compiler/xla/xla_client/xla_util.cc:76] Status: Aborted: Session 7b79209253db996f is not found.
Traceback (most recent call last):
File "test.py", line 131, in <module>
test_nms()
File "test.py", line 126, in test_nms
print(keep.cpu())
RuntimeError: Aborted: Session 7b79209253db996f is not found.
terminate called after throwing an instance of 'std::runtime_error'
what(): tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:1110 : Check failed: session->session()->Run( feed_inputs, {}, {cached_node.operations[0]}, &outputs) == ::tensorflow::Status::OK() (Aborted: Session 7b79209253db996f is not found. vs. OK)
*** Begin stack trace ***
tensorflow::CurrentStackTrace()
xla::XrtComputationClient::ReleaseHandles(std::vector<xla::XrtComputationClient::DeviceHandle, std::allocator<xla::XrtComputationClient::DeviceHandle> >*, std::function<xla::XrtSession::CachedNode const& (xla::XrtSession*, tensorflow::Scope const&, std::string const&)> const&, xla::metrics::Metric*, xla::metrics::Counter*)
xla::XrtComputationClient::HandleReleaser()
xla::util::TriggeredTask::Runner()
clone
*** End stack trace ***
I will look at the nms issue. for your question, print(dynamic_tensor) doesn't always work, I believe print looks at the shape of the tensor which is static in this case. If you use torch_xla._XLAC._get_xla_tensors_hlo([boxes[keep]], 0) I think it will return 4(dynamic first dimension size) while the shape of boxes[keep]_dim_0 is 5.
JackCaoG
on 6 Nov 2020
👍2
The crashed happened on the TPU server side and is dynamic shape related. I am working with the xla team to investigate the issue.
JackCaoG
on 10 Nov 2020
👍1
@Clive2312 I merged the fix, you should be able to see that if you use tomorrow's nightly. Dynamic shape support is still experimental so you might encounter more issues, please let us know and we will try to fix them.
JackCaoG
on 12 Nov 2020
Hi Jack, thank you very much for the fix. I will try the nightly version and do more experiments on the dynamic shape feature.
Clive2312
on 13 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
magicknight
路
3Comments
ogulcanogul
路
5Comments
cjolivier01
路
7Comments
ibeltagy
路
8Comments
CalebEverett
路
4Comments
Most helpful comment
I will look at the nms issue. for your question,
print(dynamic_tensor)doesn't always work, I believeprintlooks at the shape of the tensor which is static in this case. If you usetorch_xla._XLAC._get_xla_tensors_hlo([boxes[keep]], 0)I think it will return4(dynamic first dimension size) while the shape ofboxes[keep]_dim_0is 5.